Patent application status for multi modal interfaces

Patent application status for multi modal interfaces

(ex. voice and gesture)

Multi modality* technologies are evolving to allow humans to interact with computers more conveniently.

*Multi Modal refers to an environment where a human can communicate meaning with a computer via various forms of communication. "Modal" means modality, which refers to "the communication channel used in the interaction process."

A multi modal interface involves user-friendly technology that enables natural communication between humans and computers by fusing multiple input methods such as voice, gestures, gaze, facial expression and bio signals in addition to traditional text. Today, a change from conventional machine-centered input to human-centered natural input methods is taking place.

According to KIPO, a total of 149 patent applications were filed in relation to multi modal interfaces over the last 10 years (2009 ~ 2018).

While the number of patent applications related to multi modal interfaces was relatively small from 2009 to 2015, more recently it has increased to 12 cases in 2016, 20 cases in 2017, and 39 cases in 2018.

Looking at the most recent five years (2014-2018), approximately two inputs were typically used up until 2016, but after 2017, additional inputs such as gestures, face and bio-signals have been incorporated and thus patent applications using three or more input methods are gradually increasing.

Comparing 2014 and 2018, the bio signals area (pulse, heart rate, iris and fingerprint) showed the largest increase of 5.5 times, from 2 cases to 11 cases, followed by the sound area (voice, music and sound) showing an increase of 4 times, from 7 cases to 28 cases.

Areas related to Face, gesture and smell also increased 2.9 times, from 12 cases to 35 cases, while the traditional text-based area (keyboard, touchpad and electronic pen) declined from 9 cases to 7 cases.

Looking by applicant, Korean applications involved 88 cases (94%) in total, research institutes and university applications constituted 51 cases (54%), followed by 28 cases (9%) from individuals and Small and Medium-sized Enterprise (SME) applications, 9 cases (10%) of applications were from major companies, and finally, foreign applications accounted for 6 cases (6%).

Looking at specific applicants with multiple applications, ETRI filed 21 cases, Samsung Electronics filed 5 cases, LG Electronics filed 4 cases, AMOREPACIFIC and Korea Institute of Science and Technology filed 4 cases each, and Acryl Inc., Smart Always-On Corporation, Kookmin University and Chungnam National University each filed 3 cases.

The Chief of Computer Systems Examination Division in KIPO said, “Such multi modal technology has already been put into practical use for games and entertainment and has been spreading to technologies connected to the Internet of Things (IoT), and in the future, in a world where computers are ubiquitously worn on the whole body, it will evolve to include input methods that add artificial intelligence mimicing human learning ability.”

1. Multi modal interface patent application trend