NRI and NRI’s founder have been working in the advanced touchscreen, touch-sensor, and gesture technology areas (including hand gestures captured by video) for more than two calendar decades.
NRI’s touchscreen, touch-sensor, and gesture technology 1999 patent filing included:
- Touchscreen single finger and multi-touch gestures that are now pervasive in consumer electronics,
- Free-space hand gestures captured by video (capabilities that have been in many mobile phone products),
- Finger angle (roll, pitch, and yaw) gestures:
- Single finger touch responsive to six touch aspects simultaneously (left-right, forward-back, downward pressure, roll angle, pitch angle, yaw angle), allowing advanced metaphors, six-dimensional control, and natural intuitive control of the position and rotation control of maps, viewpoints, and 3D objects in three-dimensional space with a single finger (NRI’s 6DTouchTM)
- Multitouch, adding up to 3 additional control parameters for each added finger (left-right variation, forward-back variation, downward pressure variation; NRI’s Xdimension-MultiTouchTM),
- Finger posture and part-of-hand recognition (NRI’s TouchPostureTM),
- User training and ‘ongoing adaption’ capabilities to improved performance, gesture grammars (NRI’s GestureGrammarTM).
It is noted that rudimentary finger-angle sensing employing use of opaque (non-touchscreen) fingerprint sensor was explored in a 2002 IBM US 6,400,836 patent by Senior with 1998 priority date exactly to the day one year earlier than NRI’s 1999 parent utility patent filing, although NRI staff had developed this capability earlier than this IBM patent filing. In addition to the use of touch sensor arrays, the work patented in 1999 also included implementations of NRI’s touch technology with simple inexpensive resistive touchscreens and touchpads; using a “trick” from electronic music keyboards, a constant current source and subtraction operations permitted sensing of single fingers and multitouch adequate enough to implement all the famous iPhoneTM gestures (flick, stretch, squeeze, swipe, tap, double tap, and drag), allowing these celebrated game-changing touchscreen gestures to have been implemented in mobile devices many years prior to the commercial release of the iPhone with far-cheaper presently-available technologies.
It is also noted that using either (full capability) transparent touchsensor array touchscreens or (reduced capabilities) resistive touchscreen technologies taught in NRI’s 1999 patent filings, this early version of touchscreen gesture technology and available implementation approaches predated Jeff Han’s famous and celebrated 2006 TED talk on touchscreen-based multitouch user interfaces.
In 2007, the iPhone (with its famous flick, stretch, and squeeze gestures implemented with a transparent touch sensor array) appeared, providing a gesture touchscreen independently developed from early opaque touchpad work developed by Westerman and others of FingerWorksTM. The Westerman implementation available at the time of NRI’s patent filings and for years afterwards could not be implemented as a touchscreen; pairs of transistors collocated with every touch sensor element in the touch sensor array were required, and the first invention of transparent transistors did not occur until four years later in 2003.
NRI continued to develop its far more advanced touchscreen and touch gesture technologies with its own internal funds. Between mid-2008 and early-2012 NRI’s work towards commercializing the aspects of these technologies was subsidized by National Science Foundation Grants SBIR (Small Business Innovation Research) Phase I and II grants:
- NSF Award Number 0741237: SBIR Phase I (2008)
- NSF Award Number 0923986: SBIR Phase II (2009-2011)
- NSF SBIR-Supplemental Awards (2011, 2012).
The first video provided above on this NRI R&D/New Technology page is a conceptual piece made in this interval by NRI’s Ms. Seung Lim, used to seek commercialization investment funding in 2009 prior to Phase II funding award, although NRI had many partial prototypes of the functions depicted therein. It is noted that while at NRI Ms. Lim was the sole inventor of a series of NRI patents applications directed towards user applications of NRI’s 6DTouchTM and Xdimension MultiTouchTM technologies. The second video provided above on this NRI R&D/New Technology page was a brief demonstration of E. Sum’s (Princeton University) 2009 internship project using a TekscanTM pressure sensor array. The third video presents later examples of some of the NSF Phase II SBIR era touch functionality, controlled using an optical touch sensor. Most of the software developed in the NSF Phase II SBIR era was developed by V. Zaliva with some of this reported in part in the conference paper 3D Finger Posture Detection and Gesture Recognition on Touch Surfaces (ICARCV 2012) as well as in a number of patents in the lists below.
Also during this period, alternative sensor technologies were developed, including technologies that employed LEDs and OLEDs as optical sensors. Similar ideas were developed independently by Jeff Han at NYU, although NRI’s approach was to transform LED (these days tagged “micro-LED”) displays into optical-reflection touchscreen sensors. NRI’s techniques also includes an unperceivable modulated carrier illumination signature in the light emitted from the touchscreen for touch reflections and corresponding receiver approach to reject ambient light (up to brightness levels forcing total saturation limits of the electronics signal chain). NRI adopted the name High Definition Touch PadTM and acronym HDTPTM suggested by Dr. S. Simon (during the 2008 Phase I SBIR) as a comprehensive term for NRI’s 6DTouchTM, Xdimension MultiTouchTM, TouchPostureTM, GestureGrammarTM, and touch sensor technologies.
Subsequent NRI touchscreen and gesture work included further development of gesture grammar technology and integration of NRI’s OLED/micro-LED optical touchscreen technologies with NRI’s Lensless Light Field Imaging technologies to create powerful multifunction Integrated Lensless Camera, Touch, and Display technologies (using adaptations of commodity OLED/micro-LED displays) for smartphones and mobile devices. Many current smartphone OLED/micro-LED displays resolutions are adequate for totally-optical (self-illuminated) fingerprint acquisition from a smartphone touchscreen.
In March 2015 a number of NRI patent assets in the touch and video gesture area, some of which are cover several fundamental popular touchscreen gestures that are pervasively used in smart phones, tablets, laptops, PDAs, mobile devices, computers, kiosks, and other devices and systems, were assigned to the then newly-formed NRI spinout company Advanced Touchscreen and Gesture Technologies, LLC (ATGT) ATGT is currently working to commercialize 6DTouchTM, Xdimension MultiTouchTM, High Definition Touch Pad (HDTPTM), and GestureGrammarTM technologies originally developed by NRI which it will be licensing from NRI.
In late 2015 Qeexo, independently developed and beautifully publicized some finger angle capabilities very similar to NRI’s 6DTouchTM technologies; some of those videos are still available on the internet, for example the short “Qeexo’s FingerAngle” video. Qeexo’s FingerSenseTM technology also features part-of-hand recognition and user training to improve performance, both described in NRI’s 1999 patent filing.
Two other related comments:
-
- With appropriate caveats carved out to the work of Myron Kruger’s “butterfly installation,” this NRI technology also was a multiple-year precursor to the video-camera “free-space gesture” industry (see the early US 8,519,250 and more general-industry US 8,878,807, both with 1999 inventorship rights and also assigned to Advanced Touchscreen and Gesture Technologies, LLC (ATGT). Many years after NRI’s 1999 patent filing, “free-space gesture” technology appeared in at least Android smart phones and was more broadly featured on the front cover and feature article of Communications of the ACM Volume 54 Issue 2, February 2011.
- This pre-1999 NRI technology also included a multiple-year precursor to the wearable devices industry (see claims on the recently filed pp.119-120 of US 15/634,084 with 1999 inventorship rights; this patent asset has also been assigned to Advanced Touchscreen and Gesture Technologies, LLC (ATGT).
Issued Patents
| Title | Patent Number | Application Number | Priority Dates | Text Only | Related Patents | |
|---|---|---|---|---|---|---|
| Curve-fitting Approach to Touch Gesture Finger Pitch Parameter Extraction | 10,664,156 | 16/205,083 | 03/01/2010 | Text | Touch, Gesture, and HDTP | |
| Gesteme (gesture primitive) recognition for advanced touch user interfaces | 10,430,066 | 13/707,518 | 12/06/2011 | Text | Touch, Gesture, and HDTP | |
| Heterogeneous tactile sensing via multiple sensor types using spatial information processing acting on initial image processed data from each sensor | 10,429,997 | 16/051,233 | 10/06/2009 | Text | Touch, Gesture, and HDTP | |
| Piecewise-linear and piecewise-affine subspace transformations for high dimensional touchpad (HDTP) output decoupling and corrections | 10,216,399 | 15/207,346 | 04/23/2010 | Text | Touch, Gesture, and HDTP | |
| Curve-fitting Approach to HDTP Parameter Extraction | 10,146,427 | 13/038,372 | 03/01/2010 | Text | Touch, Gesture, and HDTP | |
| General spatial-gesture grammar user interface for touchscreens, high dimensional touch pad (HDTP), free-space camera, and other user interfaces | 10,073,532 | 15/260,840 | 05/04/2011 | Text | Touch, Gesture, and HDTP | |
| Heterogeneous Tactile Sensing Via Multiple Sensor Types Using Spatial Information Processing | 10,042,479 | 15/804,778 | 12/06/2011 | Text | Touch, Gesture, and HDTP | |
| High-Dimensional Touchpad Game Controller with Multiple Usage and Networking Modalities | 9,950,256 | 13/198,691 | 08/05/2010 | Text | Touch, Gesture, and HDTP | |
| Enhanced roll-over, button, menu, slider, and hyperlink environments for high dimensional touchpad (HTPD), other advanced touch user interfaces, and advanced mice | 9,830,042 | 13/026,248 | 2/12/2011 | Text | Touch, Gesture, and HDTP | |
| Heterogeneous tactile sensing via multiple sensor types | 9,823,781 | 13/706,214 | 12/06/2011 | Text | Touch, Gesture, and HDTP | |
| General user interface gesture lexicon and grammar frameworks for multi-touch, high dimensional touch pad (HDTP), free-space camera, and other user interfaces | 9,442,652 | 13/414,705 | 03/07/2011 05/04/2011 | Text | Touch, Gesture, and HDTP | |
| Piecewise-linear and piecewise-affine subspace transformations for high dimensional touchpad (HDTP) output decoupling and corrections | 9,389,713 | 14/175,983 | 04/23/2010 | Text | Touch, Gesture, and HDTP | |
| Heuristics for 3D and 6D touch gesture touch parameter calculations for high-dimensional touch parameter (HDTP) user interfaces | 9,052,772 | 13/572,605 | 08/10/2011 | Text | Touch, Gesture, and HDTP | |
| Multitouch parameter and gesture user interface employing an LED-array tactile sensor that can also operate as a display | 9,019,237 | 12/418,605 | 04/06/2008 09/18/2008 11/20/2008 03/23/2009 | Text | Touch, Gesture, and HDTP | |
| Touch user interface supporting global and context-specific touch gestures that are responsive to at least one finger angle | 8,894,489 | 14/198,482 | 07/12/2008 | Text | Touch, Gesture, and HDTP | |
| High-performance closed-form single-scan calculation of oblong-shape rotation angles from optical image data of arbitrary size and location for user interface applications | 8,879,844 | 14/101,111 | 03/14/2009 | Text | Touch, Gesture, and HDTP | |
| Human user interfaces utilizing interruption of the execution of a first recognized gesture with the execution of a recognized second gesture | 8,797,288 | 13/414,600 | 03/07/2011 05/04/2011 | Text | Touch, Gesture, and HDTP | |
| Sequential Classification Recognition of Gesture Primitives and Window-Based Parameter Smoothing For High Dimensional Touchpad (HDTP) User Interfaces | 8,754,862 | 13/180,512 | 07/11/2010 07/09/2011 | Text | Touch, Gesture, and HDTP |
|
| Control of the Operating System On A Computing Device Via Finger Angle Using A High Dimensional Touchpad (HDTP) Touch User Interface | 8,702,513 | 13/731,946 | 07/12/2008 | Text | Touch, Gesture, and HDTP |
|
| Piecewise-Linear and Piecewise-Affine Transformations For High Dimensional Touchpad (HDTP) Output Decoupling and Corrections | 8,686,960 | 13/093,834 | 04/23/2010 | Text | Touch, Gesture, and HDTP | |
| Advanced Touch Control of Graphics Design Application Via Finger Angle Using A High Dimensional Touchpad (HDTP) Touch User Interface | 8,643,622 | 13/786,346 | 07/12/2008 07/13/2009 | Text | Touch, Gesture, and HDTP | |
| High-Performance Closed-Form Single-Scan Calculation of Oblong-Shape Rotation Angles from Image Data of Arbitrary Size Using Running Sums | 8,639,037 | 13/846,830 | 03/14/2009 | Text | Touch, Gesture, and HDTP | |
| Advanced Touch Control of a File Browser Via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface | 8,638,312 | 13/786,265 | 07/12/2008 07/13/2009 | Text | Touch, Gesture, and HDTP | |
| Sensors, Algorithms and Applications for a High Dimensional Touchpad | 8,604,364 | 12/541,948 | 08/15/2008 | Text | Touch, Gesture, and HDTP | |
| Advanced Touch Control of Interactive Map Viewing via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface. | 8,542,209 | 13/442,806 | 07/12/2008 07/13/2009 | Text | Touch, Gesture, and HDTP | |
| High-Performance Closed-Form Single-Scan Calculation of Oblong-Shape Rotation Angles from Binary Images of Arbitrary Size and Location Using Running Sums | 8,509,542 | 13/441,842 | 03/14/2009 | Text | Touch, Gesture, and HDTP | |
| Advanced Touch Control of Interactive Immersive Imaging Applications via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface | 8,477,111 | 13/442,815 | 07/12/2008 07/13/2009 | Text | Touch, Gesture, and HDTP | |
| Control of the Operating System on a Computing Device Via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface | 8,345,014 | 12/502,230 | 07/12/2008 | Text | Touch, Gesture, and HDTP | |
| High-Performance Closed-Form Single-Scan Calculation of Oblong-Shape Rotation Angles from Binary Images of Arbitrary Size Using Running Sums | 8,170,346 | 12/724,413 | 03/14/2009 | Text | Touch, Gesture, and HDTP | |
| Control of Electronic Games via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface | 8,169,414 | 12/511,930 | 07/12/2008 07/13/2009 | Text | Touch, Gesture, and HDTP |
Pending Published Applications
| Title | Publication Number | Application Number | Priority Dates | Publish Date | Text Only | Related Patents | |
|---|---|---|---|---|---|---|---|
| Piecewise-Linear and Piecewise-Affine Subspace Transformations for Finger-Angle and Spatial Measurement Decoupling and Correction In Single-Touch and Multi-Touch Touchpad and Touchscreen Systems | 2019/0187888 | 16/283,562 | 04/23/2010 | Text | Touch, Gesture, and HDTP | ||
| Curve-fitting Approach to Touch Gesture Finger Pitch Parameter Extraction | 2019/0265879 | 16/205,083 | 03/01/2010 | Text | Touch, Gesture, and HDTP | ||
| Gesture-Level Grammar Arrangements for Spatial-Gesture User Interfaces such as Touchscreen, High Dimensional Touch Pad (HDTP), Free-Space Camera, and Other User Interface Technologies | 2019/0087010 | 16/121,402 | 03/07/2011 05/04/2011 | Text | Touch, Gesture, and HDTP | ||
| Heterogeneous Tactile Sensing Via Multiple Sensor Types Using Spatial Information Processing Acting on Initial Image Processed Data from Each Sensor | 2018/0335893 | 16/051,233 | 12/06/2011 | Text | Touch, Gesture, and HDTP | ||
| High-Dimensional Touch Parameter (HDTP) Game Controllers With Multiple Usage And Networking Modalities | 2018/0207526 | 15/926,727 | 08/05/2010 | Text | Touch, Gesture, and HDTP | ||
| Enhanced Roll-Over, Button, Menu, Slider, and Hyperlink Environments for High Dimensional Touchpad (HDTP), Other Advanced Touch User Interfaces, and Advanced Mice | 2018/0074680 | 15/815,446 | 02/12/2010 | Text | Touch, Gesture, and HDTP | ||
| Advanced touch control of a media player application via yaw finger angle using a high dimensional touchpad (HDTP) touch user interface | 2017/0220246 | 15/489,616 | 07/12/2008 | Text | Touch, Gesture, and HDTP | ||
| Wearable Gesture Based Control Device | 2014/0267123 | 14/292,770 | 05/15/1999 | 09/18/14 | Text | Touch, Gesture, and HDTP | |
| Gesteme (Gesture Primitive) Recognition for Advanced Touch User Interfaces | 2013/0141375 | 13/707,518 | 12/06/2011 | 06/06/13 | Text | Touch, Gesture, and HDTP | |
| 3D Finger Posture Detection and Gesture Recognition on Touch Surfaces | 2013/0009896 | 13/544,960 | 07/09/2011 | 01/10/13 | Text | Touch, Gesture, and HDTP | |
| Simple Touch Interface and HDTP Grammars for Rapid Operation of Physical Computer Aided Design (CAD) Systems | 2012/0280927 | 13/464,946 | 05/04/2011 | 11/08/12 | Text | Touch, Gesture, and HDTP | |
| Touch-Based User Interfaces Employing Artificial Neural Networks for HDTP Parameter and Symbol Derivation | 2012/0056846 | 13/038,365 | 03/01/2010 | 03/08/12 | Text | Touch, Gesture, and HDTP | |
| Window Manager Input Focus Control for High Dimensional Touchpad (HDTP), Advanced Mice, and Other Multidimensional User Interfaces | 2011/0202934 | 13/026,097 | 02/12/2010 | 08/18/11 | Text | Touch, Gesture, and HDTP |
Pending Unpublished Applications
| Title | Application Number | Priority Dates | Related Patents |
|---|---|---|---|
| Heterogeneous Tactile Sensing Via Multiple Sensor Types Using Spatial Information Processing Acting on Initial Image Processed Data from Each Sensor | 16/051,233 | 12/06/2011 | Touch, Gesture, and HDTP |
| Hierarchical Touch Primitives, Gesteme, and Gesture Recognition with Grammar and Prosody Features for Advanced Touch User Interfaces | 16/588,768 | 12/06/2011 | Touch, Gesture, and HDTP |
| Heterogeneous Tactile Sensing Via Multiple Sensor Types Using Sensor Fusion Processing Acting on Initial Spatial Processed Data from Each Sensor | 16/588,862 | 12/06/2011 | Touch, Gesture, and HDTP |
| Methods for High-Dimensional Touch Parameter (HDTP) Game Controllers with Multiple Usage and Networking Modalities | 16/836,877 | 08/05/2010 | Touch, Gesture, and HDTP |
| Curve-Fitting Approach to Touch Gesture Finger Pitch and Roll Parameter Extraction in the Presence of Finger Yaw Angles | 16/884,008 | 03/01/2010 | Touch, Gesture, and HDTP |
| Sequential Classification Recognition of Gesture Primitives and Window-Based Parameter Smoothing For High Dimensional Touchpad (HDTP) User Interfaces | 14/306,175 | 07/11/2010 07/09/2011 | Touch, Gesture, and HDTP |
| Human User Interfaces Recognizing Interruption of the Execution of a First Gesture with the Execution of a Second Gesture Utilizing Pause or Lift-Off | 14/450,242 | 03/07/2011 05/04/2011 | Touch, Gesture, and HDTP |