Touch, Gesture, and HDTP

NRI and NRI’s founder have been working in the advanced touchscreen, touch-sensor, and gesture technology areas (including hand gestures captured by video) for more than two calendar decades.

NRI’s touchscreen, touch-sensor, and gesture technology 1999 patent filing included:

  • Touchscreen single finger and multi-touch gestures that are now pervasive in consumer electronics,
  • Free-space hand gestures captured by video (capabilities that have been in many mobile phone products),
  • Finger angle (roll, pitch, and yaw) gestures:
    • Single finger touch responsive to six touch aspects simultaneously (left-right, forward-back, downward pressure, roll angle, pitch angle, yaw angle), allowing advanced metaphors, six-dimensional control, and natural intuitive control of the position and rotation control of maps, viewpoints, and 3D objects in three-dimensional space with a single finger (NRI’s 6DTouchTM)
    • Multitouch, adding up to 3 additional control parameters for each added finger (left-right variation, forward-back variation, downward pressure variation; NRI’s Xdimension-MultiTouchTM),
    • Finger posture and part-of-hand recognition (NRI’s TouchPostureTM),
    • User training and ‘ongoing adaption’ capabilities to improved performance,
    • Gesture grammars (NRI’s GestureGrammarTM).

It is noted that rudimentary finger-angle sensing employing use of opaque (non-touchscreen) fingerprint sensor was explored in a 2002 IBM US 6,400,836 patent by Senior with 1998 priority date exactly to the day one year earlier than NRI’s 1999 parent utility patent filing, although NRI staff had developed this capability earlier than this IBM patent filing. In addition to the use of touch sensor arrays, the work patented in 1999 also included implementations of NRI’s touch technology with simple inexpensive resistive touchscreens and touchpads; using a “trick” from electronic music keyboards, a constant current source and subtraction operations permitted sensing of single fingers and multitouch adequate enough to implement all the famous iPhoneTM gestures (flick, stretch, squeeze, swipe, tap, double tap, and drag), allowing these celebrated game-changing touchscreen gestures to have been implemented in mobile devices many years prior to the commercial release of the iPhone with far-cheaper presently-available technologies.

It is also noted that using either (full capability) transparent touchsensor array touchscreens or (reduced capabilities) resistive touchscreen technologies taught in NRI’s 1999 patent filings, this early version of touchscreen gesture technology and available implementation approaches predated Jeff Han’s famous and celebrated 2006 TED talk on touchscreen-based multitouch user interfaces.

In 2007, the iPhone (with its famous flick, stretch, and squeeze gestures implemented with a transparent touch sensor array) appeared, providing a gesture touchscreen independently developed from early opaque touchpad work developed by Westerman and others of FingerWorksTM. The Westerman implementation available at the time of NRI’s patent filings and for years afterwards could not be implemented as a touchscreen; pairs of transistors collocated with every touch sensor element in the touch sensor array were required, and the first invention of transparent transistors did not occur until four years later in 2003.

NRI continued to develop its far more advanced touchscreen and touch gesture technologies with its own internal funds. Between mid-2008 and early-2012 NRI’s work towards commercializing the aspects of these technologies was subsidized by National Science Foundation Grants SBIR (Small Business Innovation Research) Phase I and II grants:

  • NSF Award Number 0741237: SBIR Phase I (2008)
  • NSF Award Number 0923986: SBIR Phase II (2009-2011)
  • NSF SBIR-Supplemental Awards (2011, 2012).

The first video provided above on this NRI R&D/New Technology page is a conceptual piece made in this interval by NRI’s Ms. Seung Lim, used to seek commercialization investment funding in 2009 prior to Phase II funding award, although NRI had many partial prototypes of the functions depicted therein. It is noted that while at NRI Ms. Lim was the sole inventor of a series of NRI patents applications directed towards user applications of NRI’s 6DTouchTM and Xdimension MultiTouchTM technologies. The second video provided above on this NRI R&D/New Technology page was a brief demonstration of E. Sum’s (Princeton University) 2009 internship project using a TekscanTM pressure sensor array. The third video presents later examples of some of the NSF Phase II SBIR era touch functionality, controlled using an optical touch sensor. Most of the software developed in the NSF Phase II SBIR era was developed by V. Zaliva with some of this reported in part in the conference paper 3D Finger Posture Detection and Gesture Recognition on Touch Surfaces (ICARCV 2012) as well as in a number of patents in the lists below.

Also during this period, alternative sensor technologies were developed, including technologies that employed LEDs and OLEDs as optical sensors. Similar ideas were developed independently by Jeff Han at NYU, although NRI’s approach was to transform LED (these days tagged “micro-LED”) displays into optical-reflection touchscreen sensors. NRI’s techniques also includes an unperceivable modulated carrier illumination signature in the light emitted from the touchscreen for touch reflections and corresponding receiver approach to reject ambient light (up to brightness levels forcing total saturation limits of the electronics signal chain). NRI adopted the name High Definition Touch PadTM and acronym HDTPTM suggested by Dr. S. Simon (during the 2008 Phase I SBIR) as a comprehensive term for NRI’s 6DTouchTM, Xdimension MultiTouchTM, TouchPostureTM, GestureGrammarTM, and touch sensor technologies.

Subsequent NRI touchscreen and gesture work included further development of gesture grammar technology and integration of NRI’s OLED/micro-LED optical touchscreen technologies with NRI’s Lensless Light Field Imaging technologies to create powerful multifunction Integrated Lensless Camera, Touch, and Display technologies (using adaptations of commodity OLED/micro-LED displays) for smartphones and mobile devices. Many current smartphone OLED/micro-LED displays resolutions are adequate for totally-optical (self-illuminated) fingerprint acquisition from a smartphone touchscreen.

In March 2015 a number of NRI patent assets in the touch and video gesture area, some of which are cover several fundamental popular touchscreen gestures that are pervasively used in smart phones, tablets, laptops, PDAs, mobile devices, computers, kiosks, and other devices and systems, were assigned to the then newly-formed NRI spinout company Advanced Touchscreen and Gesture Technologies, LLC (ATGT). ATGT is currently working to commercialize 6DTouchTM, Xdimension MultiTouchTM, High Definition Touch Pad (HDTPTM), and GestureGrammarTM technologies originally developed by NRI which it will be licensing from NRI.

In late 2015 Qeexo, independently developed and beautifully publicized some finger angle capabilities very similar to NRI’s 6DTouchTM technologies; some of those videos are still available on the internet, for example the short “Qeexo’s FingerAngle” video. Qeexo’s FingerSenseTM technology also features part-of-hand recognition and user training to improve performance, both described in NRI’s 1999 patent filing.

Two other related comments:

  • With appropriate caveats carved out to the work of Myron Kruger’s “butterfly installation,” this NRI technology also was a multiple-year precursor to the video-camera “free-space gesture” industry (see the early US 8,519,250 and more general-industry US 8,878,807, both with 1999 inventorship rights and also assigned to Advanced Touchscreen and Gesture Technologies, LLC (ATGT). Many years after NRI’s 1999 patent filing, “free-space gesture” technology appeared in at least Android smart phones and was more broadly featured on the front cover and feature article of Communications of the ACM Volume 54 Issue 2, February 2011.
  • This pre-1999 NRI technology also included a multiple-year precursor to the wearable devices industry (see claims on the recently filed pp.119-120 of US 15/634,084 with 1999 inventorship rights; this patent asset has also been assigned to Advanced Touchscreen and Gesture Technologies, LLC (ATGT).
  • Issued Patents

    TitlePatent NumberApplication NumberPriority DatesPDFText OnlyRelated Patents
    Curve-fitting Approach to HDTP Parameter Extraction 10,146,42713/038,37203/01/2010PDFText Touch, Gesture, and HDTP
    General spatial-gesture grammar user interface for touchscreens, high dimensional touch pad (HDTP), free-space camera, and other user interfaces 10,073,53215/260,84005/04/2011PDFText Touch, Gesture, and HDTP
    Heterogeneous Tactile Sensing Via Multiple Sensor Types Using Spatial Information Processing10,042,47915/804,77812/06/2011PDFText Touch, Gesture, and HDTP
    High-Dimensional Touchpad Game Controller with Multiple Usage and Networking Modalities9,950,25613/198,691 03/15/2013
    PDFText Touch, Gesture, and HDTP
    Enhanced roll-over, button, menu, slider, and hyperlink environments for high dimensional touchpad (HTPD), other advanced touch user interfaces, and advanced mice 9,830,04213/026,2482/12/2011PDFText Touch, Gesture, and HDTP
    Heterogeneous tactile sensing via multiple sensor types
    9,823,781 13/706,21412/05/2012PDFText Touch, Gesture, and HDTP
    General user interface gesture lexicon and grammar frameworks for multi-touch, high dimensional touch pad (HDTP), free-space camera, and other user interfaces
    9,442,65213/414,70503/07/2011
    05/04/2011
    PDFText Touch, Gesture, and HDTP
    Piecewise-linear and piecewise-affine subspace transformations for high dimensional touchpad (HDTP) output decoupling and corrections9,389,71314/175,98304/23/2010PDFTextTouch, Gesture, and HDTP
    Heuristics for 3D and 6D touch gesture touch parameter calculations for high-dimensional touch parameter (HDTP) user interfaces9,052,77213/572,60508/10/2011PDFText Touch, Gesture, and HDTP
    Multitouch parameter and gesture user interface employing an LED-array tactile sensor that can also operate as a display
    9,019,23712/418,60504/06/2008
    09/18/2008
    11/20/2008
    03/23/2009
    PDFText Touch, Gesture, and HDTP
    Touch user interface supporting global and context-specific touch gestures that are responsive to at least one finger angle8,894,48914/198,48207/12/2008PDFTextTouch, Gesture, and HDTP
    High-performance closed-form single-scan calculation of oblong-shape rotation angles from optical image data of arbitrary size and location for user interface applications
    8,879,84414/101,11103/14/2009PDFText Touch, Gesture, and HDTP
    Human user interfaces utilizing interruption of the execution of a first recognized gesture with the execution of a recognized second gesture8,797,28813/414,60003/07/2011
    05/04/2011
    PDFTextTouch, Gesture, and HDTP
    Sequential Classification Recognition of Gesture Primitives and Window-Based Parameter Smoothing For High Dimensional Touchpad (HDTP) User Interfaces8,754,86213/180,51207/11/2010
    07/09/2011
    PDFText Touch, Gesture, and HDTP
    Control of the Operating System On A Computing Device Via Finger Angle Using A High Dimensional Touchpad (HDTP) Touch User Interface
    8,702,51313/731,946
    07/12/2008PDFText Touch, Gesture, and HDTP
    Piecewise-Linear and Piecewise-Affine Transformations For High Dimensional Touchpad (HDTP) Output Decoupling and Corrections8,686,96013/093,83404/23/2010PDFTextTouch, Gesture, and HDTP
    Advanced Touch Control of Graphics Design Application Via Finger Angle Using A High Dimensional Touchpad (HDTP) Touch User Interface
    8,643,622
    13/786,346
    07/12/2008
    07/13/2009
    PDFText Touch, Gesture, and HDTP
    High-Performance Closed-Form Single-Scan Calculation of Oblong-Shape Rotation Angles from Image Data of Arbitrary Size Using Running Sums
    8,639,037
    13/846,830

    03/14/2009PDFText Touch, Gesture, and HDTP
    Advanced Touch Control of a File Browser Via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface
    8,638,312
    13/786,265
    07/12/2008
    07/13/2009
    PDFText Touch, Gesture, and HDTP
    Sensors, Algorithms and Applications for a High Dimensional Touchpad8,604,36412/541,94808/15/2008PDFTextTouch, Gesture, and HDTP
    Advanced Touch Control of Interactive Map Viewing via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface.8,542,20913/442,80607/12/2008
    07/13/2009
    PDFTextTouch, Gesture, and HDTP
    High-Performance Closed-Form Single-Scan Calculation of Oblong-Shape Rotation Angles from Binary Images of Arbitrary Size and Location Using Running Sums8,509,54213/441,84203/14/2009PDFTextTouch, Gesture, and HDTP
    Advanced Touch Control of Interactive Immersive Imaging Applications via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface8,477,11113/442,81507/12/2008
    07/13/2009
    PDFTextTouch, Gesture, and HDTP
    Control of the Operating System on a Computing Device Via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface8,345,01412/502,23007/12/2008PDFText Touch, Gesture, and HDTP
    High-Performance Closed-Form Single-Scan Calculation of Oblong-Shape Rotation Angles from Binary Images of Arbitrary Size Using Running Sums8,170,34612/724,41303/14/2009PDFText Touch, Gesture, and HDTP
    Control of Electronic Games via Finger Angle Using a High Dimensional Touchpad (HDTP) Touch User Interface8,169,41412/511,93007/12/2008
    07/13/2009
    PDFText Touch, Gesture, and HDTP

    Pending Patents

    TitlePublication NumberApplication NumberPriority DatesPDFText OnlyRelated Patents
    Heterogeneous Tactile Sensing Via Multiple Sensor Types Using Spatial Information Processing Acting on Initial Image Processed Data from Each Sensor2018/033589316/051,23312/06/2011PDFTextTouch, Gesture, and HDTP
    High-Dimensional Touch Parameter (HDTP) Game Controllers With Multiple Usage And Networking Modalities2018/020752615/926,727
    08/05/2010PDFTextTouch, Gesture, and HDTP
    Enhanced Roll-Over, Button, Menu, Slider, and Hyperlink Environments for High Dimensional Touchpad (HDTP), Other Advanced Touch User Interfaces, and Advanced Mice2018/0074680 15/815,446 02/12/2010PDFTextTouch, Gesture, and HDTP
    Piecewise-linear and piecewise-affine subspace transformations for high dimensional touchpad (HDTP) output decoupling and corrections2017/022814415/207,34604/23/2010PDFTextTouch, Gesture, and HDTP
    Advanced touch control of a media player application via yaw finger angle using a high dimensional touchpad (HDTP) touch user interface2017/0220246 15/489,61607/12/2008PDFTextTouch, Gesture, and HDTP
    Wearable Gesture Based Control Device2014/026712314/292,77005/15/1999PDFTextTouch, Gesture, and HDTP
    Gesteme (Gesture Primitive) Recognition for Advanced Touch User Interfaces2013/014137513/707,51812/06/2011PDFTextTouch, Gesture, and HDTP
    3D Finger Posture Detection and Gesture Recognition on Touch Surfaces2013/000989613/544,96007/09/2011PDFTextTouch, Gesture, and HDTP
    Simple Touch Interface and HDTP Grammars for Rapid Operation of Physical Computer Aided Design (CAD) Systems2012/028092713/464,94605/04/2011PDFTextTouch, Gesture, and HDTP
    Touch-Based User Interfaces Employing Artificial Neural Networks for HDTP Parameter and Symbol Derivation2012/005684613/038,36503/01/2010PDFTextTouch, Gesture, and HDTP
    Window Manager Input Focus Control for High Dimensional Touchpad (HDTP), Advanced Mice, and Other Multidimensional User Interfaces2011/020293413/026,09702/12/2010PDFText Touch, Gesture, and HDTP

    Pending Unpublished Applications

    TitleApplication NumberPriority DatesRelated Patents
    Curve-fitting Approach to Touch Gesture Finger Pitch Parameter Extraction16/205,08303/01/2010Touch, Gesture, and HDTP

    Gesture-Level Grammar Arrangements for Spatial-Gesture User Interfaces such as Touchscreen, High Dimensional Touch Pad (HDTP), Free-Space Camera, and Other User Interface Technologies16/121,40203/07/2011 05/04/2011Touch, Gesture, and HDTP

    Heterogeneous Tactile Sensing Via Multiple Sensor Types Using Spatial Information Processing Acting on Initial Image Processed Data from Each Sensor16/051,23312/06/2011Touch, Gesture, and HDTP

    Human User Interfaces Recognizing Interruption of the Execution of a First Gesture with the Execution of a Second Gesture Utilizing Pause or Lift-Off
    14/450,24203/07/2011
    05/04/2011
    Touch, Gesture, and HDTP
    Sequential Classification Recognition of Gesture Primitives and Window-Based Parameter Smoothing For High Dimensional Touchpad (HDTP) User Interfaces
    14/306,175
    07/11/2010
    07/09/2011
    Touch, Gesture, and HDTP