In 1999 NRI’s founder developed methods for entirely-lensless light-field imaging and cameras. As an example, an array of (easily printable) simple gridded holes (or other repeated optical structures) can be used together with computational image processing to create a refocusable image entirely in software. The image is formed by a mathematical algorithm, not by optics. The early patents on this were developed while NRI’s founder was at Avistar Communications Corporation, but NRI has acquired all the patent assets, technology base, and other rights. NRI continues to actively develop this technology.
This fundamentally new type of imaging was independently re-discovered some 12 years later by Cornell and is currently under active development at Rambus
In NRI’s version, this lensless imaging technology is actually a “light-field imaging” technology capable of refocusing previously captured images (or selective parts of them) entirely in software. Initially driven by vast press coverage of the Lytro camera, “light-field imaging” has attracted tremendous commercial attention in all aspects of imaging and DSP chips (for example by Qualcomm). The attention and markets ramped up considerably when Toshiba announced a cellphone light field camera made available in 1Q14. Unlike these popular recently-commercialized light-field camera technologies (which employ both macro-lenses and as many as thousands of microlenses),NRI’s lensless light-field imaging technology employs no lenses at all, and can focus clearly at zero separation distance. It is also possible to simultaneously image from more than one location in the sensor array (permitting 3D-imaging, multiple vantage-point imaging, etc.) and to provide simultaneous imaging at different focus settings. NRI’s patents cover a wide range of computational image processing approaches (including non-singular Fourier, error-smoothing oversampled algebraic generalized-inverse, and deconvolution) as well as a wide range of light-field sensing (diffractive, non-diffractive, non-zero transfer function, vignette arrays, use of display pixels to structure light-field incidence on interleaved sensor pixels, etc.) that can even be fabricated entirely by printing.
The light sensor array used can be implemented with CMOS, but more interestingly in NRI’s approach it can be implemented using the photoelectric properties of LEDs and OLEDs, allowing implementation (including color imaging) employing an OLED display and fabrication done entirely with printing technology, even on a curved surface. Additionally, via multiplexing, a LED or OLED display can simultaneously function as both an active visual or video image display and a lensless real-time imaging or video camera (for example US 8,754,842), for example enabling (low-cost, flat, mirror-less) high-performance eye-contact in video conferencing (eye-contact remains the “#1” cited problem in video conferencing; see for example)
Importantly, because of the lensless nature, the wide range of possible fabrication techniques, and the software-based imaging, the cameras can be ultra-miniature (as with the Rambus/Cornell approach) or on the scale of a window, wall, table, etc. Large-scale implementations can additionally provide entirely new types of proximate user experiences that are impossible with lens-based or pin-hole cameras. The immense value of ultra-miniature implementations are spelled out in in Rambus’s ultra-miniature imaging whitepaper.
NRI is additionally please to have hosted two years of summer internships in our lensless imaging work involving talents 2nd and 3rd year undergraduate students from MIT, U.C, San Diego, and U.C. Irvine.
NRI’s portfolio of patent assets in this area is listed below
|Title||Patent Number||Application Number||Priority Dates||Text Only||Related Patents|
|Lensless imaging camera performing image formation in software employing micro-optic elements creating overlap of light from distant sources over multiple photo sensor elements||9,172,850||14/105,123||01/27/1999||Text||Lensless Light Field Imaging|
|Lensless imaging camera performing image formation in software and employing micro-optic elements that impose light diffractions||9,160,894||14/105,085||01/27/1999||Text||Lensless Light Field Imaging|
|Vignetted optoelectronic array for use in synthetic image formation via signal processing, lensless cameras, and integrated camera-displays ||8,830,375||12/828,171||05/25/2008||Text||Lensless Light Field Imaging|
|Vignetted planar spatial light-field sensor and spatial sampling designs for far-field lensless synthetic imaging via signal processing image formation ||8,816,263||13/452,461||04/20/2011||Text||Lensless Light Field Imaging|
|Combined display and image capture without simple or compound lenses for video conferencing eye-contact and other applications||8,754,842||12/419,229||01/27/1999||Text||Lensless Light Field Imaging|
|Synthetic Image Formation via Signal Processing for Vignetted Optoelectronic Arrays, Lensless Cameras, and Integrated Camera-Displays||8,305,480||12/828,207||05/25/2008||Text||Lensless Light Field Imaging|
|Synthetic Image Formation Signal Processing Hardware for Vignetted Optoelectronic Arrays, Lensless Cameras, and Integrated Camera-Displays||8,284,290||12/828,228||05/25/2008||Text||Lensless Light Field Imaging|
|Image Formation for Large Photosensor Array Surfaces||8,125,559||12/471,275||05/25/2008||Text||Lensless Light Field Imaging|
|Multifunction Communication Service Device||2,318,395 CA||N/A||01/27/1999||Lensless Light Field Imaging|
|Title||Publication Number||Application Number||Priority Dates||Text Only||Related Patents|
|Color imaging using array of wavelength-selective optoelectronic elements as light-field image or sensor||2017/0373214||15/676,749||03/25/2010||Text||Lensless Light Field Imaging|
|Color Imaging Using Color LED Array as Light-Field Image Sensor||2011/0234558||13/072,588||03/25/2010||Text||Lensless Light Field Imaging|
Pending Unpublished Applications
|Title||Application Number||Priority Dates||Related Patents|
|Vignetted planar spatial light-field sensor and spatial sampling designs for far-field lensless synthetic imaging via signal processing image formation||14/333,177||04/20/2011||Lensless Light and Field Imaging
|Vignetted Optoelectronic Array for Use in Synthetic Image Formation via Signal Processing, Lensless Cameras, and Integrated Camera-Displays||14/478,920||05/25/2008||Lensless Light and Field Imaging
|Advanced Lensless Light-Field Imaging Systems and Methods for Enabling a Wide Range of Entirely New Applications||15/647,230|| 07/11/2016|
|Lensless Light and Field Imaging