Lensless Light-Field Imaging

In 1999 NRI’s founder developed methods for entirely-lensless light-field imaging and cameras. As an example, an (easily 3D-printable) grid, array of holes, or other repeated optical vignetting structure can be used together with computational image processing to create a refocusable image entirely in software. The image is formed by a mathematical algorithm, not by optical processes. The early patents and prototypes were developed by NRI’s founder while at Avistar Communications Corporation, but NRI has subsequently acquired all patent assets, technology base, and other rights. NRI continues to actively develop this technology.

NRI is presently working to secure funding for a fourth spin out company, NoLens, Inc., in response to aspects of:

  • the expanding interest in plenoptic light-field imaging since the 2011 introduction of the Lytro camera and its 2011 Raytrix, 2004 Stanford University, and 1992 Adelson/Wang precursors,
  • the many emerging coded-aperture, phase-grading, and related lensless imaging cameras (circa 2011-2013 Cornell/Rambus “Fourier-Domain Microscale” and “Ultraminiature” imaging cameras, circa 2015 Rice University “FlatCam,” 2016 Hitachi Moire-Pattern lensless camera slated for 2018 productization, the 2017 CalTech Phase-Array lenseless camera, and the 2016 U.C. Berkeley “DiffuserCam,” among others).

All of these (be they vignetting, visible-light coded aperture masks, diffraction spirals, Moiré pattern masks, phase-arrays, internal-reflection propagation  of captured spatial light scatter to a transparent window’s edge, unstructured diffusion materials, the raw light-sensor array itself, and beyond) use appended or inherent optical structures to capture an incident light-field and impose a linear transformation of source light to an image sensing array. Image formation can be set up as a classical inverse problem (for example employing error minimization), or use matrix inverses/generalized-inverses/pseudo-inverses; if the transformation is such that its frequency-domain transform has no zeros, Fourier inversion via numerical division by transfer function can be used to recover an image. These concepts, methods, and prototype-provable results (and many extensions) are mentioned throughout NRI’s 1999, 2008, 2010, 2011, and 2016 NRI patent families.

However, NRI’s lensless light-field imaging technologies and intellectual property go far beyond these explorations of various optical structures and the remarkable image-formation and light-field-capture riches deliverable through applying various inverse transformations to data from light-field sensing arrangements. NRI’s lensless light-field imaging technologies and intellectual property have from their pre-1999 beginnings reached much further in capabilities, features, predictable manufacturing, transcending use of silicon CMOS sensors and its imposed scale/size/materials/fabrication limitations, integration with displays and illumination, and implementations directed to enabling wide ranges of previously-impossible commercial applications.

Although light-field imaging cameras date back to the 1908 work of Lippmann and coded-aperture (Gamma-Ray and X-Ray) imaging date back to the 1965 modulation collimator work of Oda and 1968 coded mask work of Dicke, the NRI visible-light lensless imaging camera technology suite dates back to NRI’s founder’s 1999 and 2008-2011 work at Avistar (that work and technology subsequently acquired from Avistar by NRI). The NRI approach is (1) widely-inclusive, (2) far broader in scope, implementation, features, capabilities, and technique, and (3) would appear to be of profound impact to the future of electronic imaging systems. NRI patent U.S. 10,217,235 includes a representative review to the immense implications of the NRI lensless light-field imaging approach.

NRI’s  lensless imaging technology has been from its 1999 beginnings a “light-field imaging” technology capable of arbitrarily focusing previously captured light field data (or selective parts of it) entirely in software. Initially driven by vast press coverage of the Lytro camera, “light-field imaging” has attracted tremendous commercial attention in all aspects of imaging and DSP chips (for example by Qualcomm). The attention and markets ramped up considerably when Toshiba announced a cellphone light field camera made available in 1Q14. Unlike these popular recently-commercialized light-field camera technologies (which employ both macro-lenses and as many as thousands of microlenses),NRI’s lensless light-field imaging technology employs no lenses at all, and can focus clearly at zero separation distance. It is also possible to simultaneously image from more than one location in the sensor array (permitting 3D-imaging, multiple vantage-point imaging, etc.) and to provide simultaneous imaging at different focus settings. NRI’s patents cover a wide range of computational image processing approaches (including non-singular Fourier, error-smoothing oversampled algebraic generalized-inverse, and deconvolution) as well as a wide range of light-field sensing (diffractive, non-diffractive, non-zero transfer function, vignette arrays, use of display pixels to structure light-field incidence on interleaved sensor pixels, etc.) that can even be fabricated entirely by printing.

The light sensor array used can be implemented with CMOS, but more interestingly in NRI’s approach it can be implemented using the photoelectric properties of LEDs and OLEDs, allowing implementation (including color imaging) employing an OLED display and fabrication done entirely with printing technology, even on a curved surface. Additionally, via multiplexing, a LED or OLED display can simultaneously function as both an active visual or video image display and a lensless real-time imaging or video camera (for example US 8,754,842), for example enabling (low-cost, flat, mirror-less) high-performance eye-contact in video conferencing (eye-contact remains the “#1” cited problem in video conferencing).

Importantly, because of the lensless nature, the wide range of possible fabrication techniques, and the software-based imaging, the cameras can be ultra-miniature (as with the Rambus/Cornell approach) or on the scale of a window, wall, table, etc. Large-scale implementations can additionally provide entirely new types of proximate user experiences that are impossible with lens-based or pin-hole cameras. The immense value of ultra-miniature implementations are spelled out in Rambus’s ultra-miniature imaging whitepaper and other technical papers.

NRI is additionally pleased to have hosted many lensless imaging internships involving undergraduate and post-graduate students from U.C.Santa Cruz,  MIT, U.C. San Diego, U.C. Berkeley, and U.C. Irvine.

NRI’s Lensless Light-field Imaging Technology provides a number of valuable new and unique features, most not available from other lensless computational imaging approaches. Depending on the method of implementation, these can readily include:

    • Does not require separation distance between the sensor and any masking, vignetting or aperturing array, allowing thinner image sensor than coded mask imaging;
    • No need for any vignetting or aperturing array if the photosensing elements natively have adequate angular occultation or angular selectively;
    • Vignetting or aperturing can be done at the same spatial separation as individual photosensing pixels;
    • Any vignetting or aperturing array can additionally include internally-reflective structures and have arbitrarily thin walls to preclude light loss;
    • Any vignetting or aperturing array can be also arranged to facilitate predictable or and/or reproducible surface plasmon propagation to selected light sensors comprised by the
      light sensor array in a manner that further reduces light loss;
    • Truly arbitrary image sensor size;
    • Flat or arbitrarily-curved image sensor shape;
    • Distributed image sensing;
    • Angular diversity/redundancy advantage;
    • Enveloping imaging, contact imaging (including local illumination in contact imaging);
    • Integrated in a visual light-emitting display;
    • Overlay on provided surfaces;
    • Self-illuminating;
    • Can acquire focused images at effectively-zero separation distance;
      Curved-surface contact focus;
    • Can provide one or more simultaneous computationally-controlled focus (mixed focus);
    • Can provide one or more simultaneous computationally-controlled viewpoint(s);
    • Can provide one more simultaneous computationally-controlled stereo/3D live imaging;
    • Can provide full-color and enhanced (meta-RGB) color image capture;
    • Can include IR and UV capabilities;
    • Can include multiple-wavelength spectroscopic capabilities without diffraction-grading or prism optics.

Issued Patents

TitlePatent NumberApplication NumberPriority DatesPDFText OnlyRelated Patents
Color Imaging Using Array of Wavelength-Selective Optoelectronic Elements as Light-Field or Image Sensor and Additionally Capable of Emitting Light10,546,97016/289,17503/25/2010PDFText Lensless Light-Field Imaging
Advanced Lensless Light-Field Imaging Systems for Enabling a Wide Range of Entirely New Applications 10,546,382
16/283,60407/11/2016
07/03/2017
PDFText Lensless Light-Field Imaging
Color imaging using array of wavelength-selective optoelectronic elements as light-field image or sensor10,230,01615/676,74903/25/2010PDFTextLensless Light-Field Imaging
Advanced Lensless Light-Field Imaging Systems and Methods for Enabling a Wide Range of Entirely New Applications10,217,23515/647,23007/11/2016
07/03/2017
PDFText Lensless Light-Field Imaging
Color Imaging Using Color LED Array as Light-Field Image Sensor9,735,30313/072,58803/25/2010PDFText Lensless Light Field Imaging
Lensless imaging camera performing image formation in software employing micro-optic elements creating overlap of light from distant sources over multiple photosensor elements9,172,85014/105,12301/27/1999PDFText Lensless Light-Field Imaging
Lensless imaging camera performing image formation in software and employing micro-optic elements that impose light diffractions9,160,89414/105,08501/27/1999PDFText Lensless Light-Field Imaging
Vignetted optoelectronic array for use in synthetic image formation via signal processing, lensless cameras, and integrated camera-displays
8,830,37512/828,17105/25/2008PDFText Lensless Light-Field Imaging
Vignetted planar spatial light-field sensor and spatial sampling designs for far-field lensless synthetic imaging via signal processing image formation
8,816,26313/452,46104/20/2011PDFText Lensless Light-Field Imaging
Combined display and image capture without simple or compound lenses for video conferencing eye-contact and other applications8,754,84212/419,22901/27/1999PDFText Lensless Light-Field Imaging
Synthetic Image Formation via Signal Processing for Vignetted Optoelectronic Arrays, Lensless Cameras, and Integrated Camera-Displays8,305,48012/828,20705/25/2008PDFText Lensless Light-Field Imaging
Synthetic Image Formation Signal Processing Hardware for Vignetted Optoelectronic Arrays, Lensless Cameras, and Integrated Camera-Displays8,284,29012/828,22805/25/2008PDFText Lensless Light-Field Imaging
Image Formation for Large Photosensor Array Surfaces8,125,55912/471,27505/25/2008PDFText Lensless Light-Field Imaging
Multifunction Video Communication Service Device2,318,395 CA N/A01/27/1999PDFText Lensless Light Field Imaging

Pending Published Applications

TitlePublication NumberApplication NumberPriority DatesPublish DatePDFText OnlyRelated Patents

Pending Unpublished Applications

TitleApplication NumberPriority DatesRelated Patents
Color Light-Field or Imaging Sensor Array of Inherently Wavelength-Selective Optoelectronic Elements with Color-Space Transformation, Additionally Capable of Emitting Light for Use as a Display16/773,95503/01/2010Lensless Light-Field Imaging
Non-Smooth Image Sensor Surface Lensless Light-Field Imaging Systems for Enabling a Wide Range of Entirely New Applications16/773,92507/11/16
07/03/17
Lensless Light-Field Imaging
Vignetted Optoelectronic Array for Use in Synthetic Image Formation via Signal Processing, Lensless Cameras, and Integrated Camera-Displays 14/478,920
05/25/2008Lensless Light-Field Imaging
Vignetted planar spatial light-field sensor and spatial sampling designs for far-field lensless synthetic imaging via signal processing image formation
14/333,177
04/20/2011Lensless Light-Field Imaging