Data Visualization and Sonification

1. Simultaneous Multiple-Channel Timbre-Based Data Sonification

Data sonification is the use of non-speech audio to convey information or perceptualize data. Powerful specifics of auditory perception (temporal and pitch resolution, timbral parsing, melodic parsing, cocktail-party effect, and many others) permit sonification offers an interesting alternative or complement to data visualization techniques.

NRI’s multichannel data sonification technologies provide parallel perceptual channels for representing complex numerical data to a user seeking to identify correlations within data. NRI’s multichannel data sonification technologies include:

  • Representations of several simultaneously-varying data quantities as time-varying sound for presentation to a user for observing variations and correlations among sonic events or trends.
  • Each of several simultaneously sounding data-modulated sound timbre classes can each separately rendered within a stereo sound field,
  • Interactive capabilities permitting exploration of large complex large-dimension data sets,
  • Arrangements and selections of sound timbre classes to convey sonic metaphors,
  • Providing a user interface that includes a plurality of metaphors and spatial sound rendering operations resulting in stereo audio output,
  • Each audio waveform can be positioned within a sonically-rendered sound field, associating information with positions within the sound field,
  • Mappings can be used to associate timbre and other sound parameters with selected values from the multidimensional numerical data,
  • Arrangements for explicit use of data sonification with data visualization to provide parallel perceptual channels for representing complex numerical data to a user seeking to identify correlations within the data; for example:
    • Several varying data quantities can be represented by time-varying graphics while
      several other varying data quantities are represented by time-varying sound,
    • Visual representations can be used to spatially choose or identify position or
      paths in a dataset; for example, a cursor can be moved over visualized data as
      associated multidimensional numerical data values are sonified,
    • Sonification can be used to offload some information-carrying information
      capacity from a visualization system, while other information can be rendered via
      both sonification and visualization to provide affirming or orienting redundancy,
    • Joint and coordinated visual-sonic metaphors can used for this or other purposes
      (for example, sonification can include multiple data-modulated sound timbre
      classes, each rendered within a stereo sound field according to a spatial
      metaphor that is shared with the visualization),
  • These capabilities can be implemented in the context of web browsers and spreadsheets so as to provide web access and a real-time collaboration environment.

NRI’s data sonification can be used to add valuable interactive sonification features mathematical packages such as MathematicaTM (which does not yet have real-time interactive sonficationfeatures) and MatLabTM, used to valuably supplement visual capabilities in business dashboards and spreadsheet programs, employed in plant control consoles, integrated into automotive, medical, and industrial diagnostic systems, and add similar value in many other application areas.

2. Value-Driven Visualization Primitives for Tabular Data of Spreadsheets

NRI’s tabular data visualization technology includes approaches to transcend the capabilities of (and simplify the use of) “conditional formatting” in spreadsheets and related data representation systems. Responsive to associated data value(s), or functions of those values, the color, outlines, and other visual effects of spreadsheet cells, displayed data within the cells, icons, etc. can be simply setup and vary continuously over ranges of associated data values.

NRI’s tabular data visualization technology also includes graphical tools and primitives for visualization of graphical surfaces from tabular data such as in a spreadsheet. Visualizations of a two-dimensional surface can be rendered from provided data according to position, orientation, and viewpoint, some or all of which can be controlled by a user interface. The surface can also represent interactively controlled filtered or statistically-processed tabular data. The surface can be suspended over a region of the displayed tabular data or displayed in other ways. The visually rendered surfaces can comprise symbols, colors, textures, or glyphs representing data, and virtual illumination. Visual height measuring and level sets slicing tools can be provided.

NRI’s tabular data visualization technology can be arranged to rendered graphics in a web browser and responsive to data transmitted over a network. Interactive control can also be transmitted over a network so as to provide web access and a real-time collaboration environment.

NRI’s tabular data visualization technology can be arranged to process data via function blocks, with connectivity among the blocks specified by drawing tool or palette of function blocks, for example clicking on each function block raising dialog windows that can be used for setting parameters.

3. Surface-Curve and Surface-Surface Graphical Intersection Tools and Primitives for Data Visualization

NRI’s graphical intersection-visualization technology includes graphical tools and primitives for interactively intersecting graphical surface with another graphical surface or graphical curve in a visualization setting to produce intersection data and/or intersection graphics. Among other things, such intersections can be interpreted as solutions to systems of equations implied by the data sets. Example features include:

  • Visualizations of two-dimensional surfaces are rendered from provided data according
    to position, orientation, and viewpoint, some or all of which can be controlled by a
    user interface so as to create or control the intersection of parametrized versions
    of the surfaces,
  • The provided data can be provided by an electronic spreadsheet,
  • The visualization setting for the intersection can be displayed together with the
    display an electronic spreadsheet or other tabular data representation,
  • Visually rendered surfaces can comprise symbols, colors, textures, or glyphs
    representing data, and virtual illumination,
  • Intersection data corresponding to the mathematical intersection of the surfaces is
    computed and provided to further computation. Intersection computation can comprise
    interpolation and resampling,
  • These capabilities can be implemented in the context of web browsers and spreadsheets
    so as to provide web access and a real-time collaboration environment.

Potential applications include simple business analysis, minimum cost problems, demographic analysis, economics, logistics, and a wide variety of applications in engineering and science.

4. Interactive WYSIWYG Presentation Control of Mathematical and Statistical 2D and 3D Visual Plots

NRI’s interactive WYSIWYG (“What You See Is What You Get”) plot presentation and formatting technology for mathematical and statistical visual plots provides interactive adjustment of plot and data visualization through clicks, rollovers, menus, and other familiar types of rapid user-machine interaction. Example features and capabilities include:

  • Interactive adjustments also modify associated software code used to generate the
    underlying plot or data visualization. In some implementations this feature may be
    always active,
  • Simple interactive control can be provided by simple mice and their equivalents,
  • More intricate and powerful interactive control can be provided by gesture-based
    touch interfaces advanced High-Dimensional Touch Pads (HDTP) and associated touch
    screens, game controllers, 6D-mice, and extended hyperlink objects (see Touch,
    Gesture, and HDTP
    ),

  • Arrangements to provide related features to more general programming languages not
    involved in plots or visualization, allowing users to invoke extended sets of
    options via interactive GUIs,
  • These capabilities can be implemented in the context of web browsers and
    spreadsheets so as to provide web access and a real-time collaboration
    environment.

Application areas for NRI’s interactive WYSIWYG plot presentation and formatting technology include business intelligence, simple spreadsheet plots, mathematical packages such as MathematicaTM and MatLabTM, and a wide range of data visualization applications.

Issued Patents

TitlePatent NumberApplication NumberPriority DatesPDFText OnlyRelated Patents
Multi-channel data sonification employing data-modulated sound timbre classes 10,037,186 15/585,70206/16/2009PDFText Data Visualization And Sonification
Value-Driven Visualization Primitives for Tabular Data of Spreadsheets9,665,55413/669,43109/02/2009PDFTextData Visualization And Sonification
Joint and coordinated visual-sonic metaphors for interactive multi-channel data sonification to accompany data visualization9,646,58914/176,00006/16/2009PDFTextData Visualization and Sonification
Surface-curve graphical intersection tools and primitives for data visualization, tabular data, and advanced spreadsheets
8,826,11413/673,73209/02/2009PDFText Data Visualization And Sonification
Surface-surface graphical intersection tools and primitives for data visualization, tabular data, and advanced spreadsheets 8,826,11313/670,34609/02/2009PDFText Data Visualization And Sonification
User Interface Metaphor Methods for Multi-Channel Data Sonification
8,692,100
13/846,843
06/16/2009PDFText Data Visualization and Sonification
Interactive Multi-Channel Data Sonification to Accompany Data Visualization with Partitioned Timbre Spaces Using Modulation of Timbre as Sonification Information Carriers8,440,90213/450,35006/16/2009PDFTextData Visualization and Sonification
Multi-Channel Data Sonification in Spatial Sound Fields with Partitioned Timbre Spaces Using Modulation of Timbre and Rendered Spatial Location as Sonification Information Carriers8,309,83313/450,33906/16/2009PDFText Data Visualization and Sonification
Multichannel Data Sonification System with Partitioned Timbre Spaces and Modulation Techniques8,247,67712/817,19606/16/2009PDFText Data Visualization and Sonification

Pending Patents

TitlePublication NumberApplication NumberPriority DatesPDFText OnlyRelated Patents
Sound Synthesis for Data Sonification Employing a Human Auditory Perception Eigenfunction Model in Hilbert Space2018/028641815/997,53907/31/2009PDFTextData Visualization And Sonification
Interactive graphical surface-rendering data visualization tools for tabular data and spreadsheets2017/026242515/601,89209/02/2009PDFTextData Visualization And Sonification
Interactive Data Visualization Utilizing HDTP Touchpad, HDTP Touchscreens, Advanced Multitouch, or Advanced Mice2013/009143712/875,12809/02/2009PDFTextData Visualization And Sonification
Interactive WYSIWYG Control of Mathematical and Statistical Plots and Representational Graphics for Analysis and Data Visualization2012/031750913/357,59509/02/2009PDFTextData Visualization And Sonification

Pending Unpublished Applications

TitleApplication NumberPriority DatesRelated Patents
Multi-Channel Data Sonification System with Partitioned Timbre Spaces Including Periodic Modulation Techniques 16/049,57306/16/2009 Data Visualization And Sonification