For the electronic music industry, NRI has developed a number of next-generation technologies relating to near-future mass-appeal electronic musical instruments, many of which in one way or another have large-market and market-changing opportunities.
1. ADVANCED KEYBOARDS
This NRI technology extends the functionality and applications of traditional music keyboards:
- Multiple-parameter instrument keyboards combining key-surface touch and key-displacement sensor arrays; for example up to 6 separate controls can be varied interdependently and simultaneously with a single fingertip for each key touched, permitting incredible nuance and powerful control of instrumental timbres, synthesized human-chorus vowels, etc. (US 7,408,108),
- Synthesized tactile feedback may render variable keyboard actions or create `multi-level` key travel functionality,
- Ergonomically positioned miniature keyboards can be attached to instruments such as guitars and sitars so they can be used by available free fingers,
- Stair-case keyboard can be ergonomically positioned so that each hand can access a reasonable number of keys spanning multiple keyboards in chording, melody, and contrapuntal usage.
Individual musical keyboard keys may have individual key displacement sensors measuring key travel. Individual keys may further include surface sensors measuring two additional parameters, creating three independent continuous parameters essential for controlled rendering of vowel sounds and orchestral timbre spaces. Surface sensors may comprise pressure sensor arrays, which, via geometric image-processing, may produce five to six readily controlled independent parameters per key, useful in controlling expressive soloing and long-duration background sounds. Per-key pressure sensor arrays may comprise mass-produced modular components including intelligent distributed image processing to simplify manufacturing and substantially reduce cost.
2. NEXT-GENERATION MUSICAL INSTRUMENT INFRASTRUCTURE
The following NRI technologies provide infrastructure aspects employed as options are partial basis of the Multichannel Vibrating-Element Musical Instruments discussed in Section 3 below, the Modular, Customizable, And Aggregable Musical Instruments discussed in Section 4 below, and aspects of NRI’s.
2.1 Electronically-Driven Vibrating-Element Environments with Signal Processing Feedback Loops
This NRI technology provides electronically-induced drive environments for one or more mechanically-vibrating physical elements, including those environments applicable to musical instrument application. Some key aspects of this NRI technology provide for loop signal processing and the use of variety of types of sense and drive transducers (electromagnetic coil sense/drive, piezo sense/drive, hall-effect sense, optical sense) and various types of vibrating elements (strings, tynes, etc.) in isolated, non-leakage coordinated, and grouped configurations.
In various forums, unrelated parties have associated NRI’s patents with the famous “Moog GuitarTM.” This is an honor, and NRI’s patents in these areas do pre-date the Moog Guitar product by many years and teach and claim many associated features, but NRI’s technology in this area is much more involved and produces more sophisticated effects and degrees of intuitive expressive control than is possible with the Moog Guitar. Other than admiration, NRI has had no formal association with the groundbreaking Moog Guitar instrument.
2.2 String Array Signal Processing for Electronic Musical Instruments
This NRI technology provides approaches for expanding and generalizing string selection aspects of an autoharp, and applying the resulting principles to other types of control, sound synthesis, and more general forms of string arrays. String vibration may be sympathetic or induced by strumming, plucking, and/or electrical drive transducers. Volume of individual vibrating strings within arrays of strings may be mechanically or electronically controlled by chord buttons or a music keyboard. Electronic control can also be made with foot switches or incoming MIDI control signals. These controls may be used individually or in combination. Keyboard keys may be provided with key-travel sensors, key-surface sensors, or other sensors for continuously controlling volumes and octave mixings of subgroups of individual strings. Incoming MIDI control signals may also be used for controlling volumes and octave mixings of subgroups of individual strings. Outgoing control signals, such as MIDI, may be generated responsive to the plucking and/or the amplitude of and harmonic structure in the ongoing vibration of individual strings.
2.3 Generalized Electronic Music Instrument Interfaces
This NRI technology provides a number of approaches to a next-generation generalized signal and power interface for interconnecting a wide range of electronic musical instruments and signal processing systems. These would typically include at a minimum an outgoing multi-channel audio interface and an outgoing control interface. The outgoing multi-channel audio interface receives instrument audio signals generated by an external musical instrument, while the outgoing control interface receives MIDI control signals generated by the same external musical instrument. The outgoing multi-channel audio interface and the outgoing control interface respectively communicate audio signals and MIDI control signals to the external signal processing system. Variations include the addition of multi-channel audio paths to the instrument using drive transducer signals to excite instrument vibrating elements; the use of control paths to the instrument to control on-instrument lighting, signal processing, drive transducers, controller interpretation, etc.; non-MIDI control paths out of the instrument; providing the instrument with expanded power to supporting on-instrument lighting, video devices, and other auxiliary systems; video signals out of the instrument; and video signals to the instrument to support on-instrument video display.
2.4 Touchpad and Touchscreen Control of Music Synthesis, Music Processing, Electronic Percussion, and Lighting
This NRI technology provides a number of approaches to tactile, visual, and array controllers for real-time control of music signal processing, mixing, video, and lighting. Tactile transducers may be put on instrument keys of conventional instruments, be attached to existing instruments, or be used to create entirely new instrument or controller configurations. Chemical sensor arrays and for other numerical image generation from computer processing or numeric simulation can be used to monitor or simulate natural physical phenomenon such as self-organizing process behavior or environmental conditions. Arrays of scalar or vector values are processed to extract pattern boundaries, geometric properties of pixels within the pattern boundaries (geometric center, weighted moments, etc.), and higher-level derived information (senses of rotation, segmented regions, pattern classification, syntax, grammars, sequences, etc.) which are used to create control signals for external audio, visual, and control equipment or algorithms. The invention also provides for MIDI and non-MIDI control signals.
A few historical comments before returning more generally to NRI musical instrument technologies
- Because of the close relation to common touchscreen gestures now pervasive in the consumer electronic industry and NRI’s subsequent patent assets in that area, NRI has transferred the principle patent asset directed to touchpad and touchscreen control of music synthesis, music processing, electronic percussion, and lighting (US 6,570,078) to Advanced Touchscreen and Gesture Technologies, LLC (ATGT).
- More broadly, the underlying NRI technology in this area provides for the real-time controlling of signal processors, synthesizers, musical instruments, MIDI processors, lighting, video, and special effects in performance, recording, and composition environments using images derived from tactile sensors, pressure sensor arrays, optical transducer arrays, chemical sensor arrays, body sensor arrays, and numerical computation. The NRI technology provides for pressure sensor arrays and body sensor arrays as tactile control interfaces, for video cameras and light sensor arrays as optical transducers, for chemical sensor arrays, and for other numerical image generation from computer processing or numeric simulation.
- This NRI technology was a multiple-year precursor to the wearable devices industry (see claims on the recently filed pp.119-120 of US 15/634,084 with 1999 inventorship rights; this patent asset has also been assigned to Advanced Touchscreen and Gesture Technologies, LLC (ATGT).
- With appropriate caveats carved out to the work of Myron Kruger’s “butterfly installation,” this NRI technology also was a multiple-year precursor to the video-camera “free-space gesture” industry (see the early US 8,519,250 and more general-industry US 8,878,807, both with 1999 inventorship rights and also assigned to Advanced Touchscreen and Gesture Technologies, LLC (ATGT). Many years after NRI’s patent filing, “free-space gesture” technology appeared in at least Android smart phones and was more broadly featured on the front cover and feature article of Communications of the ACM Volume 54 Issue 2, February 2011.
- The chemical sensor array aspects of this NRI technology are employed in NRI’s self-organizing performance aspects of Music/Theatre Performance Technologies (specifically US 7,217,878).
3. MULTICHANNEL VIBRATING-ELEMENT MUSICAL INSTRUMENTS
The following NRI technologies describe a number of new electronic multichannel vibrating-element musical instruments.
3.1 Autoharp-based and String-Array Electronic Musical Instruments
This NRI technology provides extensions and generalizations of an autoharp, and further makes these functionalities beneficially available on other instruments. It should be noted that NRI used and adapted strumpad tactile switch arrays developed and manufactured by Suzuki Music and has been long-featured in their early, innovative, and inspiring OminchordTM and QChordTM products.
In this NRI technology, more general tactile strumpads comprising arrays of isolated touch switches, each generating distinct signals, are used to generate MIDI note events according to interpretations of an active mapping selected from a plurality of programmable mappings. The active mapping may be selected via chord buttons or foot switches, or may be determined by keys on a melodic keyboard. Mappings in some embodiments may provide repeated notes, non-note signals, and utilize one or more MIDI channels for controlling synthesizers, lighting, electronic percussion, and the like. The strumpad may further include orienting tactile markings and provide pressure and velocity sensing. The system may be attached to or integrated within another instrument such as a guitar, keyboard, autoharp, pedal steel guitar, sitar, koto, mbria, and pipa, among others. For guitar applications, a strumpad may be attached to or integrated into a guitar pickguard.
3.2 Extensions and generalizations of the pedal steel guitar
This NRI technology provides extensions and generalizations to the pedal steel guitar. Separate transducers provided to each string permit fixed or variable pitch and timbre modifications determined by stored program control. Variable pitch and timbre modifications may be controlled by physical controllers such as single or multi-parameter foot pedals, knee levers, and wrist controllers. Physical controllers may return to an original position or hold their position after operation. A traditional mechanical tuning changer operated by traditional foot pedals and knee levers may be included; the latter may operate physical controllers. The bar position may be sensed, and the bar may include physical controllers wirelessly linked to the instrument. A miniature keyboard and strumpad may be provided in the picking area to control synthesizers and produce MIDI output. Strumpad operation can be controlled with chord buttons, foot pedals, or other physical controller devices. Physical controllers, string vibration, and other types of control devices may also be used to generate MIDI output. This NRI technology also can provide multichannel separate-string (electromagnetic coil) driven vibration employing the NRI technologies described in section 2.1 above; highly simplified version of such arrangements is provided in the 2010 Moog Lap SteelTM instrument and also described in an unrelated 2015 academic paper Feedback Lapsteel: Exploring Tactile Transducers as String Actuators. NRI has had no formal association with Moog or this instrument.
3.3 Transcending Extensions of Classical South Asian Musical Instruments
South Asia’s unique musical instruments and raag systems have provided many rich treasures to Western music. This NRI technology melds Western-music technologies with the innovations, timbre, playing techniques, and performance environments of South Asian musical instruments and traditions. Sitars, Dilrubas, and Esraj may be provided with multi-channel signal processing and pitch change for individual strings, additional playing strings, keyboards, strumpads, percussion interfaces, controls for timbre and stage environment control, synthesizer interfaces, pickups for brass strings, resonant and twang processors, and spatial sound distribution. Bows may include sensors measuring the bow’s physical activity or direct finger manipulation. Tabla and baya may be replaced with touch pads capable of recognizing relevant hand contact modalities and positions. Touch pads may be comprised of pressure sensor arrays that recognize traditional bols and permit other generalizations and deeper levels of sonic control. Sounds may be synthesized or processed from acoustic instrument sample playback.
3.4 Transcending Extensions of Traditional East Asian Musical Instruments
East Asia’s rich unique musical instruments and traditions have loosing favor to Western music and musical instruments. This NRI technology seeks ways to reverse this trend by melding cutting-edge Western-music technologies with the noble innovations, timbre, playing techniques, and performance environments of East Asian musical instruments and traditions. East Asian stringed instruments may be fitted with separate pickups for each string for amplification, music synthesizer interfacing, separate string electronic pitch shifting, exotic signal processing, etc. Additional devices and structures, such as strain gauges, electronic strumming pads, miniature keyboards, harp, bass, or sympathetic strings, percussion impact sensors, and electronic music control switches, sliders, and buttons may be included. Video cameras may be attached to an instrument, and may be used as an instrument itself in conjunction with hand gestures. Any of the above may be used to control lighting effects as well as sound electronics. Examples include the Japanese Koto, Chinese Sheng, Vietnamese Koto, Chinese Pipa, Japanese Biwa, and other similar instruments.
4. MODULAR, CUSTOMIZABLE, AND AGGREGABLE MUSICAL INSTRUMENTS
NRI has developed modular and infrastructure technologies for creating user-customizable electronic musical instruments, particularly ones that facilitate synergistic aggregations of multiple instruments and functions. One or more types of mounting frames, which can be set on a floor-stand or worn with a shoulder strap, can be used to interchangeably secure, power, and exchange signals among near-arbitrary combinations of easily user-installable electronic instrument modules, processing modules, controlling modules, synthesis modules, visual effects modules, sound production modules, and outside-interface modules.
- Instrument modules can include vibrating elements (strings, tynes, etc.) configured to create electronic signals responsive to vibration; such instrument modules can also include provisions for vibrating elements to be electronically-induced into vibration and submodular provisions for various types of fretted or unfretted necks, pickup transducers, etc.;
- Processing modules can include audio signal processors, audio mixer, audio sound production elements, control signal processors, control signal merging, and controllable audio signal synthesizers;
- Controlling modules can include a wide variety of controller elements such as a keyboard, touchpad, strumpad, impact sensor, slider control, expression wheel, joystick, ribbon controller, button, switch, pressure sensor, multiple-position selector, knob potentiometer;
- Synthesis modules can include digital or analog music synthesizer elements;
- Visual effect modules can include lighting, image display, and video cameras;
- Sound production modules can include provisions for internal (mono, stereo, multichannel) sound amplification;
- Outside-interface modules can be configured to provide multichannel outputs and one or more signal formats.
Signals exchanged among the modules can include audio, MIDI, analog control, digital control, data, and video signals. Audio signals can be mixed by a multichannel audio mixer and/or processed responsive to control signals. Control, audio, video, and data signals may be routed among modules by a switch or bus which can be configured responsive to control signals. The resulting user-customizable electronic musical instruments can transmit outgoing audio electrical signals to external audio systems, and transmit outgoing control signals to external audio, processing, mixing, lighting, video, recording, and other controllable equipment.
The NRI technology can be used to assemble large string array electronic musical instruments such as various types of electric harps, zithers, sympathetic string arrays, etc., and can for example be used to implement electronic versions of Harry Partch instruments such as the ‘Kithara’ and ‘Harmonic Cannon.’ Similarly The NRI technology can be used to assemble large percussion (impact sensor) electronic musical instruments such as Asian-tradition xylophones/marimbas and electronic versions of Harry Partch instruments such as the ‘Boo’ and ‘Quadrangularis Reversum’ (with the electronic aspects readily facilitating scalar microtonal tuning, and combinations of mounting and tuning flexibilities permitting two-dimensional array layouts matching just-intonation arrays such as Max Friedrich Meyer tonality diamonds.
More broadly, this NRI technology allows for easy field-customization of mainstream and exotic electronic musical instruments with extensive support for the easy creation of aggregated instruments. The technology leverages extensive functional customization of instruments within mainstream accepted instrument modalities as well as opening a wide range of completely new instrument modalities. Importantly, the technology further facilitates entirely new manufacturing, marketing, and sales paradigms permitting a broad range of open industry development and commerce, thus making an individual musician’s creation of new exotic instrument arrangements an economically viable sector for both mass manufacturing and the niche cottage industry. New opportunities are provided for the creation of multiple-vendor standardizations, multiple-vendor manufacturing, multiple-vendor competitive features, etc. while offering the music equipment user and music industry as a whole, access to an extensive range of instrument customization, diversification, and education. The principles of this NRI technology thus create a rich environment for instrument, user, feature, music, and market.
Issued Patents
| Title | Patent Number | Application Number | Priority Dates | Text Only | Related Patents | |
|---|---|---|---|---|---|---|
| Customizable Wearable Electronic Musical Instruments Having User-Installable Controller Modules and Synthesis Modules | 10,276,138 | 15/806,578 | 12/15/2003 | Text | Music Instruments | |
| Modular approach to large string array electronic musical instruments such as specialized harps, zithers, sympathetic string arrays, partch kithara and harmonic cannon | 9,824,672 | 14/144,000 | 12/20/2013 | Text | Music Instruments | |
| Customized Electronic Musical Instrument and User Interface | 8,716,585 | 13/662,403 | 12/15/2003 | Text | Music Instruments |
|
| Signal Distribution within Modular Structures Facilitating Aggregated and Field-Customized Musical Instruments | 8,309,835 | 12/786,438 | 12/15/2003 | Text | Music Instruments | |
| Generalized Electronic Music Interface | 8,030,567 | 10/680,591 | 05/15/1999 | Text | Music Instruments | |
| String Array Processing for Electronic Musical Instruments | 7,767,902 | 11/219,475 | 05/15/1999 | Text | Music Instruments | |
| Transcending Extensions of Classical South Asian Musical Instruments | 7,759,571 | 10/688,743 | 05/15/1999 | Text | Music Instruments | |
| Modular Structures Facilitating Aggregated And Field-Customized Musical Instruments | 7,732,702 | 10/737,043 | 12/15/2003 | Text | Music Instruments | |
| Transcending Extensions of Traditional East Asian Musical Instruments | 7,507,902 | 10/701,683 | 05/15/1999 | Text | Music Instruments | |
| Modular Structures For Interchangeable Musical Instrument Neck Arrangements | 7,417,185 | 10/737,095 | 12/15/2003 | Text | Music Instruments | |
| Strumpad and String Array Processing for Musical Instruments | 7,038,123 | 10/677,032 | 05/15/1999 | Text | Music Instruments | |
| Extensions and Generalizations of the Pedal Steel Guitar | 6,852,919 | 10/676,910 | 05/15/1999 | Text | Music Instruments | |
| Activity Indication, External Source, and Processing Loop Provisions for Driven Vibrating-Element Environments | 6,610,917 | 09/313,533 | 05/15/1998 | Text | Music Instruments |
Pending Published Applications
| Title | Publication Number | Application Number | Priority Dates | Publish Date | Text Only | Related Patents | |
|---|---|---|---|---|---|---|---|
| Electronic Musical Instrument Controller Merging Responses from a Plurality of Strumpads | 2019/0259359 | 16/397,892 | 12/15/2003 | Text | Music Instruments |
Pending Unpublished Applications
| Title | Application Number | Priority Dates | Related Patents |