Gesture Control of Spatialization

Participants: Mark T. Marshall
Joseph Malloch
Marcelo M. Wanderley

A visualization of sound source in the ViMiC spatialization system.
Collaborators: Nils Peters (RTML)
Georgios Marentakis (RTML)
Stephen McAdams (RTML)
Sean Ferguson (Composition Area)
Jonas Braasch (RPI)
Funding: NSERC New Media Initiative and Canada Council for the Arts grants
Project Type: Collaborative Project.
Time Period: Sept. 2005–Sept. 2008. (Completed.)

Project Description

This project aims to develop novel compositional and technological methods for the advanced use of the multidimensional nature of auditory space in music composition. Specifically, the project involves the examination, design and construction of novel methods for gesture control of sound source position, sound source parameters and room model parameters within a sound spatialization system.

Research being performed within this project includes:

  • Motion capture of performer gestures to determine which aspects of the gestures may be used to control spatialization
  • Development of novel gestural controllers for manipulation of sound source and room model parameters
  • Design of systems to unobtrusively capture 'accompanist gestures' which performers make during performance
  • Evaluation of existing controllers as tools for gesture controlled spatialization
  • Examination of novel methods and metaphors for manipulating sound sources in a spatialization system

Other aspects of the project which are being developed by our research partners include:

  • The design and development of a 24-channel audio virtual environment (AVE)
  • Examination of the perception of spatial audio
  • Compositional aspects and uses of gesture-controlled spatialization

Along with the research aspects of this project, there is a significant musical component. A number of pieces are being written, including works both for solo cello and also for a larger ensemble. The first of these performances took place as part of the 4th International Conference on Enactive Interfaces, and will be followed by larger performances in Montreal, QC, Canada as part of the MusiMarch08 event and again later in 2008 in BC, Canada.

Further information is available from the project wiki.