Graphical Interfaces for Mapping DMIs

Participants: Joseph Malloch
Stephen Sinclair
Vijay Rudraraju
Aaron Krajeski
Jonathan Wilansky
Marcelo M. Wanderley (supervisor)
Graphical interface for configuring DMI mapping connections.
Funding: NSERC Discovery Grant (M. Wanderley)
Project Type:

Time Period: 2007–present (ongoing)

Project Description

During a variety of projects we have found that the “mapping” task – in which correspondences are designed between sensor/gesture signals and the control parameters of sound synthesizers – is by far the most challenging aspect of digital musical instrument design, especially when attempted in collaborative settings. We have developed tools for supporting this task, including the Digital Orchestra Toolbox for MaxMSP and the software library libmapper. The latter project enables the creation of a network of distributed “devices” which may be sources of real-time control data (instruments) and/or destinations for control data (e.g. sound synthesizers). The software library handles device discovery, stream translation (e.g. type coercion, padding) and network transportation, but does not attempt to create mappings automatically. Instead, the mapping designer(s) command the library to create connections between signals, usually using a graphical user interface to interact with the mapping network. To date, GUIs for libmapper have been implemented in MaxMSP, Javascript/HTML5, and Python/wxWidgets.

The primary view used in our mapping GUIs is based on the common structure of diagrams used to describe DMI mapping in the literature – a bipartite graph representation of the connections, in which sources of data appear on the left-hand side of the visualization and destinations or sinks for data appear on the right. Lines representing inter-device links and inter-signal connections may be drawn between the entities on each side, and properties are set by first selecting the connection(s) to work on and then setting properties in a separate “edit bar”. They use a multi-tab interface in which the leftmost tab always displays the network overview (links between devices) and subsequent tabs provide sub-graph representations of the connections belonging to a specific linked device.

We have also explored alternative visualization and interaction techniques, which allow more informed and flexible interaction with the mapping network. Crucially, we believe that there is no need for a single “correct” user interface; rather, different network representations and interaction approaches may be useful to different users, for different mapping tasks, or at different times.

All libmapper GUIs function as “dumb terminals”—no handling of mapping connection commands takes place in the GUI, but rather they are only responsible for representing the current state of the network links and connections, and issuing commands on behalf of the user. This means that an arbitrary number of GUIs can be open simultaneously supporting both remote network management and collaborative creation and editing during the mapping task. This approach has brought our attention to interesting research into other collaboratively-edited systems [Ressel and Gunzenhäuser 1999]; our approach to undo/redo functionality is based on this work.

Alternative Visualization Techniques

Although the spreadsheet-like interface shown above has been used successfully in a number of individual and collaborative projects, there are scenarios in which alternative layouts and interaction techniques may be useful. Since signal names are used to distinguish mapping sources and destinations, this approach lends itself well to mapping scenarios which involve small numbers of devices with heterogeneous parameters-spaces; a scenario which requires mapping of a large number of identical devices (e.g. wireless motes) may confuse the user, since the GUI display would be full of entries like this:

  • /mote.1/acceleration
  • /mote.2/acceleration
  • /mote.3/acceleration
  • etc.

The user would likely be better served by a GUI that displayed devices based on their physical location, or the performer’s name if they are worn or hand-held, or perhaps their activity over some specified period of time. To this end, we began exploring alternatives for the network visualization, with the philosophy that no one approach would be appropriate for all users or at all times. Instead, the GUI should allow the user to choose from different views of the mapping network, and even customize and create new views. The data driving the GUI view is supplied by libmapper, which already supports tagging of all entities (devices, signals, connections) with extensible metadata. Prototype versions of the GUI have enabled users to both sort entities using tagged metadata, and to apply new tags.

GUI functionality:

  • browsing active devices and their signals
  • drag-and-drop mapping connections
  • mode and function editor for connections
  • collaborative undo/redo
  • filtering parameter lists by OSC prefix or string-matching
  • saving and loading mapping sets (including consideration of mapping transportability per the GDIF project)
  • multiple “views” of the mapping network, using different visualization techniques

  • Joseph Malloch, Stephen Sinclair, and Marcelo M. Wanderley. “Distributed tools for interactive design of heterogeneous signal networks”. Multimedia Tools and Applications, 73(2), 2014.
  • Joseph Malloch. “A Framework and Tools for Mapping of Digital Musical Instruments”, Ph.D. thesis, Music Technology, McGill University, Montréal. Thesis defended on November 4, 2013.
  • Aaron Krajeski. “A Flexible Tool for the Visualization of Musical Mapping Networks.” M.A. thesis, November 2013.
  • Jonathan Wilansky. “A Production Tool for Analysis of the Mapping Layer in Digital Musical Instruments.” M.A. thesis, November 2013.
  • Vijay Rudraraju. “A Tool for Configuring Mappings for Musical Systems using Wireless Sensor Networks”, M.A. thesis. December 2011.
  • Joseph Malloch, Stephen Sinclair, and Marcelo M. Wanderley. “A network-based framework for collaborative development and performance of digital musical instruments”. In R. Kronland-Martinet, S. Ystad, and K. Jensen (Eds.): CMMR 2007, – Proc. of Computer Music Modeling and Retrieval 2007 Conference, LNCS 4969. Berlin Heidelberg: Springer-Verlag, pp. 401–425, 2008.
  • Joseph Malloch, Stephen Sinclair, and Marcelo M. Wanderley. “From controller to sound: Tools for collaborative development of digital musical instruments”. In Proceedings of the 2007 International Computer Music Conference, Copenhagen, Denmark, pp. 65–72, 2007.
  • Joseph Malloch. “A Consort of Gestural Musical Controllers: Design, Construction, and Performance”. M.A. thesis, Music Technology, McGill University, Montréal. August 2007.

More Information