… the algorithm for the translation of sensor data into music control data is a major artistic area; the definition of these relationships is part of the composition of a piece. Here is where one defines the expression field for the performer, which is of great influence on how the piece will be perceived. —(Waisvisz; 1999)

Artists have been making electronic musical instruments and interactive installations using sensors and computers for several decades now, yet there is still no book available that details the process, approaches and methods of mapping gestures captured by sensors to output media such as sound, light and visuals.

Tools for this kind of mapping keep evolving and a lot of knowledge is embedded in these tools. However, this knowledge is in most cases not documented outside of the implementation itself. So the question comes then how we can preserve knowledge of how a performance or instrument works if tools become obsolete, file formats are not accessible (or documented) and source code is unavailable? How can we learn from what other artists have made before us? How can artists communicate about their approaches to mapping if they are using different tools for doing so?

Just a Question of Mapping

Ins and outs of composing with realtime data

The book aims to give an overview of the process of mapping and common techniques that can be used in this. These methods will be described in a way that gives an artist a guideline of when to use the method and how to implement the method in the environment they work in. Examples of implementations of these methods will be provided seperate from the book in a repository to which readers of the book can contribute.

The book will have various parts:

  • Introduction - framing the book in historical and esthetical context
  • Case studies - describes concrete works (instrument, performances and/or installations) based on interviews and in depth study of the implementation with references to the methods described later in the book.
  • Process - how to start, build and develop a project involving mapping.
  • Physicality - details on sensors, circuits, communication protocols and the physical interface.
  • Understanding data - ways of understanding and looking at data, and characteristics of data.
  • Processing data - everything from range mapping, filtering, buttons, modes, output parameter space
  • Behaviours of data - advanced methods, e.g. machine learning (in broad terms).
  • Tuning data - calibration, dealing with noise, tuning parameters.
  • Conclusion and outlook

Outline

1. Introduction

  • Introduction
  • Mapping aesthetics
    • Music
    • Dance
    • Media art
  • The question of mapping
    • Describing mappings
    • Following the flow
    • Artist’s choices
    • The context and process

2. Case Studies

  • Introduction to the case studies
  • STEIM’s softwares Spider (Sensorlab), JunXion, LiSa and RoSa, and The Hands.
  • Andi Otto’s Fello. (augmented instrument)
  • Jeff Carey’s ctrlKey, a digital instrument consisting of a joystick, a keypad, pads & faders. (digital instrument)
  • Cathy van Eck (composed instrument)
  • Roosna & Flak’s ongoing explorations with dance, accelerometer sensors, sound and light since 2013. (dance)
  • Jaime del Val (participatory movement)
  • Sonia Cillari (interactive installation)

3. Process

  • The project
    • Defining the project
    • Different roles in interaction
    • Music
    • Installation (media art)
    • Dance
  • Starting points
    • Imagining the instrument
    • If this were my instrument
    • Sonification (or perceptualisation)
  • Development process
    • Concept to sketch
    • Prototype and test
    • Consolidate

4. Physicality

  • Introduction to physicality
  • Input elements
    • Custom sensor interfaces
      • Sensors
      • Electronic circuits
      • Microcontrollers
    • Off-the-shelf controllers
    • Microphone as sensor
    • Camera as sensor
    • Choosing inputs
  • Elements for processing
    • Computers
    • Software & computation
  • Output elements
    • Sound
    • Light
    • Video
    • Haptics
    • Mechatronics
  • Connections, communication and protocols
    • The physical connection
    • Digital protocols used by sensors and actuators
    • Serial protocols
    • HID
    • MIDI
    • OSC
    • DMX
    • Art-Net
    • Wireless
  • Interface
    • Semantics of the interface
    • Effort and ease of use
    • Within and out of reach

5. Understanding data

  • Contintuous data streams and events
    • Interpretation
    • Transmutation: from one type to another
    • Continuous data streams within events
    • Events within events
  • Characteristics of data
    • Timescale
    • Range
    • Resoluton of the amplitude
    • Linearity
    • Repeatability and reproducability
    • Dimensionality
  • Exploring data
    • Seeing the data
    • Analysing data
    • Improving the data quality and optimisation
    • Informing aesthetic choices

6. Processing data

  • From one range to another
    • From your input value to a standardized range
    • Unipolar and bipolar signals
    • Inverting the range
    • Out of range
    • From a standardized range to a parameter range
    • Nonlinear approaches
    • Segmenting the range
    • Splitting the range
    • Stages of range mapping
  • Filtering and deriving data
    • Smoothing
    • Changes
    • Rectification
    • Ranges and thresholds
    • Minimum, maximum, bandwidth
    • Root mean square
    • Spectral: frequency or rate of change
    • Onset density
  • Buttons
    • Buttons, keys and switches
    • On and off, down and up
    • Trigger
    • Toggle
    • Counting button presses
    • Combinations
    • Order of keypresses
    • Timing
    • Glitches: debouncing
  • States and modes of behaviour
    • States and conditions
    • Combining events and datastreams
    • Modal control
    • Picking up where you left off
  • Mixing and splitting signals
    • Multiple inputs and multiple outputs: matrix approach
    • Mixing inputs to control one output parameter
    • Splitting signals: generating multiple output parameters from one input signal
    • Preset interpolation
    • Modal control
  • Output parameter space
  • Order of processing

7. Behaviours of data

  • Time and memory
  • Ccomputational behaviour
  • Machine learning
  • Exploration

8. Tuning data

  • Calibration
  • Sampling and quantisation
  • Dealing with disturbances
  • Tuning

9. Conclusion and outlook