… the algorithm for the translation of sensor data into music control data is a major artistic area; the definition of these relationships is part of the composition of a piece. Here is where one defines the expression field for the performer, which is of great influence on how the piece will be perceived. —(Waisvisz; 1999)

Artists have been making electronic musical instruments and interactive installations using sensors and computers for several decades now, yet there is still no book available that details the process, approaches and methods of mapping gestures captured by sensors to output media such as sound, light and visuals.

Tools for this kind of mapping keep evolving and a lot of knowledge is embedded in these tools. However, this knowledge is in most cases not documented outside of the implementation itself. So the question comes then how we can preserve knowledge of how a performance or instrument works if tools become obsolete, file formats are not accessible (or documented) and source code is unavailable? How can we learn from what other artists have made before us? How can artists communicate about their approaches to mapping if they are using different tools for doing so?

Just a Question of Mapping

Ins and outs of composing with realtime data

The book aims to give an overview of the process of mapping and common techniques that can be used in this. These methods will be described in a way that gives an artist a guideline of when to use the method and how to implement the method in the environment they work in. Examples of implementations of these methods will be provided seperate from the book in a repository to which readers of the book can contribute.

The book will have various parts:

  • Introduction - framing the book in historical and esthetical context
  • Physical elements - details on sensors, circuits, communication protocols and the physical interface.
  • Computation - everything from range mapping, digital signal processing to machine learning (in broad terms).
  • Process - how to start, build and tune a project involving mapping.
  • Case studies - describes concrete works (instrument, performances and/or installations) based on interviews and in depth study of the implementation with references to the methods described in the first part of the book.

Outline

1. Introduction

  • Arbitrary instruments
  • Views and esthetics
  • The question of mapping

2. Physical elements

  • Elements
    • Sensors
    • Electronic circuits
    • Voltage controlled (modular) systems
    • Microcontrollers
    • Computers
    • Software & computation
    • Output
  • Communication
    • The physical connection
    • Digital protocols used by sensors and actuators
    • Serial protocols
    • HID
    • MIDI
    • OSC
    • DMX
    • Art-Net
    • Wireless
  • Interface
    • Semantics of the interface
    • Effort and ease of use
    • Within and out of reach

3. Computation

  • Looking at data
    • timescale
    • dimensionality
    • range, resolution, linearity and reproducability
  • events and time
  • from one range to another
    • introduction
    • from your input value to a standardized range
    • unipolar and bipolar signals
    • inverting the range
    • from a standardized range to a parameter range
    • nonlinear approaches
    • segmenting the range
    • using a table
  • modes of behaviour
    • crossing a threshold
    • false triggers
    • states and conditions
    • combining events and datastreams
    • modal control
    • picking up where you left off
  • changes, variation and filtering
    • slope
    • realtime statistics
    • moving average
    • deviation from the mean
    • envelope following
  • computational behaviours
  • exploring the mapping space
  • machine learning
  • transitions in the output

4. Process

  • Starting points
    • Imagining the instrument
    • If this were my instrument
    • Sonification (or perceptualisation)
  • Tuning
    • Calibration
    • Dealing with quantisation
    • Dealing with jitter
    • Dealing with loss of signal
    • Order of processing

5. Case Studies

  • Introduction to the case studies
  • STEIM’s softwares Spider (Sensorlab), JunXion, LiSa and RoSa, and The Hands.
  • Andi Otto’s Fello. (augmented instrument)
  • Jeff Carey’s digital instrument consisting of a joystick, a keypad, pads & faders. (digital instrument)
  • Roosna & Flak’s ongoing explorations with dance, accelerometer sensors, sound and light since 2013. (dance)
  • Jaime del Val (participatory movement)
  • (composed instrument)
  • interactive installation

6. Conclusion