Sensemble: A Wireless, Compact, Multi-User Sensor System for Interactive Dance

1. Sensemble: A Wireless, Compact, Multi-User Sensor System for Interactive Dance Ryan Aylward Joseph Paradiso MIT Media Laboratory International Conference on New Interfaces for Musical Expression June 5, 2006

2. Technical Goals • Capture expressive movements in very high detail • Distribute points of measurement to multiple people and multiple locations on the body • Interactive music control - demands real-time data collection and analysis for entire group • Typically must collect and fuse over 100 data streams into 10-20 parameters relevant to composer and/or choreographer with acceptable latency (< 100ms) • Limit the constraints on performance setting • Limit physical impact on dancers

3. Previous Approaches • Video motion capture and computer vision • Large scale with Vicon, etc. • Small scale with EyesWeb, Jitter, etc. • Shoe-based sensors • Taptronics • Expressive Footwear (MIT Media Lab, 1997) • Nike+ system • Full body wireless sensor systems • Flex sensors • Resistive • Fiberoptic, e.g. Measurand ShapeTape • Inertial sensors (Accelerometers and Rate Gyroscopes)

4. Limitations • Video motion capture and computer vision • Expensive, large infrastructure, constrained environment • Problems of occlusion with multiple users • Changing light conditions • Limitations on frame rate • Cheap webcam systems compromise accuracy and must work harder to address these limitations • Shoe-based sensors • Typically designed for the bandwidth requirements of one user • Only two measurement points on the body • Full body wireless sensor systems • Typically wired across body to central radio pack • This infrastructure can be cumbersome to scale to group

5. Our Approach • System of wireless nodes each equipped with full 3-axis acceleration and rate gyro sensing • Compact devices designed to be strapped to the wrists and ankles • Each node has its own low-power, high-bandwidth radio, eliminating wires across the body • Many other body sensor designs use standard RF protocols that are either too power hungry for our distributed design (WiFi, Bluetooth), or lack the bandwidth we require (Zigbee) • A simple custom time division protocol allows us to push the limits of bandwidth, allowing fine measurement across ensemble in real-time

6. Hardware Overview • 3 axes of accelerometer (ADXL210) • 3 axes of rate gyroscope (ADXRS300) • Capacitive node-to-node proximity sensing • Node measures 4x4x2cm, size of large wristwatch • Lightweight - 45g including small LiPo battery providing 4hrs of charge • 1Mbps data radio (Nordic 2401A) • Run 25 nodes with 100Hz full updates from a single basestation • Reliable range about 15m

7. Typical Raw Inertial Sensor Output • Sampled at 100Hz • Accelerometers have a full range of ±10g • Gyros tuned to approximate range of ±1200 deg/sec • Sensors accurate to 10 bits

8. Capacitive Sensing System • Capacitive system measures relative spacing between nodes • Employs a “transmit mode” configuration • One node transmits a sinusoidal pulse at 90kHz, others measure amplitude of received pulse • Nodes trade roles as transmitters and receivers as arbitrated by the wireless basestation • Sensitive up to 50cm with bracelet-sized electrode and shared ground through the body

9. Capacitive Sensing In Action

10. Considerations for Feature Extraction in Group Settings • Measuring Group Parameters • Bulk features (net energy / jerk, tempo, hands versus feet, etc.) • Leader versus follower • Cooperating or not cooperating • Similarity of gestures among participants • Predominant motions across ensemble • Data Reduction • Form clusters based on observed patterns • Single out unique events • Focus attention only on predominant patterns, majority rule • Apply heavier analysis techniques on the reduced parameters • Ultimately reduce to handful of features

11. Cross-covariance • Given two signals of length N from different sources, a cross-covariance function of length M=2N-1 can be computed as: • Use of cross-covariance instead of cross-correlation eliminates bias present in raw data • Average cross-covariance implies that the calculation above has been made for each sensor axis and the results have been combined through averaging

12. Describing Similar Motions with Cross-covariance • Below: data from the wrists of three subjects raising and lowering hands in sequence • Location of cross-covariance peaks correspond to lag times between the gestures relative to subject one • Height of cross-covariance peaks describe the similarity of the gestures relative to subject one Subject 1 (raw pitch gyro signal) Subject 2 (raw pitch gyro signal) Subject 3 (raw pitch gyro signal) Average XCOV for subjects relative to Subject 1

13. Dancer A with respect to Dancer B Dancer A with respect to Dancer C Lag Time (Seconds) Dancer B with respect to Dancer C Other Activity Right Leg Swing Sequence Other Activity Time Elapsed (Seconds) Correlated Activity Among Dancers • Running average cross-covariance for the right ankles of three dancers performing a series of synchronous leg swings • Step size 250 ms, window size 1 sec

14. Quantifying Activity Levels • Dancer transitions between slow kicks and fast tense kicks • Arm motions frame the sequence of gestures • Variance envelope calculated with 100ms window and smoothing filter • Changing activity profile, upper versus lower body movement Normalized Sensor Values 2

15. New Developments • Full deployment at a rehearsal with 20 sensor nodes running on five dancers • Feature extraction and mapping algorithms which are fed logged data at the sample rate to simulate real time operation • Audio rendered in real-time as features are calculated • Processing, mapping, and MIDI control tested in Max/MSP • Sound generation tested in Reason • Runs on a 1.6 GHz G5 with 1GB RAM

16. Rough Demo Mapping Direct Features Intuitive Features Musical Parameters

17. Demo Excerpts

18. Future Directions • With a full system running in real time, dancers will be able to explore and learn, leading to a performance exploring collaborative interface • Other applications in personal and professional athletic training, physical therapy • Eventually, smaller, faster, cheaper, more transparent to user • Already easily 2x smaller • 1 mm3 (minus power, antenna) in a few years • Becomes integrated into clothing

19. Conclusions • The system has been successful in generating several collective activity features relevant to dance • We have built a network large enough to instrument arms and legs of a small dance ensemble and have implemented real-time data collection • Further, the analysis, interpretation, and effective realization of sensor data as sound can be accomplished in real-time and with reasonable processing power • The use of group features and the ability to distribute measurement over multiple people creates a collaborative interface with intriguing possibilities for performance art