e-Science Collides with the Performing Arts

1. e-Science Collides with thePerforming Arts Dr R P Fletcher University of York

2. So what is this e-Science?? • At the inception of the UK e-Science programme John Taylor said that: • “e-Science is about global collaboration in key areas of science and the next generation of infrastructure that will enable it” • Another source says it is: • “science increasingly done through distributed global collaborations enabled by the Internet, using very large data collections, terascale computing resources and high performance visualisation “ • Malcolm Atkinson has said that: • “e-Science is the development and exploitation of advanced informatics methods to support collaborative research in all disciplines “

3. OVERVIEW • Academic Background – am I a real scientist? • Involvement in Performing Arts – am I a fraud? • Involvement with technology – am I a technologist? • Technology • Bespoke hardware and software • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future – some ideas • Questions – whatever you want to throw at me!

4. My Academic background • 1968 Degree in Biology • General • 1971 D.Phil in Neurophysiology • Information processing in single sensory nerve cells • 1975 Research Fellow in Psychology • Risk of schizophrenia • Psychophysiology • 1981 Computing Service • Graphics • Database systems • Programming • Mathematical and statistical systems and libraries

5. Academic Background • Involvement in Performing Arts • My involvement with technology • Technology • Bespoke hardware • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future • Questions

6. My “Performing Arts” background • 1965 Staged school bands • 1968 Staged University bands • 1971 President of the Electronic Arts Society • 1972 Recording engineer York Musicians Cooperative • 1972 Worked with Trevor Wishart • 1973 Recording/mixing engineer campus band • 1974 Built own synthesiser system • 1976 Programmed FX on PDP-11 • 1978 Recording/mixing engineer campus band (again) • 1991 SGI Indigos bootstrap Music Technology • 1996 Chaotic Constructions • 1999 RIMM • 2005 UKLight for Collaborative Composition/Performance • 2006 PEACHES committee of TERENA • 2007 Content Creation tools for Digital Cinema?

7. Technology Input and Output • Specialised software to edit synthetic speech • User interfaces for Music Technology • Courses in graphics for Music Technology • External Examiner MA Digital Arts (Middlesex) • Midi I/O systems for Sun Workstations • Developer for GIMP (video plugins for Unix) • Developer for XMX (collaborative X-Windows) • Developer for .ogg audio formats • Still ambisonic developments • Tracking technology for Chaotic Constructions • Graphics system designer/developer for RIMM • Real-time Interactive Multiple Media • PEACHES committee at TERENA • Panel of Experts for Arts, Culture and Humanities E-Science

8. Academic Background • Involvement in Performing Arts • My involvement with technology • Technology • Bespoke hardware • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future • Questions

9. Technologies • Aim to lead you through some technologies • Ones I have worked on over the last 15 years (or so!) • At some point technology overlaps e-Science • Need to engage with the new technologies • Need to help drive their development • Performing Arts need new/different facilities • either from the existing infgrastructures • or via new facilities • But, unless we know what’s possible • How do we know what to ask for?

10. YORKTALK • Worked with the Language and Linguistics staff • Their technology allowed about six synthesis runs • PER DAY! • Data transferred via Kermit file transfer program! • Real informed trial and error to fix problems • I created a graphical interface to the data • Ran on a then high performance system • Could interact with the data • Synthesis was up to near real time • Combined visualisation to “see” errors • Used interactive graphics to “fix” the problems

11. YORK TALK • Research vehicle for speech synthesis • DecTalk example (may recognise the voice) • YorkTalk examples • Same words as DecTalk • A sentence (with a cold!)

12. Engaging with “artists • Next project was with staff in Music • Needed help to create interfaces for a system • Needed some lateral thinking • Definitely moving into the Arts domain • High performance computing was needed • Multiple tasks by same computer • Video • Audio • Graphics • Midi

13. Chaotic Constructions - 1994 • Certain amount of serendipity • Peter Fluck talking on Radio 4 about Chaotic motion in kinetic scuplture • Tony Myatt talked a few weeks later about Chaos in Music • Phoned each other • classic “WE SHOULD TALK” moment • By 1996 developed a project • to use the chaotic motion of a kinetic sculpture to drive a music composition system • PROBLEM – how to capture the motion • ANSWER – ask me

14. Chaotic Constructions – the Tracking System • Request came to me • How can we do this? • and … • It will be installed in Tate Modern in March 1997 • and … • It was now January 1997! • First thoughts were to use shape tracking systems • Problems: • most systems were proprietary and expensive • shapes to be tracked changed because sculpture was 3 dimensional • could not use trajectory prediction due to the chaotic motion • ANSWER – simply track unique coloured items

15. VIDEO EXAMPLE The Making of CHAOTIC CONSTRUCTIONS

16. Stage 1 – Define a Colour “Fingerprint” • Program creates a set of “unique fingerprints”

17. Tuning the “Fingerprints” • Check motion does not lose sync with colour

18. Peter Fluck Hand Cranking the Sculpture

19. Running with Tracking and Music Program • “Fingerprint” used by second program • Motion tracking is just one element

20. Changing a Tracking Point • What are the best points to track? • Due to chaotic motion some trial and error!

21. And finally …the prototype • … we get there … a small excerpt

22. And the real thing

23. Credits and Awards • Exhibited to great acclaim in Tate Modern • Other UK Art Galleries • Travelled to Hong Kong and Corsica • Herald Angel Award by the Glasgow Herald for the best piece of visual art at the 1997 Edinburgh Festival • International Critics Award as part of the 1997 Demarco European Art Foundation exhibition

24. Not really “Performing Arts” I hear you say • In St. Ives staff gave children coloured “bats” • Switched off the sculpture • Children waved the bats around • And, of course, “made music” • Security was called once to “remove” a visitor • Would not leave at closing time • Transfixed by the whole thing • Some people came 2 or 3 times a week • For a “fix”

25. How can this Technology be used … • … for Performing Arts • Mount a camera above a stage • Give actors coloured hats • Now you have their position on stage • Link their movement around the stage • To … • Orchestrate lighting changes • Hotspots • Move to one place on stage and make a sound • Detect “collisions” • Allow actors to fully interact with their environment

26. …or … • Point the camera at a tropical fish tank • Let the fish make music

27. Bringing e-Science into the Frame • So far York work done on a single modest computer • Scale up • To multiple specialist computers • Networks • Wireless devices • Complex tracking • 3D sound • Interactive graphics • All controlled by … • The PERFORMER

28. Real Time Interactive Multiple Media - RIMM • “Construction 3” • Realised by the Real-time Interactive Multiple Media Project • University of York Music Department (UK), IRCAM (France) and SIMPK (Germany) • European Commission Information Society Technologies Programme • Incorporates a live performer on soprano saxophone • Three-dimensional sculpture • Computer generated graphics • Ambisonic surround sound

29. construction 3 is a synergetic composition • The musical and graphical elements of the piece have been composed concurrently • Both the computer graphics and music are triggered by the saxophonists natural gestures • Through infra-red video tracking • Use i/r device and i/r filter on camera • Electronic sensors • Footswitches, accelerometers • Machine listening • Score following • Gesture recognition using neural networks • e.g. “throw” sound into the audience

30. Block Diagram of System

31. System Diagram

32. The “Team” • Lots of people needed to make this happen • Project manager to keep us in line • The Composer/Sound stage designer • The Graphic Designer • The Graphics Programmer • The Audio Programmer • The Electronics technical wizard • The Local site organiser • The commercial suppliers • And … • THE PERFORMER

33. Team Photo

34. Screens and Performer

35. Examples of the RIMM Computer Graphics

36. Technology becomes e-Science • Clearly a lot of technology to make it work • Some is bespoke electronics • Majority of tasks were programmed • The SGI Origin 2000 • 8 fast processors • Highly parallel • jMax • 24-bit optical output for audio • High speed network interconnect to graphics computer • SGI O2 graphics • Optimised for OpenGL • Texture memory drawn from main memory

37. Academic Background • Involvement in Performing Arts • My involvement with technology • Technology • Bespoke hardware • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future • Questions

38. Technologies for All • Many of these technologies are available • Many of the ideas are now embodied in software • Many are free • Pure Data – aka PD (with GEM add-in) • Eyesweb • VJ-ing is growing • Many VJ products around • Grid Computing is available • Not easily accessible • Lacks some functionality for Performing Arts • Photonic networks are available • High speed links point to point • Up to 10 gb/s possible now • 1 gb/s across the globe

39. jMax, Pure Data/GEM • Graphics Programming • Lots of modules • Connect up inputs and outputs • Uses the modular synthesiser metaphor • GEM • Graphics Environment for Musicians • Link audio to graphics • Limited by modules unless you write your own

40. eyesweb • Video programming with modules • Huge number of modules • Includes “man” tracking capability • Can link to rudimentary audio • But … • Can also link to PD • using Open Sound Control protocols - OSC

41. REACTABLE • Collaborative electronic music instrument • Tabletop tangible multi-touch interface • Several simultaneous performers • Share complete control over the instrument • by moving and rotating physical objects • on a luminous round table surface. • Objects represent components of a classic modular synthesizer • can create complex and dynamic sonic topologies • a kind of tangible modular synthesizer or • graspable flow-controlled programming language

42. REACTABLE

43. REACTABLE - demo Click on Image to go to site with Videos [will open a browser]

44. More Video Tracking and Performance Messa di Voce Tmema,Blonk,La Barbara http://www.tmema.org/messa/messa.html

45. Messa di Voce - demo Click image to go to the Host site [will try to play Quicktime in a browser]

46. And just for fun … another demo

47. Academic Background • Involvement in Performing Arts • My involvement with technology • Technology • Bespoke hardware • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future • Questions

48. So what is this Grid Computing? • The term originated in the early 1990’s • Metaphor for making computer power as easy to access as an electric power grid • Many years on and still mainly in the domain of “BIG SCIENCE” • Most Grids are run like 80’s batch style computing • Submit a job description to a scheduler • It decides when and where the job runs – more or less • This is a computational grid • Many computers and clusters are interconnected • Can be globally distributed via photonic networks

49. What is this Data Grid? • A grid computing system that deals with data • The controlled sharing and management of large amounts of distributed data • These are often, but not always, combined with computational grid computing systems • Finding data is not easy • Accessing it is not easy • Mainly in the domain of “BIG SCIENCE” • Some shift to Arts and Humanities • We have lots of data too! • More later in context of National Grid Service

50. What are these Photonics Networks • A worldwide network of optical fibres • Speeds are 1gbit/s to 40 gbit/s • Run by different organisations • UK - Janet • Europe - GEANT2 • USA - StarLight • USA - National Lambda Rail • Canada – CANARIE • etc … • A connection is usually point-to-point • Very exclusive • Lots of bandwidth • Low latency and jitter (more of this later)