Breath as an Embodied Connection for Performer-System Collaborative Interaction

2. Breath as an Embodied Connection for Performer-System Collaborative Interaction Greg Corness TheclaSchiphorst Simon Fraser University

3. Overview • Background + Literature Review • Initial Inspiration • Key Interaction paradigms • The Ariel System • Methodology • Studio Session (Performers improvising with Ariel) • Interview technique • Procedures for Data Analysis • The Three Primary Results • Conclusion + Contribution

4. “The holy grail for me as a director is to produce a dramatic relationship between performer and media, to grant media real agency, casting them in a role on par with the live performers”[Saltz, D. “Live Media: Interactive Technology and Theatre”]

5. Background – Literature Review • Performer- System Interaction: the interaction between a Performer and a Computer System • Performer-Performer Interaction: the interaction between TwoHuman Performers

6. Background (Performer-System) • Traditional ‘Sense- Respond’ paradigm tends to focuses on after-the-event interaction. Rowe, R. (1999). The Aesthetics of Interactive Music Systems. Winkler, Todd. (1995). Making Motion Musical: Gesture Mapping Strategies for Interactive Computer Music.

7. Background (Performer-System) • Traditional Sense- Respond paradigm tends to focuses on After-the-Event interaction. • Multi-modal base interaction considers the system’s ability to track and anticipate the performer by sensing more channels of information. Camurri, Antonio et. al. EyesWeb: Toward Gesture and Affect Recognition in Interactive Dance and Music Systems.

8. Background (Performer-System) • Traditional Sense- Respond paradigm tends to focuses on After-the-Event interaction • Multi-modal base interaction considers the system’s ability to track and anticipate the performer by sensing more channels of information • The performer’s ability to anticipate the system is rarely discussed.

9. Background (Performer-Performer) • Performance practice • Research in neuro-science • Social Cues in Embodied Conversational Agent

10. Background (Performer-Performer) • Performance practice • Agency, Trust, Collaboration, Negotiation • Intuition related to Intention • Research in neuro-science • Social Cues in Embodied Conversational Agent Lockford, L., & Pelias, R. J. (2004). Bodily Poeticizing in Theatrical Improvisation: a Typology of Performative Knowledge. Bogart, A., & Landau, T. (2005). The Viewpoints Book: a Practical Guide to Viewpoints and Composition

11. Background (Performer-Performer) • Performance practice • Research in neuro-science • Mirror neurons • Cognitive latency • Social Cues in Embodied Conversational Agent Gallese, V. et. al. (2007). Intentional Attunement: mirror neurons and the neural underpinnings of interpersonal relations. Wagner, D. M. (2003). The Illusion of Conscious Will

12. Background (Performer-Performer) • Performance practice • Research in neuro-science • Social Cues in Embodied Conversational Agent • Generating facial expressions • Adding Breath Sounds Foster, M. E. (2007). Enhancing Human-Computer Interaction with Embodied Conversational Agents Whalen, D. H. (1995). The Effect of Breath Sounds on the Perception of Synthetic Speech

13. How can the interaction between a movement based performer and an autonomous generative music system use breath to leverage the performer’s sense of intuition?

14. -Performers invited to improvise with the system over several sessions. -System designed to act as an autonomous agent generating music in the improvisational sessions. -System provided two basic conditions 1) presenting a preparation breath before it acts, 2) act without any breath. Research Design

15. The Ariel System • Designed to be responsive • Designed to be Autonomous • Incorporates a model of embodied/social cues for projecting intention.

16. The Ariel System • Designed to be responsive • Designed to be Autonomous • Incorporates a model of embodied/social cues for projecting intention. • Responds to performer’s actions through a Computer Vision Tracking • Responding to motion • Responding to shapes

17. Motion tracking Shape tracking

18. The Ariel System • Designed to be responsive • Designed to be Autonomous • Incorporates a model of embodied/social cues for projecting intention. • Responds to performer’s actions through a Computer Vision Tracking • Responding to motion • Responding to shapes • Combine external and internal influences on the system’s response using weighted random processes

20. The Ariel System • Designed to be responsive • Designed to be Autonomous • Incorporates a model of embodied/social cues for projecting intention. • Responds to performer’s actions through a Computer Vision Tracking • Responding to motion • Responding to shapes • Combine external and internal influences on the system’s response using weighted random processes • Incorporated a simulated breathing behaviour

21. Bite Tail Breath Phrase Loudness Basis for modeling relationship between the Preparation Breath and following gesture

22. The Ariel System The system was designed and tested over three pilot studies. “It feels like ... It feels musical, as in…what feels good...it feels like it makes sense” (with breath) “… it feels understandable ‘you know why there was silence” (with breath) “… it felt like more background to me this time…like …it was more like a track of music playing..” (without breath) “…(it) felt more chaotic” (without breath)

23. Studio Sessions • System switched between presenting/not presenting breath to provided comparison experiences for performers. • Structured improvisations guided performers to avoiding habitual approaches in interacting with a computer system. • Interviewing technique facilitating thick description of experience

24. Interview Technique • Based on Interview of Explicitation (Petitmengin) • Re-living in an “inner way” • Guiding performer’s attention of the experience • Putting into words Petitmengin, C. (2006). Describing one’s subjective experience in the second Person: an Interview Method for the Science of Consciousness.

25. Interview Technique • Re-living in an “inner way” • Metaphor of video to encourage reflection • Focus on selected Sensory Triggers found by performer • Sensitive to Body and verbal cues of Re-living • Guiding performer’s attention of the experience • Putting into words

26. Interview Technique • Re-living in an “inner way” • Guiding performer’s attention of the experience • Stabilizing Attention • Move from representation to experience using how questions • Directing attention to dimensions of experience • Putting into words

27. Interview Technique • Re-living in an “inner way” • Guiding performer’s attention of the experience • Putting into words • Open use of language and gestures

28. Data Analysis • Series of procedures • Close reading • Open coding (developing themes) • Construction of models • synchronic (independent of time) • diachronic (time dependent) • Construction of a General model Petitmengin, C. (2006). Describing one’s subjective experience in the second Person: an Interview Method for the Science of Consciousness.

30. Close Reading - (Perceiving Intention) • The simulated breath is perceived as a cue • The volume of the breath affects the performer’s perception of the system’s intention.

31. Close Reading - (Perceiving Intention) • Perception of the breath as a cue included: • A conscious acknowledgment of the breath as a signal • An embodied understanding of the upcoming gesture “I find it was easier to expect. Although it (the music) was still really fast and sporadic at least you get a sense of ‘ it’s coming…. it’s coming… it’s coming” (system with breath).

32. Close Reading - (Perceiving Intention) • The volume of the breath affected the perception of the system’s intention as unnatural or demanding. “When the breath is loud it sounds like an action being taken, but when it is soft it sounds like a physical cue as if the flute is in the room and getting ready to act.”.   “ … so it was no-longer a command or anything like that, it was like a cue… so it cues you that the music is starting but it doesn’t impede in a way…it doesn’t command you to follow…”  

34. Open Coding – (Quality of Action)

36. “breath was the safest time to move because you know the music is going to start “we were starting to just take silences “it felt wrong to make big motions on complete silence “there was more freedom with no breath…”

37. Open Coding- (Quality of Action)

38. Modelof Expected Reciprocation “… it felt like more background to me this time…like …it was more like a track of music playing..” ( no Breath) “I was surprised at how responsive…it felt like there was somebody…like give and take” (full Breath)

39. Model of Expected Reciprocation “the flute breaths and we say go ahead and sometimes it doesn’t and we are like ‘you let us down’ (with breath) “I found I was responding to multiple things at once so picking up elements of what M and S and the music were doing. I also felt I could pay attention to effort and energy rather then precise movement” (with breath)

40. Micro and Macro-actions • Macro-actions refers to large actions undertaken with conscious intentionality . • micro-actions refer to small actions performed with little awareness or conscious intention • Addresses the the affect of performer’s attention • Perceiving the system as one of the partners • Social and artistic expectation

41. Stages in Performers’ Dynamic knowledge

42. Pre-Knowledge/ Start of Improvisation New Knowledge Reflection Test

43. Stages in Performers’ Dynamic knowledge “…you can anticipate what is coming next-even though it wasn’t patterned.” (with breath) “My body know the length of the gesture by learning the breathing of the system”

50. Conclusion How can the interaction between a movement based performer and an autonomous generative music system use breath to leverage the performer’s sense of intuition?