Piano touch, timbre, ecological psychology, and cross-modal interference

1. Piano touch, timbre, ecological psychology, and cross-modal interference Richard Parncutt Centre for Systematic Musicology University of Graz, Austria ISPS, 28-31 August 2013, Vienna International Symposium on Performance Science

2. Abstract The piano has a wide timbral range, and performance quality is often judged in timbral terms. Yet despite decades of research, there are still fundamental disagreements about the nature and origin of piano touch. Scientists (acousticians) maintain that the timbre of a single tone cannot be varied independently of its loudness. Performers, humanities scholars and concert audiences take the opposite for granted: timbre and loudness can be independently varied by gestural means. Both sides are right, but their implicit definitions of timbre differ, and both fail to clearly distinguish between physical measures and descriptions of subjective experience. Scientists assume that timbre depends only on physical sound parameters; but experiential parameters generally depend on concurrent input from other senses, the listener’s relevant knowledge and expectations, and immediately preceding and following events. The paradox of timbre disappears if we accept, based on empirical evidence, that timbre generally depends on input from more than one sensory modality (weak synesthesia). Embodied corporality and conceptual metaphors are the norm - not the exception. Gestural and ecological approaches to timbre perception pose existential challenges to disembodied cognitive orientations.

3. Timbre in traditional psychoacoustics American Standards Association (1960): “thatattributeofsensation in termsofwhich a listenercanjudgethattwosoundshavingthe same loudnessandpitcharedissimilar” A negative definition! • Neitherpitchnorloudness • The multidimensional remainder

4. Timbre isexperience! • Traditional psychoacoustics: cleardistinctionbetweenphysicalandexperiential variables • But manypsychoacousticianstalkabouttimbreasifitwerephysical • I assumethattimbreisalwaysexperiential

5. Timbre depends on Time domain • temporal envelope Frequencydomain • spectral envelope

6. Vocaltimbre Time Domain: Articulation Temporal envelope Frequency Domain Vocal Tract Shape Spectral Envelope

7. Timbre of musical instruments • Mixture of time and frequencydomain • Spectral flux: Different temporal envelopes of partials • Absolutefrequency • spectral & temporal envelopesdepend on fo • timbre learned for eachabsolutefrequency • Spectral interval structure • Clarinet, bowedviolin: exactlyharmonic • Bell: inharmonic • Piano, guitar: stretchedharmonic Spectral flux in a bass clarinet tone Grey & Moorer (1977) McAdams (1993)

8. TimbralspaceSpatial representation of timbralrelationships Method: • Choosesome timbres • Rate similarity of all pairs • Multidimensionalscaling • Resultisusually 3-D, axes: • onset, centroid, roughess… • depending on testedsounds Interpretation: • Resultsdifficult to generalise • Cognitive representation? • Distances in spacedepend on exposure & familiarity McAdams et al. (1995), Psychological Research

9. Ecologicalpsychologyoftimbre • Environmental interaction • Experience as byproduct of interaction (epiphenomenon) Theory of James J. Gibson: • Affordances and information pick-up • Invariances of sound sources • Perceptuallearning Complementary to the cognitive approach Relationship to evolutionarypsychology

10. Distal versus proximal stimuliExternal versus internalfocusofattention In sport and music, performance may be better if attention directed to effects of movements - not movements themselves (Wulf& Prinz, 2001): • Golf: attention on goal, not movements • Pianists: attention to sound, not technique Why? • Sophisticated unconscious motor control mechanisms • Perception of goal inseparable from proprioception Implication Perception of timbre involves source perception

11. Ecological-evolutionary foundationsof auditory perception Assumption: Sounds are only interesting( consciously perceived) if they carry information about environmental interaction that could affect survival or reproduction. The affordances of a sound source • what it means to us • what we can do with it  Timbre reflects affordances! • Human voice • Lion footsteps, waterfall, car, gun…

12. Words todescribethe jazz voice pleasant easy resonant black LOOSE SMOKEY dark forward BRIGHT FORCEFUL SPOKEN round OPEN NASAL CLASSICAL HARSH swinging AIRY BREATHY CLEAR VIBRATO soft FIXED yawning NATURAL PRESSED relaxed held back support (Prem & Parncutt, 2007)

13. “I fell in love with his voice. He is African American, and his voice has this depth, it is souly, in a wonderful way. He is like a gentleman, and has this warmth. He gives his voice space and that arouses a very balanced feeling in me. His sadness, his elegance, his joyfulness really come through, so that I can experiencethem.” Affordancesofvocaltimbre Data collectedby Ella Prem

14. Timbre is multimodal • Performer can’t separate timbre (as experience!) from gestures used to achieve it • Listeners share the performer’s experience (empathy) How? • audiovisual mirror neurons (Kohler et al., 2002) • auditory-motor interactions (Zatorre et al., 2007)

15. Proprioception Perceptionofpositionandmovementofbodypartsandassociatedmusculartension Part ofthesomaticsense touch, proprioception, temperature, pain An aspectofcorporality Aspectsofmusicexperiencethatdepend on thebodiesofperformersandlisteners

16. Weaksynesthesiacross-modal interference • All perceptual modalities: seeing, hearing tasting… • The norm - not the exception • Ecologically natural - cognitively strange The limitsofsubjectiveanalysis We cannot completely separate… • input from different modalities: hearing, vision, touch • sensations within a modality (pitch, timbre).

17. A non-musical example When you taste yoghurt with a lighter (table-) spoon, it seems • thicker • sweeter • more expensive Harrar & Spence (2013). The taste of cutlery. Flavour, 2: 21.

18. Weaksynaesthesia in music • Musical pitch “rises” and “falls” • A piece “moves” • A voice is “warm”, “round”, “relaxed”

19. Embodiedcognitionin different disciplines • Psychology: cognitiondepends on featuresofthephysicalbodyof an agent(Wilson & Foglia, 2011) • Music psychology: dance, rhythm, aesthetics etc. (Leman, 2008). • Humanities: conceptual metaphor, ideas in one domain help us understood in another (Lakoff & Johnson, 1980).

20. Timbralrangeofthe piano One note: • Louder is brighter Many notes: • higher is brighter • physical interactions among strings, soundboard, internal resonances • dependence on timing, dynamics, pedaling

21. The pianist’s “touch”A traditional psychoacoustic approach Physics: • Hammer hits string in free flight • Only free parameter is hammer velocity (shank rotation?) • Fingertips hit the keys differently in legato and staccato touch Perception: • Any effect beyond hammer-velocity effect is usually inaudible due to masking (cf. “touch precursor”, “early noise”; Goebl et al., 2004)

22. Piano touch: A synaestheticapproach For the pianist: • All aspects of somatic sense: touch, position, movement, temperature, pain… For the audience: • Vision (perception of gesture) • Projected somatic sense (empathize with performer) Forboth: Multisensoryperceptionoftheinteractionbetweentheinstrumentandtheperformer

23. The feelof a piano “Pianos and rooms are generally interdependent: anyone who has ever travelled with a piano knows that the same Steinway or Bösendorfer not only sounds different in different halls, but also seems to react differently in its mechanism. Indeed, the resistance of the key, over and above the measurable mechanical aspect, is a psychological factor” Alfred Brendel (1976)

24. The feelof a piano For pianists, the physical resistance or bounciness of the piano mechanism depends • physically on temperature and humidity • psychologically on touch-sound relationship(to which they are very sensitive)

25. Multisensory piano timbre In physics, spectral and temporal envelopes of an isolated piano tone cannot be changed independent of intensity In psychology and experience, timbre involves interactions between: • performers’ proximal sensations (tactile, auditory, visual, proprioceptive) • distalperception and cognition (performance space; communication with the audience; cultural context) These statements are not contradictory!

26. Weaksynasthesia in wind/stringtimbre • independent control over pitch and timbre, but: • intonation and timbre not psychologically separate (Ely, 1992; Geringer & Madsen, 1981; Platt & Racine, 1985) • musicians who play with good timbre are judged by experts and amateurs to have good intonation and vice-versa

27. Weaksynesthesia in Renaissance choraltimbre Listeners, audiences want “authentic” timbre. Some confuse that with just intonation. The sound is presumably optimal near 12-tone equal temperament (Devaney et al, 2011)

28. Weak synaesthesia in acousmatic music Acousmatic music is abstract, electronically synthesized sound from loudspeakers. Ecological approach to aesthetics: • Listeners constantly guess and imagine sources or causes of musical sounds – as in everyday life • Electronically generated sounds sound more “musical” if similar to familiar sounds

29. Disembodimentofmusictheoryandcognitivemusicpsychology • Music theory depends on the score • Cognitive music psychology depends on abstract cognitive structures The ecological solution: Study our experience of our physical interactions! Example: complex tones in speech and music (Terhardt, 1984).

30. A newdefinitionoftimbre • Experiential • Holisticand multimodal • Quantitative and qualitative • Proximal and distal

31. 1. Timbre isexperiential • Like pitch, loudness, and (in vision) color • Not a physical parameter! • Correspondsto physical states and events

32. 2. Timbre isholisticand multimodal Timbre depends generally and intrinsically on • input from all senses: hearing, vision, touch, gesture • physical context: acoustics and appearance of listening space • temporal context – immediate, global • empathy: shared or projected knowledge, experience, expectations • emotional reactions • associations Our ability to separate these components is limited.

33. 3. Timbre is qualitative and quantitative • complete description: both kinds of data • both are intrinsically vague and intangible • quantitative approach: • multidimensional (axis labels are qualitative) • qualitative descriptors • refer to environment, body, speech (Traube, 2004)

34. 4. Timbre is distal and proximal • A complete description includes both • Traditional psychoacoustic approach is proximal • Combine with ecological source-based approach Why? Timbre generally depends on imagined visual and tactile properties of sound sources, and past experience of those sources e.g. different clarinet registers are all “clarinet”

35. Solving the “touch” problem • Redefine timbre: combine experiential psychophysics with ecological psychology • Promote interdisciplinary interaction: humanities, sciences and performance • Abandon philosophical materialism: (artistic) experienceexists!