Advanced HCI Affective Interaction

1. Advanced HCIAffective Interaction Affect is dealt with in: Preece, Sharp and Rogers (Chapter 5) Benyon, Turner and Turner (Chapter 17) This lecture reviews explanations of why and how interactive products influence users’ emotions, and through this, their attitudes and behaviour.

2. Performance & Emotion • Early cognitive contributions to HCI focus on improving users’ performance • Usability • Making tasks easier (preventing error, providing help etc) • Lightening the “cognitive load” • Making tools intuitive • Main aim in the late ‘80s: to improve design of computer-based work • Users’ emotions only viewed as byproduct of unhindered cognitive processing • Software/Hardware works = user satisfaction/acceptance • Focus of computing changing: from workplace to lifestyle • iPod, Xbox, Palm Pilot, games, chat, shopping, entertainment, leisure… • Activities associated with pleasure (emotion) • Not just productivity CHILL OUT!

3. Performance & Emotion • What are emotions? • Why, and how, do they affect our performance? • In what ways can emotion be taken into account when designing programs or devices? • 3 main research areas: • Computer recognition of human emotion • Computer adapts its behaviour based on what it knows about user • Computer simulation of human emotion • Computer appears to display human characteristics • Computer evocation of human emotion • Computer targets human emotions and encourages particular responses

4. What is an emotion? • A neural impulse that moves an organism to action • We cannot choose our emotions (see below) • Emotions happen spontaneously in response to events in the environment • Causes physiological changes visible in behaviour • Facial expression, faster heartbeat, perspiration, aggression, crying etc. • Deeply rooted survival mechanism enabling organism to adapt to environment • cf. Method Acting • The actor “replicates” the emotional conditions of the character in order to add realism to a dramatic performance • Actor draws on own “emotional memory” Pairs of emotions (Plutchik, 1980)Primary emotions can be combined to form secondary emotionsDisgust + Sadness = Remorse

5. Where do emotions come from? • ¢Cerebral Cortex (Man) • Planning, language etc. • ¢Limbic System (other mammals) • Emotions • £ Basal Ganglia • 2-way messaging system between Cerebral Cortex & Limbic System • ¢Brain Stem (present in reptiles) • Basic body functions • Hunt, kill, mate, defend territory… Evolution of the brain • Common belief was that rationality & emotion are separate cf. early cognitive view that treats user as an information processor • Emotion plays some part in decision-making etc. Therefore affects performance, response etc.

6. Reflective control Sensory Behavioural Motor control Visceral Don Norman: Revised model of cognition • Visceral level • Fast • Makes rapid judgements • Good/bad; safe/dangerous • Signals motor system & alerts brain • Inhibited/enhanced by control systems • Behavioural level • Learned behaviour coming from experience • Largely automaticised • Can inhibit/enhance visceral layer • Inhibited/enhanced by reflective layer • Reflective level • No direct access to sensory input or control of behaviour • Watches over, reflects upon, & tries to influence behaviour level • Responses slow based on assessment, judgement & decision-making • Norman, 2004

7. Emotional responses to stimuli

8. Emotional response to stimuli • Ekman, Friesen & Ellsworth (1972) • identified 6 basic emotions • recognised & expressed in same way in all cultures anger fear disgust surprise happiness sadness Which emotions are these people displaying? (Ekman, Friesen & Ellsworth, 1972)

9. Relationships with Computers • Humans respond to computers like real people • We prefer computers that flatter us • We prefer computers which have a similar personality to our own • We are polite to computers • Therefore, we seem to have social relationships with computers • Emotions are an important component of social relationships… • If computer has a friendly face, will we “like” it more?

10. Facial Action Coding System (FACS) • Ekman and Friesen (1978) develop FACS • Taxonomy of expressions • Defines expressions as 1 of 46 “Action Units” • Contraction or relaxation of one or more muscles • Cowe & Johnston (Psychology Dept. University College, London) • system automatically generates a virtual puppet, or avatar, then drives it by mimicking the facial movements of an actor. • Applications? • Recognising user expression • Generating appropriate responses for avatars actor avatar

11. Recognizing emotions • Tele-healthcare (Lisetti, et al. 2003) • System models patient’s affective state using multiple inputs from wearable sensors and other devices such as camera • Detected emotions mapped onto intelligent agents embodied as avatars • Avatar able to chat to patient to confirm emotion • 90% success recognizing sadness • 80% success for anger • 70% success for frustration

12. Expressing emotion: affective input to interactive systems • Can humans express emotions to computers? • What if human wants to explicitly identify an emotion to influence the behaviour of a character in a game? • SenToy • Doll-based interface to game • Gestures indicate emotions • Emotions affect game strategy happiness sadness anger fear surprise gloating

13. Expressing emotion: affective input to interactive systems • Users a little confused • Hard to understand opponent’s emotions • BUT many related to doll (projection) • Higher satisfaction Bullock, A. and Gambäck, B. (2003)

14. The tenuous area of “Affective Computing” • Rash of projects exploring “Affective Computing” • Example: Gustbowl project (Keller, van der Hoog, and Stappers, 2004) • “Digital communication technology is increasingly affecting the way people organize their social contacts. Product designers therefore must understand users’ needs, not only on a functional level (what information they share) but also on an affective, experiential level (what emotions are involved). Technological communications solutions can easily fail because they reduce affective interactions to functional ones. Our team from Delft University of Technology’s ID-Studiolab (http://studiolab.io.tudelft.nl) designed the Gustbowl to promote and support informal, unobtrusive interactions in families whose members live apart. The Gustbowl helps families keep in touch, rather than just exchange information, by letting members be a part of each other’s daily routines.” • Use scenario: son throws keys into the Gustbowl; mother notices the wobble and sees the picture appear in her Gustbowl. • Can such projects really be classed as “affective computing”?

15. The tenuous area of “Affective Computing” • Example: A Foundation for Emotional Expressivity (Stahl et al. ) • “To express emotions to others in mobile text messaging in our view require designs that can both capture some of the ambiguity and subtleness that characterizes emotional interaction and keep the media specific qualities. Through the use of a body movement analysis and a dimensional model of emotion experiences, we arrived at a design for a mobile messaging service, eMoto. The service makes use of the sub-symbolic expressions; colors, shapes and animations, for expressing emotions in an open-ended way.” 8 backgrounds each “expressing” a different emotion?

16. Subjective association between colour & emotion? • Can we agree on emotion - colour linkage? • What would you say your emotional state is at the moment? • Don’t say “boredom”!

17. Computers as Persuasive Technology • Captology (Fogg, 2002) • Interactive technology to persuade people to change ideas & behaviour • Computing technologies have some advantages over human persuaders. For example, computers can: • Be more persistent than humans • Offer greater anonymity • Manage huge volumes of data • Go where humans cannot go or may not be welcome • Computing technologies persuade in different ways, depending on their role: • Tool • Social actor • Medium

18. Computers as Persuasive Technology • A tool can be persuasive by: • Making target behaviour easier to do • Leading people through a process • Performing calculations or measurements that motivate people • A social actor can be persuasive by: • Rewarding people by giving positive feedback • Modelling a target behaviour or attitude • Providing social support • A medium can be persuasive by: • Allowing people to explore cause and effect relationships • Providing people with various experiences that motivate them • Helping people to rehearse a behaviour

19. Persuasive Technology: some difficulties • Ethical issues • When and how am I being persuaded? Do I know? • Intrusive? • Research • How easy is it to study persuasive technology? • If your bank is doing this, will they tell you? • How can you find out the effect? • Even if you can find out: • Aren’t people who have been “persuaded” likely to be defensive?

20. References • Bullock, A. and Gambäck, B. (2003). Evaluating affective interaction in gaming. Presented at the 8th European Conference on Computer Supported Cooperative Work (ECSCW 03), 14th-18th September 2003, Helsinki, Finland. • Ekman, P., Friesen, W.V. and Ellsworth, P. (1972). Emotion in the Human Face. Pergamon, NY. • Ekman, P. and Friesen, W.V. (1978). The Facial Action Coding System. Consulting Psychologists’ Press, Palo Alto, CA. • Fogg, B.J. (2002). Persuasive Technology: Using Computers to Change What We Think and Do. Morgan Kaufmann. • Keller, I., Van der Hoog, W. and Stappers P.J. (2004). Gust of Me: Reconnecting Mother and Son. IEEE Pervasive Comp. 3,1(2004), 22-28. • Lisetti, C., Nasoz, F., LeRouge, C., Ozyer, O. and Alvarez, K. Developing multimodal intelligent affective interfaces for tele-home health care. Int. J. Human-Computer Studies 59 (2003), 245-255. • Norman, D. (2004). Emotional Design: Why we love (or hate) everyday things. Basic Books, NY. • Plutchik, R. (1980). Emotion: A Psychobioevolutionary Synthesis. Harper and Row, NY. • Reeves, B. and Nass, C. (1996) The Media Equation: How people treat computers, television, and new media like real people and places. Cambridge University Press, New York. • Ståhl, A., Sundström, P., and Höök, K. (2005). A Foundation for Emotional Expressivity. In: Designing For User Experience, DUX 2005, 3-5 Nov, 2005, Fort Mason, San Francisco, CA.