Future of Emotional Interaction

1. THE CENTER FOR KNOWLEDGE AND INNOVATION RESEARCH H E L S I N K I S C H O O L O F E C O N O M I C S Future of Emotional Interaction Niklas Ravaja, Ph.D. M.I.N.D. Lab/CKIR, Helsinki School of Economics, Finland email: ravaja@hse.fi

2. Emotions (I) • Emotions are biologically based action dispositions that have an important role in the determination of behavior (e.g., when interacting with computers) • Approach or withdrawal • Three components • Subjective experience • Physiological activation • Expressive behavior

3. Emotions (II) • Positive emotions (e.g., joy) serve as • A motivating source of goal-directed behaviors • Increase the subjective perception that one is capable of performing these behaviors • Increase the expectation that goal-directed activities ultimately will be rewarded • An affective reward for goal-directed behaviors • A paradox: people may also like situations eliciting negative emotional responses (e.g., suspense, fear, anger)

4. Basic Emotions • Anger • Disgust • Fear • Joy • Sadness • Surprise

5. Aroused Alert Excited Energetic Peppy Joyful Enthusiastic Angry Aggressive Fearful Anxious Frustrated Arousal Sad Dissatisfied Disappointed Happy Satisfied Displeasure Pleasure Tired Bored Dull Helpless Relaxed Calm Inactive Idle Passive Valence–Arousal Circumplex Model of Emotion

6. An Alternative Conceptualization: Positive Affect and Negative Affect Highwithdrawalmotivation Highapproachmotivation • High Negative Affect • High BIS activity • High Positive Affect • High BAS activity Arousal Displeasure Pleasure • Low Positive Affect • Low BAS activity • Low Negative Affect • Low BIS activity BAS = Behavioral Activation system BIS = Behavioral Inhibition System

7. Constructs and Measures There is a difference between constructs (e.g., emotion, attention) and measures or ways to identify them (e.g., facial electromyographic [EMG] activity, heart rate [HR]). Constructs are always hypothetical and not directly observable, but measures are.

8. Confirmatory Factor Analysis Self-reported experience Emotion Electrodermal activity Expressive behavior Rectangles indicate measured variables, ellipses represent latent constructs, and small circles represent residual or disturbance variables (variances). Latent factors (e.g., emotion) capture the true score of the common variation among the indicator variables, whereas the unique or specific aspects of the indicators are separated into a uniqueness or error term.

9. Different Ways to Detect Emotions in Emotional Interaction • Automatic detection of (emotional) facial expressions from a camera image • Behavioral measures (e.g. key pressures) • Psychophysiological measures

10. Advantages of Psychophysiological Measures over Self-Report • More objective data (not dependent on language and memory) • Measurement can be performed continuously during message processing • Measurements may provide information on emotional and attentional responses that are not available to conscious awareness

11. Psychophysiological Measures of Emotional Valence (I) • The contraction of facial muscles is an important form of emotional expression • The facial EMG provides a direct measure of the electrical activity associated with facial muscle contractions • The primary psychophysiological index of hedonic valence (pleasant vs. unpleasant) • Zygomaticus major (cheek) muscle area • An index of positive emotions • Corrugator supercilii (brow) muscle area • An index of negative emotions • Orbicularis oculi (periocular) muscle area • An index of positively valenced high-arousal emotions • Facial expressions not only reflect one’s emotional state, but also serve a social communicative function • The use of facial EMG as a measure of emotions (emotional valence) requires construct validation

12. Psychophysiological Recordings in Real-World Contexts • Varioport-B mobile physiological data acquisition system (Becker Meditec, Karlsruhe, Germany) for measuring ECG, facial EMG, EEG, EDA, respiration, physical activity, and ambient temperature, noise, and illumination. • Emotion- and attention-related psychophysiological recordings can be carried out when the participants are interacting in different contexts (e.g., laboratory, their own home) • Increases ecological validity

13. Psychophysiological Measures of Arousal (Bodily Activation) • Electrodermal activity (EDA) • The primary psychophysiological index of arousal • As people experience arousal their sympathetic nervous system is activated, resulting in increased sweat gland activity and skin conductance • Heart rate (HR; or interbeat interval, IBI) • HR accelerates with increasing arousal during active coping tasks (sympathetic nervous system) • HR decelerates with increasing attentional engagement during sensory intake tasks (parasympathetic nervous system) Ravaja, N. (2004). Contributions of psychophysiology to media research: Review and recommendations. Media Psychology, 6, 193-235.

14. Phasic Emotion-Related Psychophysiological Responses to Game Events Game: Super Monkey Ball 2 Event: Monkey picks a banana Arousal  Positive affect  Positive affect  Game events elicit reliable emotional valence- and arousal-related psychophysiological responses Ravaja, N., Saari, T., Salminen, M., Laarni, J., & Kallinen, K. (2006). Phasic emotional reactions to video game events: A psychophysiological investigation.Media Psychology, 8, 343-36

15. Problems Associated with Psychophysiological Measures: Heart Rate as an Example • Heart rate (HR) is the measure of how many times the heart beats in a minute • HR is influenced by both the sympathetic nervous system (SNS) and parasympathetic nervous system (PNS) • SNS activity causes the heart to speed up • PNS activity causes the heart to slow down • HR is primarily under SNS control during sensory rejection tasks (i.e., attention to internal stimuli), such as mental arithmetic and emotional imagery • SNS activity increases with increasing emotional arousal • HR indexes arousal during these tasks • HR is primarily under PNS control during sensory intake tasks (i.e., attention to external stimuli), such as finding errors from a text and watching a film • PNS activity increases with increasing attentional engagement • HR indexes attentional engagement during these tasks • Interpretation is highly dependent on the research paradigm, task demands, and even the content of the stimuli

16. Emotions and EEG

17. Emotions and frontal EEG asymmetry: Valence model • The left and right frontal cortical regions are asymmetrically involved in the experience and expression of emotion. • Left frontal cortical activity is associated with positive emotions and approach motivation, whereas right frontal cortical activity is associated with negative emotions and withdrawal motivation (Harmon-Jones, 2003). • Relatively greater left frontal cortical activation has been associated with positive affective states, and relatively greater right frontal activation with negative emotional states, during directed facial action task, music listening, and exposure to different types of films.

18. Emotions and frontal EEG asymmetry: Motivational direction model • The left and the right anterior region of the brain are part of two separate neural systems underlying approach and withdrawal motivation, respectively (e.g., Davidson, 1995) • Emotions that are most often “approach” oriented, such as joy, interest, and even anger, are associated with relatively greater left frontal activation, whereas emotions that are most often “withdrawal” oriented are associated with relatively greater right frontal activation. • Thus, although anger is negative in valence, it often evokes approach motivation and is associated with relatively greater left frontal activation (Harmon-Jones, 2003).

19. Psychophysiological Measures of Attention • Respiratory sinus arrhythmia (RSA; long-term attentional engagement) • high-frequency component of HR variability: the frequency band ranging from 0.15 to 0.40 Hz (respiratory-locked oscillations in HR) • the greater the vagal (parasympathetic) regulation of metabolic activity, the more HR will accelerate and decelerate in response to respiration, producing an RSA waveform with larger amplitude • Phasic heart rate (HR) deceleration (short-term attentional selection) • Electroencephalography (EEG; reduction in alpha-wave power)

20. Social and Emotional Cues in Mediated Communication

21. Social and Emotional Cues in Mediated Communication (II) • Negative (fake) cues of group emotion (depressed, nervous/stressed) elicited higher corrugator supercilii EMG activity during the task compared to cues of positive group emotion (positively excited, positively relaxed), F(1,29) = 6.76, p = .015, η2 = .19 • Corrugator supercilii EMG activity was higher during routine tasks than during creative tasks, F(1,29) = 19.16, p < .001, η2 = .40 (corrugator activity increases not only during negative emotions but also during effortful attention).

22. The Effect of Opponent Type on Emotional Responses when Playing Digital Games (I) Arousal (as indexed by self-report and heart rate) and positively valenced emotional responses (as indexed by self-report and facial EMG) increased in the order of playing against a computer < playing against a (co-located or non co-located) stranger < playing against a (co-located or non co-located) friend

23. The Effect of Opponent Type on Emotional Responses when Playing Digital Games (II) The mere knowledge that one’s avatar is interacting with an avatar of another human strongly influences emotional responses as indexed by facial EMG when playing digital games. Ravaja, N., Saari, T., Turpeinen, M., Laarni, J., Salminen, M., & Kivikangas, M. (2006). Spatial presence and emotions during video game playing: Does it matter with whom you play? Presence: Teleoperators and Virtual Environments, 15, 381-392.