Affective Computing: Machines with Emotional Intelligence

1. Affective Computing: Machines with Emotional Intelligence Hyung-il Ahn MIT Media Laboratory

2. …doesn’t notice you are annoyed. [Doesn’t recognize your emotion] You express more annoyance. He ignores it. [Stupid about handling your emotion] He winks, and does a happy little dance before exiting. [Stupid about expressing emotion.]

3. Skills of EmotionalIntelligence: • Expressing emotions • Recognizing emotions • Handling another’s emotions • Regulating emotions \ • Utilizing emotions / (Salovey and Mayer 90, Goleman 95) if “have emotion”

4. Research Areas • Robotic Computer - Recognizing another’s emotions - Expressing emotions - Handling another’s emotions • Affective and Cognitive Decision Making - Regulating and utilizing emotions - Affect as a self-adapting control system Affect changes the operating characteristics of other three domains (cognition, motivation, behavior)

5. Recognizing Emotions

6. Recognition of three “basic” states:

7. Future “teacher for every learner”

8. Sit upright Lean Forward Slump Back Side Lean Can we teach a chair to recognize behaviors indicative of interest and boredom? (Mota and Picard)

9. Boredom Interest

10. What can the sensor chair contribute toward inferring the student’s state: Bored vs. interested? Results (on children not in training data,Mota and Picard, 2003): 9-state Posture Recognition: 89-97% accurate High Interest, Low interest, Taking a Break: 69-83% accurate

11. Detecting, tracking, and recognizing facial expressions from video (IBM BlueEyes camerawith MIT algorithms)

12. Autism Spectrum Conditions Center for Disease Control and Prevention (2005) • 1 child in 166 has ASC

13. Mind-Read > Act > Persuade hmm … Roz looks busy. Its probably not a good time to bring this up Inference and reasoning about mental states Modify one’s actions Persuade others Analysis of nonverbal cues

14. Real time Mental State Inference El Kaliouby and Robinson (2005) Facial feature extraction Head & facial action unit recognition Head & facial display recognition Mental state inference Head pose estimation Feature point tracking* hmm … Let me think about this * Nevenvision face-tracker

15. Baron-Cohen et al. AUTISM RESEARCH CENTRE, CAMBRIDGE Assertive Committed Persuaded Sure Agreeing Absorbed Concentrating Vigilant Concentrating Complex Mental States (subset) Disapproving Discouraging Disinclined Disagreeing Asking Curious Impressed Interested Interested Brooding Choosing Thinking Thoughtful Thinking Baffled Confused Undecided Unsure Unsure Affective-Cognitive Mental States

16. Physically animated Robotic Computer (joint with Prof. Cynthia Breazeal) Goal: increase user movement without distraction and annoyance, further social-rapport building

17. Robotic Computer (RoCo): A physically animated computer Learning: the user can guide RoCo’s behavior by explicit and implicit rewards and punishments (Reinforcement Learning)

18. RoCo’s postures congruous to the user affect “Stoop to Conquer” : Posture and affect interact to influence computer users’ comfort and persistence in problem solving tasks People tend to be more persistent and feel more comfortable when RoCo’s posture is congruous to their affective state N=(17)

19. Procedure and Tasks Tracing Task: a solvable and an unsolvable puzzle Decision-making Task (in Experiment 2): to make subjects keep the target posture longer

20. Affective Cognitive Decision Making

21. (Example 1) Two-armed bandit gambling tasks Inspired by Bechara & Damasio’s IOWA gambling tasks (Bechara et al. 1997) The left arm has ‘Negative Valence’ Arousal (uncertainty) as ‘feeling uneasy’ The right arm has ‘Positive Valence’ Arousal (uncertainty) as ‘feeling lucky’

22. - $4000 (Pr=0.8) $ 0 (Pr=0.2) $4000 (Pr=0.8) $ 0 (Pr=0.2) - $3000 (Pr=1) $3000 (Pr=1) (Example 2) Decision making under risk Loss aversion: People strongly prefer avoiding losses than acquiring gains ‘Risk-Averse’ choices in the domain of ‘Likely Gains’ > Option 1 Option 2 < Expected value = $3000 (Gain) Expected value = $4000 * 0.8 + $0 * 0.2 = $3200 (Gain) ‘Risk-Seeking’ choices in the domain of ‘Likely Losses’ < Option 1 Option 2 > Expected value = - $3000 (Loss) Expected value = - $4000 * 0.8 + $0 * 0.2 = - $3200 (Loss)

23. Reference Dependence: gains and losses are defined relative to the reference point - Concave above the reference point - Convex below the reference point The PT (Prospect Theory) value function • Diminishing sensitivity: less sensitive to outliers for both gains and losses • Loss aversion: the function is steeper in the negative (loss) domain (Tversky & Kahneman)

24. Endowment Effect • people place a higher value on objects they own relative to objects they do not. • In one experiment, people demanded a higher price for a coffee mug that had been given to them but put a lower price on one they did not yet own. • The endowmenteffect was described as inconsistent with standard economic theory which asserts that a person's willingness to pay (WTP) for a good should be equal to their willingness to accept (WTA) compensation to be deprived of the good. This hypothesis underlies consumer theory and indifference curves. • The effect is related to loss aversion and status quo bias in prospect theory.

25. (Example 3) Effects of mood on decision making (Lerner & Keltner 2000, 2001, 2004) Happiness Anger Optimistic about judgments of future events Optimistic judgments of future events, Risk-Seeking choices Reverse Endowment Effect Pessimistic judgments of future events, Risk-Aversive choices Fear Sadness

26. Subjective Value Function(mood influences decision making)

27. Affective Cognitive Learning and Decision Making • A new computational framework for learning and decision making inspired by the neural basis of motivations and the role of emotions in human behaviors • A motivational value (reward)-based learning theory: decision value = extrinsic (cognitive) value + intrinsic (affective) value extrinsic value from the cognitive (deliberative and analytic) systems intrinsic value from multiple affective systems such as Seeking, Fear, Rage, and other circuits. • Probabilistic models: Cognition (cognitive state transition), Multiple affect circuits (Seeking, Joy, Anger, Fear, …), and Decision making model • Any prior and learned knowledge can be incorporated for expecting the consequences of decisions (or computing the cognitive value)

28. To destroy the ring in Mordor with less effort Choice 1 Effort (r = -80) Prob Fearless/ Neutral / Fearful Mood Incidental Emotions Reward -30 0 20 70 Expected Values Cognitive Expectations choice 1 = 20, choice 2 = 20 Choice 2 Effort (r = -30) Valenced Uncertainty Values Anticipatory Emotions from the Seeking Circuit choice 1 = positive, choice 2 = negative Pr = 0.5 Success (r = 100) Fail (r = 0) Fear Anticipatory Emotions from Other Circuits