Exploring Emotion and Motivation in Brain-Inspired Developmental Networks

1. Emotion in Brain-Inspired Developmental Networks Juyang (John) Weng Computer Sci., Neurosci., Cognitive Sci.Michigan State University East Lansing, MI 49924 USA weng@cse.msu.edu

2. Patterns of Communication (a) Point to Point(b) Release hormones into blood (c) Autonomic nervous system; (d) Diffuse modulatory systems Beart, Connors & Paradiso 2007

3. Major Neurotransmitters Beart, Connors & Paradiso 2007

4. BMI 831 Cognitive Sciencefor Brain-Mind Research Lecturer:Juyang Weng

5. BMI 871 Introduction to ComputationalBrain-Mind Lecturer:Juyang Weng

6. Motivation/emotion in the Brain • Motivational system in the brain:biological causality for the development of emotion • Lower (physiology) emotion: • Arousal: noradrenaline, oxytocin, cortisol • Habituation sensitization, familiarization(acetylcholine and norepinephrine systems) • Pain avoidance (serotonin system) • Pleasure seeking (dopamine system) • Basic emotion: angry, happy, sad, scared, disgust, etc. • Higher emotion: moods, dispositions (e.g., jealousy)

7. Neuromodulatory Systems Jeff Krichmar, AB, 2008

8. Quiz: Why Modulations Quiz: The main purpose of modulatory systems is: • Animal like • Establish bounds with human • Look intelligent • Speed-up learning • Being quick to fight or flee

9. Quiz: Why Modulations Quiz: The main purpose of modulatory systems is: • Animal like • Establish bounds with human • Look intelligent • Speed-up learning • Being quick to fight or flee

10. Why Motivation? • Evolution: Pressure of survival • Understanding of the environment should not be the primary goal of a life • Fruit flies, Rats, Humans • Sexual selection: survival is not sufficient • Males: Fight with males, court with females • Females: Select the male winners • Motivation: quick ways to sense a variety of values • Neuromodulators, diffused transmission

11. In the blind, visual cortex is reassigned to audition and touch.Therefore, we chose not to statically model brain areas!Brain areas should emerge

12. Motivation: Architectural Concepts

13. Symbolic Value Systems • Sutton & Barto 1981: • reward as positive values • Delayed rewards • Ogmen 1997: Punishments, rewards, novelty • Sporns et al. 1999: Darwin robot • Kakade & Dayan 2002: novelty and shaping • Oudeyer et al. 2007: error max; progress max; similarity-based progress maximization • Huang & Weng 2007: punishment, reward, novelty • Cox & Krichmar 2009: Neuromodulation as a robot controller • Singh et al. 2010: reward and evolution

14. Symbolic: Q-Learning • Symbolic reinforcement learning • No need for state transition probability • Value driven • Does not allow state inconsistency • Future value estimation for delayed rewards • Time discount model: state-action value:

15. Q-Learning Limitations • Symbolic: • Exponential number of states, static • Brittle • Fixed, greedy value function: Immediate rewards are preferred • Does not allow perception • Does not allow creativity for other non-modeled concepts

16. Theory: For Any AFA There Is a GDN Marvin Minsky criticized ANNs AFA: Agent Finite Automaton GDN: GenerativeDevelopmental Network Weng IJCNN 2010

17. Emergent: 5-HT & DA Systems • Blue: 5-HT system • Red: DA system • Mechanism 1 (extra cell): • Mechanism 2 (within cell):zip increases the threshold Tzis reduces the threshold TIf pre-action > T, fire

18. 5-HT, DA, Ach, NE • Serotonin (5-HT): pain, stress, threats and punishment • Dopamine: (DA)pleasure, wanting, anticipation, and reward • Acetylcholine (Ach): expected uncertainty? • Norepinephrine (NE): novelty (unexpected uncertainty)?

19. Emergent Value Systems • Daly, Brown, Weng 2011, Paslaski, VanDam, & Weng 2011, Weng et al. Neural Networks 2013: • Neuromorphic Motivated Systems based on DN: 5-HT and DA • Wandering or foraging, face recognition • Wang, Wu, & Weng 2011, 2012 • Neuromorphic Motivated Systems based on DN: Ach and NE • Novelty and Uncertainty • Synapse maintenance • Segmentation of objects from backgrounds • Zheng, Qian, Weng, Zheng 2013 • Effects on internal brain areas: change learning rates

20. Effects of 5-HT and DA on Y? • Effect on Z area: Inhibit and excite actionsWeng et al. IJCNN 2011 • Effect on Y area:on learning rateZheng, Qian, Weng & Zhang, IJCNN 2013

21. Dopamine System

22. Serotonin System

23. Acetylcholine System

24. Norepinephrine System

25. Bottom-Up Weights and 5-HT

26. Y Weights for Z Neurons Paslaski et al. IJCNN 2011

27. Error Rates for Recognition Paslaski et al. IJCNN 2011

28. Navigation, Wandering, Foraging • Three synthetic agents: • Self (S) • Attractor (A) • Repulsor (R) • 5-HT: distance-lonely and distance-fear • DA: d < distance-desire

29. Setting

30. Environmental Settings • No brainer (control): no pain, no pleasure • Love or War (D-lonely, D-fear) = (50, 50) • Little Danger (D-lonely, D-fear) = (25, 125) • Much Danger (D-lonely, D-fear) = (125, 25) • Looking over the Fence(D-lonely, D-fear) = (125, 125)

31. Average Distance Daly et al. IJCNN 2011

32. Ach and NE: Goal for Segmentation (a) Bottom-up input to a neuron. (b) True object contour. (c) Estimated synaptogenic factor Wang, Wu and Weng, IJCNN 2011

33. Global Confidence: Face Recognition

34. Ach & NE: Receiver Operating Curve

35. ACh Estimates

36. DN: Motivational System as Emotion • Emergent nervous system • Modeled: • 5-HT (serotonin) • DA (dopamine) • Ach (acetylcholine) • NE (norepinephrine) • Basic and higher emotion: developed from experience