The Brain’s Concepts The Role of the Sensory-Motor System in Reason and Language George Lakoff University of California

1. The Brain’s Concepts The Role of the Sensory-Motor System in Reason and Language George Lakoff University of California, Berkeley (with Vittorio Gallese)

2. With Thanks to The Neural Theory of Language Group International Computer Science Institute University of California, Berkeley Especially Jerry Feldman, Srini Narayanan, Lokendra Shastri, and Nancy Chang. http://www.icsi.berkeley.edu/NTL

3. What Concepts Are: Basic Constraints Concepts are the elements of reason, and constitute the meanings of words and linguistic expressions.

4. The Traditional Theory Reason and language are what distinguish human beings from other animals. Concepts therefore use only human-specific brain mechanisms. Reason is separate from perception and action, and does not make direct use of the sensory-motor system. Concepts must be “disembodied” in this sense.

5. We Claim Human concepts are embodied. Many concepts make direct use of the sensory-motor capacities of our body-brain system. Many of these capacities are also present in non-human primates. One example, the concept of grasping, will be discussed in detail.

6. Amodality The traditional theory implicitly claims that even action concepts, like grasp, do not make use of the sensory-motor system. As a concept, even grasp must be disembodied. Thus, it is claimed that the concept grasp is amodal. Since it is a concept, it must be modality-free, even if it designates an action in a specific modality.

7. Concepts Are: • Universal: they characterize all particular instances; e.g., the concept of grasping is the same no matter who the agent is or what the patient is or how it is done. • Stable. • Internally structured. • Compositional. • Inferential. They interact to give rise to inferences. • Relational. They may be related by hyponymy, antonymy, etc. • Meaningful. • Independent of the words used to express them.

8. Concepts may be either ‘concrete’ (sensory-motor) or ‘abstract’ (not sensory-motor).

9. Basic Ideas • Multimodality— Permits universality • Functional Clusters— High-level, function as conceptual units • Simulation — Necessary for meaningfulness and contextual inference • Parameters — Govern simulation, strict inference, link to language

10. Multimodality The action of grasping is not amodal, but multi-modal in a way that makes for universality.

11. Functional Clusters • Functional clusters form high-level units — with the internal relational structure required by concepts. • There are two types: Local clusters and Network clusters. • Multi-modality is realized in the brain through network clusters, that is, parallel parietal-premotor networks. • Network clusters are formed by interconnected local clusters of neurons, like canonical and mirror neurons.

12. Simulation • To understand the meaning of the concept grasp, one must at least be able to imagine oneself or someone else grasping an object. • Imagination is mental simulation, carried out by the same functional clusters used in acting and perceiving. • The conceptualization of grasping via simulation therefore requires the use of the same functional clusters used in the action and perception of grasping.

13. Simulation and Enactment Visual imagination uses part of the same neural substrate as vision. Motor imagination uses part of the same neural substrate is motor action. Since you can understand a concrete concept like grasping only if you can imagine doing it or observing it, the capacity for mental simulation is taken as the basis for meaningfulness. Thus, action and observation provide the basis for meaningfulness in NTL.

14. Parameters • All actions, perceptions, and simulations make use of parameters and their values. Such neural parameterization is pervasive. • E.g., the action of reaching for an object makes use of the parameter of direction; the action of grasping an object makes use of the parameter of force. • The same parameter values that characterize the internal structure of actions and simulations of actions also characterize the internal structure of action concepts.

15. Structured Neural Computation in NTL The theory we are outlining uses the computational modeling mechanisms of the Neural Theory of Language (NTL). NTL makes use of structured connectionism (Not PDP connectionism!). NTL is ‘localist,’ with functional clusters as units. Localism allows NTL to characterize precise computations, as needed in actions and in inferences. Because it uses functional clusters, NTL is not subject to the “grandmother cell” objection.

16. Advantages of Structured Connectionism • Structured connectionism operates on structures of the sort found in real brains. • From the structured connectionism perspective, the inferential structure of concepts is a consequence of the network structure of the brain and its organization in terms of functional clusters.

17. Structured Connectionism comes with: • A dynamic simulation mechanism that adapts parameter values to situations. • A neural binding theory. • A spreading-activation probabilistic inference mechanism that applies to functional clusters. • These jointly allow for the modeling of both sensory-motor simulations and inference.

18. In NTL, there are fixed structures called schemas. For example, a schema that structures an action has an internal structure consisting of Roles, Parameters, and Phases. The ideas of Multimodality, Functional Clusters, Simulation, and Parameters allow us to link NTL, with structured connectionism, to neuroscience.

19. The Neuroscience Evidence Shows • In the sensory-motor system, it is possible to • characterize these aspects of concepts: • Universality • Semantic Role Structure • Aspectual Structure (Phases) • Parameter Structure

20. The Concept Of Grasping

21. Universality Is Achieved by MultiModality • Multimodal functional clusters for an action like grasping fire when: • Grasping is performed, observed, imagined, inferred, or heard; • The grasping is of any type, done by any agent, on any object, • in any manner, and in any location. • In showing such multimodality for a functional cluster, we are showing that the functional cluster plays the conceptualrole of universality.

22. Multi-Modal Integration The premotor cortex is not a uniform field, but a mosaic of functionally distinct areas (F1 to F7). Each of these premotor areas is reciprocally connected with distinct regions of the posterior parietal cortex. The premotor cortex ispart of a series of parallel functional network clusters.

23. Multi-Modal Integration Cortical premotor areas are endowed with sensory properties. They contain neurons that respond to visual, somatosensory, and auditory stimuli. Posterior parietal areas, traditionally considered to process and associate purely sensory information, also play a major role in motor control.

24. A New Picture Rizzolatti et al. 1998

25. The fronto-parietal networks Rizzolatti et al. 1998

26. Area F5 Three classes of neurons: -Motor General Purpose neurons -Visuo-Motor neurons: -Canonical neurons -Mirror neurons

27. Area F5 General Purpose Neurons: General Grasping General Holding General Manipulating

28. General Purpose Neurons in Area F5 A Grasping with the mouth B Grasping with the cl. hand C Grasping with the ipsil. hand (Rizzolatti et al. 1988)

29. General Purpose Neurons Achieve Partial Universality: Their firing correlates with a goal-oriented action of a general type, regardless of effector or manner.

30. F5c-PF Rizzolatti et al. 1998

31. The F5c-PF circuit Links premotor area F5c and parietal area PF (or 7b). Contains mirror neurons. Mirror neurons discharge when: Subject (a monkey) performs various types of goal-related hand actions and when: Subject observes another individual performing similar kinds of actions

32. Area F5c Convexity region of F5: Mirror neurons

33. F5 Mirror Neurons Gallese and Goldman, TICS 1998

34. Strictly congruent mirror neurons (~30%) (Rizzolatti et al. Cog Brain Res 1996)

35. Category Loosening in Mirror Neurons (~60%) (Gallese et al. Brain 1996)

36. PF Mirror Neurons (Gallese et al. 2002)

37. A (Full vision) B (Hidden) C (Mimicking) D (HiddenMimicking) Umiltà et al. Neuron 2001

38. Like humans, monkeys can also infer the goal of an action, even when the visual information about it is incomplete.

39. F5 Audio-Visual Mirror Neurons Kohler et al. Science (2002)

40. Somatotopy of Action Observation Foot Action Hand Action Mouth Action Buccino et al. Eur J Neurosci 2001

41. The Mirror System in Humans BA6

42. The Simulation Hypothesis How do mirror neurons work? By simulation. When the subject observes another individual doing an action, the subject is simulating the same action. Since action and simulation use some of the same neural substrate, that would explain why the same neurons are firing during action-observation as during action-execution.

43. Mirror Neurons Achieve Partial Universality, since they code an action regardless of agent, patient, modality (action/observation/hearing), manner, location. Partial Role Structure, since they code an agent role and a purpose role. The Agent Role: In acting, the Subject is an agent of that action. In observing, the Subject identifies the agent of the action as having the same roleas he has when he is acting – namely, the agent role. The Purpose Role: Mirror neurons fire only for purposeful actions.

44. Mirror Neurons Achieve Category tightening and loosening Limited Prototype Structure

45. F5ab-AIP

46. The F5ab-AIP circuit Links premotor area F5ab and parietal area AIP. Transforms intrinsic physical features of objects (e.g., shape, size) into hand motor programs required to act on them Examples: Manipulate objects, grasp them, hold them, tear them apart.

47. Area F5ab Bank region of F5: Canonical neurons

48. F5 Canonical Neurons Murata et al. J Neurophysiol. 78: 2226-2230, 1997

49. F5 Canonical Neurons Murata et al. J Neurophysiol. 78: 2226-2230, 1997

50. The Simulation Hypothesis How Do Canonical Neurons Work? By Simulation. The sight of a graspable object triggers the simulation of grasping. Since action and simulation use some of the same neural substrate, that would explain why the same neurons are firing during object-observation as during action-execution.