From Mere Embodiment to the Cyborg Mind

1. From Mere Embodiment to the Cyborg Mind Andy Clark Cognitive Science Program Indiana University andy@indiana.edu As of Fall 2004 Dept of Philosophy Edinburgh University Scotland

3. disembodied symbolic abstract rule-following clunky chunky high-level slow brittle…..etc

6. embodiment grounding dynamics perception-action systems real-world real-time embedded enactive……etc

7. Just how different ARE these two paradigms really?Just what is it about embodiment that (when taken seriously) really matters for the very shape of a science of the mind? What is the RELATION between those two apparently very different sets of target activities (between abstraction - rich reflective reason and more basic capacities for embodied action)? If embodiment matters, does it still matter ‘all the way up’?

8. Two Questions √ 2.Embodiment: What Really Matters? 3.Sensing 4.Incorporation versus Use 5.Mind and Reason 6.Can the be a Science of Hybrids?

9. Two Questions √ 2.Embodiment: What Really Matters? 3.Sensory Substitution 4. Incorporation versus Use 5.Mind and Reason 6. Can there be a Science of Hybrids?

10. Mere Embodiment Modest Embodiment Profound Embodiment

12. For Shakey, the body and the environment were first and foremost problems to be solved. The environment was the problem arena. The sensors detected the lay-out in that arena. The reasoning system planned a solution. The body was just another problem, that then needed to be micro-managed so as to put the solution into practice.

13. Passive Dynamic Walkers (PDW’s) • (Andy Ruina Lab, Cornell: original work by Tad McGeer) • No actuation except gravity, and no control system, except for a mechanical knee. • Inner and outer legs are paired to constrain it from falling over sideways. • Surprisingly, PDW’s are capable (when set on a gentle incline) of very stable, human-looking walking.

15. Systematically pushing, damping and tweaking a system in which Passive Dynamic effects play a very major role • = a very simple example of • ecological control

16. Ecological Control • An ecological control system is one in which goals are not achieved by micro- managing every detail of the desired action or response, but by making the most of robust, reliable sources of relevant order in the bodily or worldly environment of the controller. • (See also Rob Wilson on ‘exploitative control’).

17. systems that are specifically designed so as to constantly search for opportunitiesto make the most of body and world, checking for what is available, and then (at various time-scales and with varying degrees of difficulty) integrating it deeply, creating new functional wholes. i.e. a dynamically adaptive form of ecological control.

18. Any control system that is highly engineered so as to be able to learn to make maximal problem-simplifying use of an open-ended variety of internal, bodily, or external sources of order. = dynamically adaptive ecological control

19. “Ecological control” names an overall effect not a single mechanism. That effect (the achievement of delicate adaptive balances between environmental, neural and bodily dynamics) comes in many degrees and flavors, all the way from more-or-less hard-wired ecological balances to learnt-on-the-fly ecological balances.

20. Stelarc

21. The third hand is controlled by EMG signals detected by electrodes placed on four strategic muscle sites on Stelarc’s legs and abdomen. The third hand is controlled by Stelarc’s brain via muscle commands to these sites that are then relayed to the prosthesis. Since these sites are not normally used for hand control, the third hand can be moved independently of the other two

22. Stelarc simply feels as if he wills the third hand to move, just as he wills his biological hands to move. In each case, the control is fluent and intuitive. It seems to require no special effort or conscious focus.

23. Normally, you don’t feel as if you are (for example) USING your hand to do the washing up. Instead, you just feel as if YOU are washing up. Your hand functions as what some philosophers call TRANSPARENT EQUIPMENT : equipment through which you can act on the world without first willing an act on anything else. (see Heidegger (1927) on the ‘ready-to-hand’)

24. The same is true in the biological case. The human infant must learn, by trial and error and practice, which neural commands bring about which bodily effects,and must then practice until she is skilled enough to issue those commands without conscious effort (so the body becomes transparent equipment)

25. A monkey, with implanted electrodes monitoring brain activity, learns to control a joystick to move a cursor to get rewards. The monitoring computer learns what neural commands correspond to what joystick motions. Next, the joystick is disconnected. The monkey discovers, though, that it can still use its own neural commands ( as transmitted by the implanted electrodes and decoded by the monitoring computer ) to directly control the cursor. Finally, the commands are diverted to control a distant robot arm, whose motions are reflected in the on-screen cursor movements, thus closing the loop.

26. Picture from New Scientist webpage

27. “the dynamics of the robot arm (reflected by the cursor movements) become incorporated into multiple cortical representations…we propose that the gradual increase in behavioral performance…emerged as a consequence of a plastic re-organization whose main outcome was the assimilation of the dynamics of an artificial actuator into the physiological properties of frontoparietal neurons” Carmena et al Public Library of Science: Biology Vol 1: 2:2003 p.205

28. Whereas modest embodiment treats the body as a fixed (though highly significant) resource, profound embodiment is characterized by constant learning and re-calibration. Biological forms of embodiment, unlike a lot of current work in robotics, all tend towards the ‘profound’ end of this spectrum, though we primates seem especially plastic and well-engineered for multiple embodiment and fluent tool -use.

29. Even the minds that, in the movie The Matrix, populate the Matrix ‘dream-world’ count as profoundly embodied, since those minds display the same adaptive ecological control abilities as our own. For example, a Matrixer could learn to fluently incorporate a Stelarc-style third-hand, or to use a thought-controlled robot arm. It is just that the physical dynamics of the new components would be held in place by the Machines’ computer simulation rather than worldly physics.

30. These kinds of minds are promiscuously body-and-world exploiting. They are forever testing and exploring the possibilities for incorporating new resources and structures deep into their problem-solving regimes. They are indeed the minds of Natural-Born Cyborgs (shameless plug): systems continuously re-negotiating their own limits, components, and (as we’ll next see) data-stores and interfaces.

31. Two Questions √ 2.Embodiment: What Really Matters?√ 3.Sensing 4. Incorporation versus Use 5.Mind and Reason 6.A Science of Hybrids?

32. Tactile Visual Sensory Substitution (TVSS) Work by Paul Bach y Rita and colleagues

33. STOP feeling the tickling on the back and START to report rough, quasi-visual experiences of looming objects etc. After a while, a ball thrown at the head causes instinctive and appropriate ducking. The causal chain is ‘deviant’: it runs via the systematic input to the back. But the nature of the information carried, and the way it supports the control of action, is distinctive of the visual modality.

34. “TVSS systems [have] been sufficient to perform complex perception and ‘eye’-hand co-ordination tasks. These have included face recognition, accurate judgment of speed and direction of a rolling ball with over 95% accuracy in batting the ball as it rolls over a table edge, and complex inspection-assembly tasks”. Bach-y-Rita 2001

35. The head-mounted camera was under the subject’s motor control. This meant that the brain could, in effect experiment via the motor system, giving commands that systematically varied the input, so as to begin to form hypotheses about what information the tactile signals might be carrying. For example, you hear someone approaching from the left, turn the camera that way, and see what tactile pattern corresponds to this event…

36. The motor system operating the camera could be changed, eg to a hand-held camera, with no loss of acuity. The touch pad, too, could be moved to new bodily sites. Also, there was no confusion: an itch scratched under the grid caused no ‘visual’ effects. (see Bach y Rita and Kercel “Sensory Substitution and the Human-Machine Interface” Trends in Cognitive Sciences 7:12:2003)

38. Leprosy patients who have lost feeling in their hands. Fitted with a sensor-laden glove that transmits signals to a forehead mounted tactile disc-array, they report feeling sensations of touch at the fingertips . This is because the motor-control over the sensors runs via commands to the hand, so the sensation is projected to that site. Bach y Rita and Kercel op cit. (notice that this has clear implications for tele-presence based touch, etc).

39. Tactile Flight Suit (US Navy) Jacket delivers small puffs of air controlled by complex sensors that determine if a plane or helicopter is tilting to the right or left or forward or backward. The pilot feels a puff-induced vibrating sensation on the side of the body corresponding to the direction of tilt, and can control the vehicle’s response by moving their body so as to cancel the puff/vibration.

40. The suit is so good at transmitting and delivering information in an intuitive way that it allows even inexperienced helicopter pilots to perform difficult tasks such as holding the helicopter in a stationary hover, while military fighter pilots can use it to fly blindfold. The suit thus rapidly links the pilot to the aircraft in the same kind of closed loop interaction that linked Stelarc and the third hand, or the monkey and the robot arm, or the blind person and the TVSS system While wearing the suit, the helicopter itself behaves very much like an extended body/sensory sheath for the pilot.

41. What matters, in each case, is the provision of closed-loop signaling so that motor commands affect sensory input. What varies is the amount of training (and hence the extent of deeper neural changes) required to fully exploit the new agent-world circuits thus created.

42. The specific details of the (old or new) circuitry by which the world is engaged fall ‘transparent’ in use. The conscious agent is aware of the oncoming ball, not of seeing the ball, or (by the same token) of using a tactile substitution channel to detect the ball. The pilot becomes aware of the plane’s tilt and slant, not of the puffs of air…

43. Integrated but constantly negotiable platforms of sensing, moving and (as we’ll see later) reasoning. Platforms able to fluidly incorporate new bodily and sensory kit so as to engage (in the service of goal-directed activity) a larger and potentially hostile world. = Profoundly Embodied Agents

44. Two Questions √ 2.Embodiment: What Really Matters?√ 3.Adaptive Ecological Control√ 4. Incorporation versus Use 5.Mind and Reason 6.Can there be a Science of Hybrids?

45. You are making quite a song and a dance out of this, what with talk of “incorporation of new bodily and sensory kit” and so on. But we all know we can use tools and stuff, and learn to use them better (more ‘transparently’ if you must). Why talk of extended bodies and reconfigured users, rather than talk of the same old embodied user just being in command of a new tool?

46. Any control system that is highly engineered so as to be able to learn to make maximal problem-simplifying use of anopen-ended varietyof internal, bodily, or external sources of order. = dynamically adaptive ecological control

47. “Pre-motor, parietal and putaminal neurons that respond both to somatosensory information from a given body region (ie the somatosensory Receptive Field; sRF) and to visual information from the space (visual Receptive Field;vRF) adjacent to it” Maravita and Iriki “Tools for the body (schema)” Trends in Cognitive Sciences vol 8:2:2004, p. 79 For example, some respond to somatosensory stimuli (light touches) at the hand AND to visually presented stimuli near the hand, so as to yield an action-relevant coding of visual space.

48. After 5 minutes of rake-use, the responses of some bi-modal neurons whose original vRFs picked out stimuli near the hand now expanded to include the entire length of the tool, “as if the rake was part of the arm and forearm” (op cit). And other bi-modal neurons, that previously responded to visual stimuli within the space reachable by the arm, now had vRFs that covered the space accessible by the arm-rake combination.

49. “Such vRF expansions may constitute the neural substrate of use-dependent assimilation of the tool into the body-schema, suggested by classical neurology” And note that: “any expansion of the vRF only followed active, intentional use of the tool not its mere grasping by the hand” (op cit 80,81). (see also work on haptic and dynamic touch- eg Turvey and Carello (1995))

50. In a patient whose neglect selectively affected space close to (one side of) the body, use of a stick extends the neglect to the whole area reachable by the tool. See Berti and Frassinetti “When Far Becomes Near: Re-mapping of space by tool use” Journal of Cognitive Neuroscience 12 (2000) 415-420