The problem of nonconceptual representation of space and scenario content

1. Lecture 4. Representing Space II: Shortcomings of the inner space proposals and a possible alternative • The problem of nonconceptual representation of space and scenario content • The internalized space options • Internalizing constraints: Marr & Shepard • Functional Space • Literal (Neural) space • The empirical evidence for a special form of reconstructed spatial representation: Some imagery demonstrations and experimental findings

2. A quick summary of some empirical findings involving mental imagery: orWhat people mean when they say images are ‘spatial’ Two general kinds of studies • Studies showing that imagery ‘involves’ the visual system • I will briefly mention these because most of them lead to the ‘images are spatial’ view • Studies showing that image representations have spatial properties

3. Summary of some empirical findings on mental imagery and vision • Imagery and Vision • Early studies of interference between vision and imagery (Brooks) • Activation of visual cortex during imagery • Why is this relevant? • Because V1 is retinotopic! • Image-vision superposition studies • Most of these studies relating imagery and vision are of interest to us only because they were thought to support a picture-theory of images in whch images are laid out in space

4. Is mental imagery is carried out by the early (modular) visual system? Suppose that mental imagery involved the visual system, what would that tell us about how space is represented? Before drawing conclusions, consider: • Virtually all mental imagery phenomena (scanning, mental rotation, etc) are exhibited by congenitally blind people. • Evidence for the involvement of primary visual cortex is not univocal. The majority of studies report that many cortical areas are involved in mental imagery • There is clear evidence of spared visual imagery with severely damaged visual perception and vice versa (this applies to achromatopsia, agnosia, visual neglect and cortical blindness) • There is good evidence that information in imagery is not reperceived through vision but only inferred

5. Summary of some empirical findings concerning the spatial character of mental images. What makes images “spatial”?

6. Why do we think images are spatial? • The strong intuition that things in an image have a location, in some relevant sense – e.g., you can point to them! • How could a “symbolic” format have scenario content? Directions? • Spatial interaction between location in an image and location in a stimulus • Spatial Interference (Brooks) • Attentional enhancements (e.g., Podgorny & Shepard) • Illusions in combined image-perception displays • Effects of image distances (e.g., scanning) • The connection with the motor system • Eye movements during mental imagery (Brandt &Stark, 1997) • Visual-motor adaptation (Finke) • The Simon effect (S-R compatibility with images)

7. Brooks’ spatial interference study Respond by pointing to symbols in a table or by saying the words left or right

8. Shepard & Podgorny experiment Both when the displays are seen and when the F is imagined, RT to say whether the dot was on the F was fastest when the dot was at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.

9. Visual illusions with projected images Bernbaum & Chung. (1981)

10. Studies of ‘mental scanning’Do they show that images have metrical properties?

11. Other reasons for thinking that images are “spatial”: The connections among vision, imagery and motor-control • Eye movements during imagery are similar to those during visual perception of the same figure • Visual-motor adaptation occurs with imagined hand position that is similar to adaptation with prism glasses (Finke, 1979) • Stimulus-response compatibility effects are observed between location in an image and the side of the response • Patients with visual neglect often show imaginal neglect • Famous 1978 Milan Duomo experiment of Eduardo Bisiach • A patient with post-operative tunnel vision also showed “tunnel imagery”(But by the time she was tested, the patient know how things looked to her) • It takes longer to report details from a “small” image!

12. If images are spatial, where is the space? • Is it in the head? • Is it an abstract representation? • Functional space? • Analogue space? • Tacit (conceptual) knowledge of space? • Is it in the world that is concurrently perceived and in which objects are indexed?

13. Are images displayed in a functional space? • Can we explain the spatial character of images by appealing to a “functional space”? • What properties does a functional space have? • In virtue of what (natural laws, principles, architectural properties of mind) does the functional space have the properties it does?

14. The picture theory and “Functional Space” “The space in which the points appear need not be physical…, but can be like an array in a computer, which specifies spatial relations purely functionally. That is, the physical locations in the computer of each point in an array are not themselves arranged in an array; it is only by virtue of how this information is “read” and processed that it comes to function as if it were arranged into an array (with some points being close, some far, some falling along a diagonal, and so on).”(Kossyln, 1994) Is there any coherent sense of “space” that is neither literal (real, physical) space nor merely knowledge of spatial properties, with no commitment about the format in which this knowledge is represented?

15. In what way is a matrix data structure spatial? Must the cells be accessed in any particular order? Which cells are “between” two given cells? Which cells are collinear? What makes that so?

16. In what way is a matrix data structure spatial? Must the cells be accessed in any particular order? Which cells are “between” two given cells? Which cells are collinear? What makes that so?

17. In what way is a matrix data structure spatial? Must the cells be accessed in any particular order? Which cells are “between” two given cells? Which cells are collinear? What makes that so?

18. Questions about the Functional Space option Where does the “spatial” property of “functional space” come from? Why does a matrix seem to have the spatial properties assumed in picture-theories of mental imagery? • Must Pythagoras’s theorem hold of the matrix? Why? • Why does attention have to pass through “intermediate” cells in getting from A to B. • What makes these cells “intermediate”? In what sense are they “between” A and B? • Why does it take longer to move attention through more cells? • Why does adding point C not change the relation between points A and B? • What makes it the case that when C is added, then the location of B is between A and C?

19. Spatial properties of functional space There are two possibile reasons why spatial properties apparently hold of functional (matrix) space: • The principled alternative. These are principled (lawlike, inherent) properties because the matrix is (tacitly) assumed to be just a simulation of real space, or • The ad-hoc alternative. The properties are not inherent properties of the functional space. Rather, they are just a way of depicting the properties that were observed in the data (this is just like listing the findings without explaining them).

20. Ways of explaining the scanning effect: • If images have spatial extent, then the explanation is straightforward: Time =Distance Speed • If images represent distance, then Time =Representation-of (distance)? Representation-of (speed) Only (1) constitutes an explanation!

21. Aside: Is image space represented in an analogue form? • An analog representation is not subject to the arguments presented against “functional space” • This is a much more difficult proposal than generally recognized – which is why nobody has seriously proposed an analogue representation of space other than space itself • It would require a representation that not only results in Time = Rep(distance)  Rep (speed), but the properties that represent distance and speed would also have to meet many other requirements – e.g., they would have to model Euclid’s Axioms, and much of basic contact physics (since these are obeyed in imagery). • This would still not do because most imagery phenomena are Cognitively Penetrablesothey do not arise from properties of the architecture

22. A puzzle … • Why do we find it natural to assume that brain processes underlying mental imagery share certain properties with what is represented, but not others? • Do the brain processes involved in representing a certain event take the same amount of time as the duration of the imagined event? A monotonic function of real time? • Does the size of the brain state corresponding to a large image reflect the relative size of what is imagined? • Are distances in the neural representation of a certain scene a monotonic function of distances in the scene? • Does the color (volume, mass, temperature) of the brain state vary with the equivalent represented property?

23. The Intentional Fallacy strikes again!

24. The literal space option This was not taken serously until the recent findings that: • The early visual cortex (V1) is spatially mapped, and • Visual cortex is active (as determined by PET and fMRI) when people examine a visual image

25. Activity on visual cortex of primates is retinotopic Tootell, R. B., Silverman, M. S., Switkes, E., & de Valois, R. L. (1982). Deoxyglucose analysis of retinotopic organization in primate striate cortex. Science, 218(4575), 902-904.

26. Does the retinotopic mapping of visual cortex support the view that images are displayed in real space in V1? • The “display” in visual cortex is retinotopic, which means it is narrow in scope, moves with eye movements, and is inherently 2-dimensional! • But scenario content is panoramic and 3-dimensional! • Mental images are experienced as being in allocentric coordinates and are panoramic or even cycloramic • Every imagery finding applies equally in 3D as in 2D • Image size does not map onto size in V1, so the “hard to see” explanation of small images does not work

27. The topographical structure of the visual cortex could not support mental images The 2D mapping of retinal activity in V1 cannot be identified with the mental image which is panoramic, 3-dimensional, dynamic and has many other properties that could not be mapped onto V1, so we would need a different theory for them.

28. …and there are many other differences … • Image content is not visually reinterpreted: it does not have the signature properties of visual interpretation (e.g., amodal completion, spontaneous 3D interpretation, visual ambiguity, Emmert’s law, etc etc) • Image content cannot be accessed in free order • Image content is whatever we want it to be! i.e., it is cognitively penetrable because it is your image! • e.g., our mental scanning findings Exactly the same applies to the effect of image size, visual angle of the mind’s eye, the oblique effect, etc

29. Imagine two parallelograms

30. Imagine two parallelograms

31. Imagine two parallelograms

32. Amodal completion in imagery?

33. Amodal completion in imagery?

34. Slezak figures Pick one of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.

35. Rotated Slezak figures • No subject was able to recognize the mentally rotated figure • Subjects remembered the figures well enough so if they drew it they could recognize the rotated figure PS. There have been differing claims about such results!

36. …and there are many other differences … • Image content is not visually reinterpreted: it does not have the signature properties of visual interpretation (e.g., amodal completion, spontaneous 3D interpretation, visual ambiguity, Emmert’s law, etc etc) • Image content cannot be accessed in free order • Image content is whatever we want it to be! i.e., it is cognitively penetrablebecause it is your image! • e.g., image size? • e.g., our mental scanning findings Exactly the same applies to the visual angle of the mind’s eye, the oblique effect, and dozens of other findings

37. Studies of ‘mental scanning’Do they show that images have metrical properties? (Pylyshyn & Bannon. See Pylyshyn, 1981)

38. An alternative view of the source of spatial properties of images • There is something special about images that are ‘projected’ onto a perceived scene • Can phenomena involving projected images be explained in terms of indexing of objects and binding them to imagined things?

39. Shepard & Podgorny experiment Both when the displays are seen and when the F is imagined, RT to say whether the dot was on the F was fastest when the dot was at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.

40. Visual illusions with projected images Bernbaum & Chung. (1981)

41. Visual adaptation and aftereffects through images (Ron Finke) • Instead of seeing their hand in the wrong location (due to wedge prisms) subjects were told to imagine that there hand (which was actually under a screen) was at some shifted location • After moving their hands a few times and each time imagining them to be at the false location, subjects were allowed to see their hand reaching and it did show a negative aftereffect • But the conditions for adaptation are known to be (1) a mismatch between felt and seen position and (2) a conviction that the hand is actually at the given location. The same result can be obtained by a light held by the hand. • No actual visual properties of the hand are required

42. S-R Compatibility effect with a visual display

43. S-R Compatibility effect with a mental image

44. What happens when you use imagery with your eyes closed? • The existence of a very accurate spatial sense suggests that there are mechanisms similar to visual indexes in other modalities • We don’t need to have a representation of empty places so long as we have coordinate transformation operationslinking gestures and representations of filled locations (tactile perceptions are of tactile objects)

45. Can indexing convert an apparently mental-space phenomenon to a perceived-space phenomenon? • When the imagined stimulus is projected onto a visually perceived background (as in the S-R compatibility example) it is easy to see how it can become a visual phenomenon – subjects must respond to a place selected in the actual display (or near it) • Can we apply this approach when there is no visual input to which to bind imagined objects? • This is where the idea of spatial sense plays a role. We can convert between sensed modalities with coordinate transformation processes.

46. A spatial sense is very different from a mental picture • Exercise of the spatial sense need not be accompanied by a perceptual experience. It is amodal and typically free of any conscious content • It does not require that any properties of imagined objects be represented – just their identity (indicated by binding them to sensory objects) • The spatial sense exploits visual, auditory, kinesthetic and proprioceptive information • The sense of space does not need an internal spatial medium. It arrises purely from the capacity to bind object files (possibly empty) to sensed objects.

47. We don’t need a spatial display in our head if we have a way to pick out and keep track of a few sensory objects in real space • None of the experiments that suggest a spatial display in visual cortex need to appeal to anything more than a small number of imagined locations (e.g., Podgorny Shepherd , Finke, Tlauka, …). • All spatial imagery phenomena have been found in the blind • If we can pick out a small number of occupied locations in real space we can use these to allocate attention or to target eye movements. This is likely what goes on in the Shepard & Podgorny, Bernbaum & Chung and other cases involving combined vision and imagery.

48. We don’t need a spatial display in our head... • If these selected perceived objects are also bound to objects of thought (e.g., “Object Files”) it can result in the objects of our thought having persisting spatial locations and spatial relations (e.g., obeying metrical Euclidean axioms). • This would also explain why some patients with visual neglect sometimes show neglect to the same side of their image(Bisiach & Luzzatti, 1978). • Patients may have projected imagined objects onto real space in the course of their imagining, and their visual neglect resulted in failure to orient to the indexed objects in the neglected field. Neglect appears to involve a failure to orient attention (Bartolomeo & Chokron, 2002) • The patient whose surgery resulted in tunnel vision and tunnel imagery may simply have been showing she knew how things looked to her post-surgery

49. Should you try to rescue the picture theory?

50. Summary • I have presented a view of how conceptual representations might be grounded in causal connections to the world, through a perceptual mechanism that enables individuation, selection, and reference to individual sensory objects • I have proposed that this mechanism picks out a limited number of sensory individualsin the world – individuals that usually turn out to be physical objects, although they are picked out nonconceptually and therefore not as objects • While identifying and tracking individuals in general requires a conceptualapparatus for individuation and identity, perceptual systems are equipped to provide this function in a way that almost always works in our kind of world. Without such a mechanism, grounding of concepts in experience would not be possible