Imagery slides

1. Imagery slides

2. Imagery and Memory • Memory Examples: Dual Code Theory • To recall Y you must first recall X • Windows, doorknob, glasses, other facial features, global-to-local • But: Something like the same thing happens in recall of alphabet letters and many other memorized lists • Imageabilityrating are more effective than frequency of occurrence or frequency of co-occurrence in paired-associates learning.

3. Vision is clearly involved when images are superimposed onto vision • Many experiments show that when you project an image onto a display the image acts very much like a superimposed display • Shepard & Podgorny (paper folding task…) • Interference effects (Brooks) • ControvercialPerky effect: Perception or response bias?

4. Project an image onto a perceived form

5. Brooks’ spatial interference study Respond by pointing to symbols in a table or by saying the words left or right

6. Perception or attention effects? • Many impressive imagery effects can be plausibly attributed to attention • Bisiach widely-cited finding on visual neglect • Bartolomeo, P., & Chokron, S. (2002). Orienting of attention in left unilateral neglect. Neuroscience and Biobehavioral Reviews, 26(2), 217-234. • Dulin, D., Hatwell, Y., Pylyshyn, Z. W., & Chokron, S. (2008). Effects of peripheral and central visual impairment on mental imagery capacity. Neuroscience and Biobehavioral Reviews, 32(8), 1396-1408. Does neglect require vision? • Chokron, S., Colliot, P., & Bartolomeo, P. (2004). The role of vision in spatial representations. Cortex, 40, 281-290.

7. We can to some extent control our attended region Is an image being projected onto a percept, or just a selective attention? Farah, M. J. (1989). Mechanisms of imagery-perception interaction. Journal of Experimental Psychology: Human Perception and Performance, 15, 203-211.

8. Shepard & Podgorny experiment Both when the displays are seen and when the F is imagined, RT to detect whether the dot was on the F is fastest when the dot is at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.

9. Similarities between perception of visual scenes and ‘perception’ of mental images • Judgments from mental images • Shape comparisons (of states: Shepard & Metzler) • Size comparisons (Weber fraction or ratio effect) • What do they tell us about the format of images? • But this applies to nonvisual properties (e.g., price, taste)

10. More demonstrations of the relation between vision, imagery (and later action) • Images constructed from descriptions • The D-J example(s) • Perception or inference/guessing • But there are even more persuasive counterexamples we will see later • The two-parallelogram example • Amodal completion • Reconstruals: Slezak

11. Dynamic imagery Imagining actions: Paper Folding

12. Mental rotation Time to judge whether (a)-(b) or (b)-(c) are the same except for orientation. Time increases linearly with the angle between them (Shepard & Metzler, 1971)

13. What do you do to judge whether these two figures are the same shape? Is this how the process looked to you? When you make it rotate in your mind, does it seem to retain its rigid 3D shape without re-computing it?

14. Mental rotation – the real story In mental rotation the phenomenology motivates the theory of “rotation” – but what the data actually show is that, Mental rotation is only found when the comparison figures are enantiomorphs or if the difference between figure pairs can only be expressed in figure-centric coordinates eg. they are 3D mirror-images No rotation occurs if the figures have landmarks that can be used to identify the relations among their parts. Records of eye movements show that mental rotation is done incrementally: It is not a holistic rotation as often reported. If fact even the phenomenology is not of a smooth continuous rotation. The “rate of rotation” depends on the conceptual complexity of both the figure and comparison task so that, at least, is not a result of the architecture (Pylyshyn, 1979). There are even demonstrations that it depends on how the subject interprets the figure (Kosslyn, 1994).

15. Mental Scanning • Hundreds of experiments have now been done demonstrating that it takes longer to scan attention between places that are further apart in the imagined scene. In fact the relation is linear between time and distance. • These have been reviewed and described in: • Denis, M., & Kosslyn, S. M. (1999). Scanning visual mental images: A window on the mind. Cahiers de Psychologie Cognitive / Current Psychology of Cognition, 18(4), 409-465.

16. Studies of mental scanningDoes it show that images have metrical space? Does this show that images are spatial, or have spatial properties, or that they “preserve metrical spatial properties”?(Kosslyn, S. M., T. M. Ball, et al. (1978). "Visual images preserve metric spatial information: Evidence from studies of image scanning." Journal of Experimental Psychology: Human Perception and Performance 4: 46-60.

17. The idea of images being in some sense spatial is an interesting and important claim • I will discuss this claim at some length later because it reveals a deep and all-consuming error that runs through all imagery theorizing – by psychologists, neuroscientists and philosophers. • This is in addition to the errors I discussed earlier: The idea that subjects understand the task of imagining something to be the task of pretending they are seeing it, and the idea that certain properties of the world are properties of the image (the intentional fallacy)

18. Constructing an image • What determines what the image is like when it is constructed from memory or from knowledge? • After constructing an image can you see novel aspects of the imagined situation? • Examples

19. height Time since drop Examples to probe your intuition and your tacit knowledge Imagine seeing these events unfolding… • You hit a baseball. What shape trajectory does it trace? It is coming towards you: Where would you run to catch it? If you have ever played baseball you would have a great deal of “tacit knowledge” of what to do in such (well studied) cases. • You drop a rubber ball on the pavement. Tap a button every time it hits the ground and bounces. Plot heightvstime. • Drop a heavy steel ball at the same time as you drop a light ball (a tennis ball), e.g., from the leaning tower of Pisa. Indicate when they hit the ground. Repeat for different heights. • Take a clear glass containing a colored liquid. Tilt it 45º to the left (counter-clockwise). What is the orientation of the liquid? What is responsible for the pattern shown here?

20. What color do you see when two color filters overlap? ?

21. Where would the water go if you poured it over a full beaker of sugar? Is there conservation of volume in your image? If not, why not?

22. Seeing Mental Images • Do images have size? • Can we say that one image is larger than another? • If so, what properties do we expect the smaller/larger image to have?

23. Do mental images have size?Imagine a very small mouse. Can you see its whiskers? Now imagine a huge mouse. Can you see its whiskers?

25. Connect each corner of the top parallelogram with the corresponding corner of the bottom parallelogram Do this imagery exercise:Imagine a parallelogram like this one Now imagine an identical parallelogram directly below this one • What do you see when you imagine the connections? • Did the imagined shape look (and change) like the one you see now?

26. Slezak figures • Pick one (or two) of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.

27. Slezak figures rotated 90o

28. P 29 Space

29. Images and the representation of spatial properties • We need to understand what it could mean for a representation to be spatial. • At the very least it must mean that there are constraints placed on the form of the representation that do not apply when the representation is not spatial.

30. The idea that images are in some sense spatialis an interesting and important claim • I will return to this claim later because it reveals a deep and ubiquitous error that runs through most (all?) imagery theorizing – by psychologists, neuroscientists and philosophers. This is the error of mistaking descriptive adequacy with explanatory adequacy. Let’s call this conflating, the missing constraint error. • This is in addition to the two errors I discussed earlier: • Ignoring the fact that the task of imagining something is actually the task of pretending you are seeing it, and • The mistaken assumption that certain properties of the world are properties of the image (the intentional fallacy)

31. Both vision and visual imagery have some connection to the motor system • There are a number of experiments showing the close connection between images and motor control* • You can get Stimulus-Response compatibility effects between the location of a stimulus in space and the location of the response button in space, • Ronald Finke showed that you could get adaptation with the position of the misperceived hand that was similar to adaptation to displacing prism goggles, • Both these findings provide support for the view that the spatial character of images comes from somethingbeing projected onto a concurrently perceived scene and then functioning much as objects of perception. • This is the main new idea in Chapter 5 of Things & Places) an image imagined

32. Recall the studies of mental scanning… Does this result show that images havemetrical properties? Does this result show that images havespatial properties?  We showed that the image scanning effect is Cognitively Penetrable But the way we compute the time it takes to scan across an image is by imagining something moving across the real perceived display. Without this display, we could not use our time-to-collision computation to compute the time to cross various distances on the image because there are no actual distances on the image!(Pylyshyn & Cohen, 1999)

33. Using a concurrently perceived room to anchor FINSTs tagged with map labels

34. The Spatial character of images What does it mean to say that images are spatial? • It means that certain constraints hold among spatial measures (e.g., axioms of geometry and measure theory, such as triangle inequality, symmetry of distances, Euclidean axioms, Pythagoras’ theorem…} • That certain constraints hold among “distances”, that certain relations can be defined among these distances (e.g., ‘between’, ‘farther than’), that Newtonian Physics holds between the terms that are used in explanations (e.g., distances and time). • That mental images and motor control interact with one another to some degree – so you can “point to” objects in your image. • Certain visual-motor ‘reflexes’ are automatic or preconceptualThey are computed within the encapsulated Visual Module • Preconceptual motor control is not sensitive to visual illusions, relative to control that is computed by the cognitive (‘seeing as’) system.

35. Mental images as “depictive” representations • “A depictive representation is a type of picture, which specifies the locations and values of configurations of points in a space. • The space in which the points appear need not be physical but can be like an array in a computer, which specifies spatial relations purely functionally. That is, the physical locations in the computer of each point in an array are not themselves arranged in an array; it is only by virtue of how this information is “read” and processed that it comes to function as if it were arranged into an array…. • Depictive representations convey meaning via their resemblance to an object. • When a depictive representation is used, not only is the shape of the represented parts immediately available to appropriate processes , but so is the shape of the empty space … [and] one cannot represent a shape in a depictive representation without also specifying a size and orientation….”

36. Form vs Content of images • As in earlier discussion, one must be careful in distinguishing form from content. We know that there is a difference between the content of images and the content of other (nonimaginal) thought: Images concern sensory appearances while ‘propositions’ can express most* other contents. • In attributing a special form of representation to images one should ask whether some symbolic system (e.g., sentences of LOT) would not do. Simplicity (Occam’s Razor) would then prefer a single format over two, especially if the one format is essential for representing thoughts and inferences [Fodor, J. A. and Z. W. Pylyshyn (1988). "Connectionism and cognitive architecture: A critical analysis." Cognition 28: 3-71.] • The most promising contents that might require different forms of representation are those that essentially represent magnitudes. Of the magnitudes most often associated with images are spatial ones. • There has been a long-standing debate in Artificial Intelligence concerning the advantages of logical formats vs other symbol systems vs something completely difference (procedure).

37. Thou shalt not cheat • There is no natural law that requires the representations of time, distance and speed to be related according to the motion equation. You could just as easily imagine an object moving instantly or with constant acceleration or with any motion relation you like, since it is your image! • There are two possible reason why the observed relation Actual Time =Representation of distance Representation of speed typically holds in an image-scanning task: • Because subjects have tacit knowledge that this is what would happen if they viewed a real display, or • Because the matrix is taken to be a simulation of a real physical display, as it often is in computer science. • Notice that in the second case the explanation for the Reaction Time comes from the simulated real display and not from the matrix.

38. The missing constraint in appeals to “space” in both scanning and mental rotation What is assumed about the format or architecture of the mental representation in the examples of mental rotation? According to philosopher Jesse Prinz (2002) p 118,“If visual-image rotation uses a spatial medium of the kind Kosslyn envisions, then images must traverse intermediate positions when they rotate from one position to another. The propositional [i.e., symbolic] system can be designed to represent intermediate positions during rotation, but that is notobligatory.” This is a very important observation, but it is incomplete. One still needs to answer the question: What makes it obligatory that the object must ‘pass through intermediate positions’ when rotating in ‘functional space’, and what constitutes an ‘intermediate position’? These terms apply to the represented world, not to the representation!

39. The important distinction between architecture and represented content It is only obligatory that a certain pattern must occur if the pattern is caused by fixed properties of the architecture as opposed to being due to properties of what is represented (i.e., what the observer tacitlyknows about the behavior of what is represented) If it is obligatory only because the theorist says it is, score that as a free empirical parameter that any theory can assume. This failure of image theories is quite general – all picture theories suffer from the same lack of principled constraints.

40. The important distinction between descriptive and explanatory adequacy It is important to recognize that if we allow one theory to stipulate what is obligatory without there being a principle that mandates it, then any other theory can stipulate the same thing. Such a theories are unconstrained so they can fit any possible observation – i.e., they are able to describeanything but explain nothing. A theory that does not explain why some pattern is obligatory can still be useful the way an organized catalog is useful. It may even list the features according to which it is organized. But it does not give an account of why it is organized that way rather than some other way. To do that it needs to appeal to something constant such as a law of nature or a fixed property of the architecture.

41. How are these ‘obligatory’ constraints realized? Image properties, such as size and rigidityare assumed to be inherent in the architecture (e.g., of the ‘display’) That raises the question of what kind of architecture could possibly enforce rigidity of shape? Notice that there is nothing about a spatial display, let alone a functional space, that makes it obligatory that shape be rigidlymaintained as orientation is changed. Such rigidity could not be a necessary property of the architecture of an image system because we can easily imagine that rigidity does not hold (e.g. imagine a rotating snake!). There is also evidence that ‘mental rotation’ is incremental, not holistic, and the speed of rotation depends on the conceptual complexity of the shape and the comparison task.

42. What makes some properties seem “natural” in a matrix but not so natural in a symbolic data structure? A matrix is generally viewed as a two-dimensional structure in which specifying the x and y values (rows and columns) specifies the location of any cell. But that’s just the way it is conventionally viewed. Rows, columns and cells are not actually spatiallocations. In a computer there is no requirement that in getting from one cell to another one mustpass through any other specified cells nor is there any requirement that there be empty cells between any pairs of cells.

43. What makes some properties “natural” in a matrix while not so natural in a symbolic data structure? The main reason it is natural to view a matrix as having spatial constraints is that one is tacitly assuming that it represents some space. Then it is the represented space that has the constraints, not the matrix. Notice the subtle succumbing to the intentional fallacy again! Any constraints that the functional space exhibits are constraints extrinsic to the format. Such constraints reside in the external world which the ‘functional space’ represents. But such extrinsic constraints can be added to anymodel of scanning, including a propositional one.

44. What warrants the ‘obligatory’ constraint? But it is no more obligatory that the relation between distance, speed and time hold in functional space than in a symbolic (propositional) representation. There is no natural law or principle that requires it. You couldimagine an object moving instantly or according to any motion relation you like, and the functional space would then comply with that motion since it has no constraints of its own. • So why does it seem natural for imagined moving objects to traverse a ‘functional space’ than a sequence of symbolic representations of locations? • There are at least two reasons why a ‘functional space’ might seem more natural than a symbolic representation of space, and both depend on (1) subjective experience and (2) the intentional fallacy.

45. Where does the obligatory constraint come from? There are at least two reasons why the following equation holds in the mental image scanning task, even though, unlike in the real vision case, it does not follow from a natural law. Actual Time =Representation of distance Representation of speed • Because subjects have tacit knowledge that this is what would happen if they viewed a real display, and they understand the task to be one of reproducing properties of this viewing, or • Because the matrix is taken to be a simulation of real space. In that case the reason that the equation holds is that it is supposed to be simulating real space and the equation holds in real space. • In that case it is not something about the form of the representation that provides the principled constraint, it’s the fact that it is supposed to be simulating real space which is where the obligation comes from. But the same thing can be done for any form of representation.

46. Why is it ‘natural’ to assume that functional space is like real space? There are several reasons why a functional space, such as a matrix data structure, appears to have natural spatial properties (e.g., distances, size, empty places): • Because when we think of functional space, such as a matrix, we think of how we usually interpret it. • A matrix does not intrinsically have distance, empty places, direction or any other such property, except in the mind of the person who draws it or uses it! • Moving from one cell to another does not require passing through intermediate cells unless we stipulate that it does. The same goes for the concept of ‘intermediate cell’ itself.

47. Why is it ‘natural’ to assume that functional space is like real space? • Because when we think of a functional space, such as a matrix, we think of it as being a way of simulating real space in the model – making it more convenient to build the model which otherwise would require special hardware • This is why we think of some cells as being ‘between’ others and some being farther away. This makes properties like distances seem natural because we interpret the matrix as simulating real space. • In that case we are not appealing to a functional space in explaining the scanning effect, the size effect, etc. The explanatory force of the explanation comes from the real space that we are simulating. • This is just another way of assuming a real space (in the brain) where representations of objects are located in neural space • All the reasons why the assumption of real brain space cannot be sustained in explanations of mental imagery phenomena apply to this version of ‘functional space.’

48. Why is it ‘natural’ to assume that functional space is like real space? • Because what we really want to claim is that images are displayed on a real spatial surface – a blackboard. But to model this we would need to build a hardware display. {An easier way to do this is simply to claim explicitly that there is a display or even simulate one using software(such as Kosslyn, et al. (1979) claim to have done*)}. • This allows us to view some cells as being ‘between’ others and some being farther away. This makes properties like distances seem natural because we interpret the matrix as simulating or standing in for a real spatial display board or screen. • In that case we are not appealing to a functional space in explaining the scanning effect, the size effect, etc. The explanatory force of the explanation comes from the real space that we are claiming and simulating. This is just another way of assuming a real space (in the brain) where representations of objects are located in neural space.

49. Functional space and explanatory power • There is a notion of explanatory power that needs to be kept in mind. It is best illustrated in terms of models that contain empirical parameters, as in fitting a polynomial curve to data. • The general fact about fitting a model to data is that the fewer parameters that need to be estimated from the data to be fitted, the more powerful the explanation. The most powerful explanation is one that does not have to use the to-be-fitted data to tune the model. • In terms of the current example of explaining results of experiments involving mental imagery, appealing to a “functional space” leaves open an indeterminate number of empirical parameters, so it provides a very weak (or vacuous) explanation. • A literal (brain) space, on the other hand, is highly constrained since it must conform to Euclidean axioms and Newtonian physics – otherwise it would not be the space of natural science. But that kind of space implies that images are displayed on a surface in the brain and while that is a logical possibility it is not an empirical one

50. Explanation and Description • Another way to look at what is going on is to think about the difference between a description and an explanation. The two ways of characterizing a set of phenomena appear similar – they both speak of how things are and how they change (think of the Code Box example). • But a description of a system’s behavior can apply to many different types of system with different mechanisms and different causal properties. And the same mechanisms can also produce very different behaviors under different circumstances. Although a general statement of what constitutes scientific explanation and how it differs from description has a long and controversial history, the simple Code Box example will suffice to suggest the distinction I have in mind.