Why do things look the way they do?

1. Why do things look the way they do? • Because that’s they way they really are!? That question is misleading because “how things look” is ill defined in the absence of further theoretical analysis. • In this class we will look at a somewhat related question: • What does the way things appear to us in our conscious experience tell us about how vision and mental imagery works and about the nature of the representations we construct when we see or visualize? • LaGrande Illusion is the belief that we have access to the form of representation we construct in episodes of cognition • Recent example: Wegner, in The Illusion of Conscious Will, Argues that what we are conscious of is not what causes our voluntary acts • ZP: “…the ‘illusion’ is really that the experience we have of causing certain things to happen explains why it happened”

2. This is what our conscious experience suggests goes on in vision…

3. This is what the demands of explanation suggests must be going on in vision…

4. Subjective experience suggests that when we see, we are examining an inner display Are there any reasons (other than conscious experience) for taking this view seriously? • The completion of percepts from partial information • Blind spot, resolution and color sensitivity • ‘amodal completion’ (most scene objects are occluded) • Completions have real behavioral consequences • ‘illusory contours’ and amodal completions act in many ways like real figures • Visual representations appear to be panoramic, stable, and in an allocentric frame of reference • The superposition theory has a lot of intuitive appeal

5. Completions … Where’s Waldo?

6. Standard view of saccadic integration by superposition

7. Neuroscience also suggests that the visual pattern is displayed on the surface of the cortex There is a topographical projection of retinal activity on the visual cortex of the cat and monkey. We will return to this later in connection to theories of mental images Tootell, R. B., Silverman, M. S., Switkes, E., & de Valois, R. L. (1982). Deoxyglucose analysis of retinotopic organization in primate striate cortex. Science, 218(4575), 902-904.

8. The superposition view fails O'Regan, J. K., & Lévy-Schoen, A. (1983). Integrating visual information from successive fixations: Does trans-saccadic fusion exist? Vision Research, 23(8), 765-768.

9. Vision consists of many modules with restricted lines of inter-communication

10. Not every way of creating a contour yields the same result

11. The visual experience may be global but interpretation is local Partly Ambiguous Unambiguous #1 Unambiguous #2

12. Visual interpretation is local

13. Visual interpretation is local Coordinating the interpretations among parts is more difficult if they are not both visible at the same time

14. When temporal integration occurs is shows the effect of memory load Higher memory load Lower memory load

15. When temporal integration occurs is shows the effect of memory load

16. When temporal integration occurs order of presentation is important

17. Preview of the thesis of the modularity of early vision • Later I will argue that vision is not cognitively penetrable – it is an encapsulated module with its own logic and rules (some of which are explainable on the basis of a wired-in harmony with our kind of world).

18. Amodal completion is not “rational”?Early vision has its own “rules”

19. The correspondence problem for apparent motion is also not “rational” The grey disks correspond to the first flash and the black ones to the second flash. Which of the 24 possible matches will the visual system select as the solution to this correspondence problem? What principal does it use? Curved matches Linear matches

20. Here is how it actually looks

21. Yantis use of the “Ternus Configuration” to demonstrate the early visual effect of objecthood Short time delays result in “element motion” because the middle object is seen to persist as the “same object” so it does not appear to move

22. Long time delays result in “group motion” because the middle object is not perceived as persisting but is perceived as a new object each time it reappears

23. But with long delays, if the disappearance appears to be due to occlusion by an opaque surface, objects appear to endure so the display behaves like the short delay display

24. With long delays if the disappearance is perceived as due to occlusion by an opaque surface the display behaves like the short delay one (if the timing and attention focus is just right!)

25. Errors in recall suggest how visual information is encoded Children have very good visual memory, yet often make egregious errors of recall • Errors in relative orientation often take a canonical form • Errors in reproducing a 3D image preserve 3D information

26. Errors in recall suggest how visual information is encoded • Children more often confuse left-right than rotated forms • Errors in imitating actions is another source of evidence

27. Ability to manipulate and recall patterns depends on their conceptual, not geometric, complexity • Difficulty in superimposing shapes depends on they are conceptualizedLook at first two shapes and superimpose them in your mind; then draw (or select one) that is their superposition

28. Many studies have shown that memory for shapes is dependent on the conceptual vocabulary available for encoding theme.g., recall of chess positions by beginners and masters

29. Finally, the phenomenology of seeing (including its completeness, its filled-out details, and its panoramic scope) turns out to be an illusion! • We see far less, and with far less detail and scope, than our phenomenology suggests • Objectively, outside a small region called the fovea, we are colorblind and our sight is so bad we are legally blind. The rest of the visual field is seriously distorted and even in the fovea not all colors are in focus at once. • More importantly, we register surprisingly little of what is in our field of view. Despite the subjective impression that we can see a great deal that we cannot report, recent evidence suggests that we cannot even tell when things change as we watch.

30. Examples of attentional inhibition • Negative Priming (Treisman & DeShepper, 1996). • Is there a figure on the right that is the same as the figure on the left? • When the figure on the left is one that had appeared as an ignored figure on the right, RT is long and accuracy poor. • This “negative priming” effect persisted over 200 intervening trials and lasted for a month!

31. Another negative attention effect: Inattentional Blindness

32. Inattentional Blindness • The background task is to report which of two arms of the + is longer. One critical trial per subject, after about 3,4 background trials. Another “critical” trial presented as a divided attention control. • 25% of subjects failed to see the square when it was presented in the parafovea (2° from fixation). • But 65% failed to see it when it was at fixation! • When the background task cross was made 10% as large, Inattentional Blindness increased from 25% to 66%. • It is not known whether this IB is due to concentration of attention at the primary task, or whether there is inhibition of outside regions.

33. Change Blindness:What changes between flashes? Harborside Airplane Helicopter Dinner Farm scene Paris corner

34. Where does this leave us? • Should we conclude that seeing is a process of constructing conceptual descriptions? • Most cognitive scientists and AI people would say yes, although there would be several types of exception. • There remains the possibility that for very short durations (e.g. 0.25 sec) there is a form of representation very like visual persistence – sometimes called an ‘iconic storage’(Sperling, 1960). • From a neuroscience perspective there is evidence of a neural representation in early vision – in primary visual cortex – that is retinotopic and therefore “pictorial.” Doesn’t this suggest that a ‘picture” is available in the brain in vision? • We shall see later that this evidence is seriously misleading and does not support a picture theory of working memory or LTM • A major theme of this course will be to show that an important mechanism of vision is notconceptual but causal: Visual Indexes • Many people continue to hold a version of the “picture theory” of mental representation in mental imagery. More on this later.

35. Conceptual & Methodological Problems: What does “see” or “looks like” mean? • There is a problem about what it means to see: Ordinary usage confounds: seeing X, seeing X as R, seeing that P, and believing that P. Distinguishing these is a cottage industry in philosophy. • Some examples of the prescientific sense of see and why it is a problem. See examples… • Ordinary language also confounds the content and the vehicle of perception: what perceptual representations are about and what properties the representations themselves have • There is a serious problem about what to make of phenomenology – about the conscious experience of seeing. In view of how misleading subjective experience can be, and in view of the fact that what progress there is cognitive science has been made by ignoring the distinction between processes of which we are consciously aware and those of which we have no conscious experience, cognitive science, unlike philosophy, has tended to shy away from the evidence of introspection.

36. Conceptual & Methodological Problems in the study of Visual Perception • Part of the puzzle about vision rests on the failure to make certain distinctions (such as those alluded to earlier: between seeing and believing, between content and form). These two distinctions will play a major role in our later discussion. • The question posed at the beginning of this lecture, Why do things look the way they do? asks why the appearance is conceptualized (or described) a certain way. The difficulty with this idea is dramatized by the Wittgenstein story.

37. What does “how things look” mean? • We speak of someone “looking sick,” or “looking happy” or “looking like it’s going to snow.” This informal sense of looking is too diverse to be useful. e.g., Droodles • Wittgenstein’s anecdote (as told by Tom Stoppard), goes:Meeting a friend, a philosopher says, “Tell me, why do people always say it was natural for men to assume that the sun went around the earth rather than that the earth was rotating?” His friend said, “Well, obviously, because it just looks as if the sun was going round the earth.” To which the philosopher replied, “Well, what would it have looked like if it had looked as if the earth was rotating?” • The problematic nature of How Things Look will become even more apparent when we discuss mental imagery later. As a take-home exercise you might think about “Why do mental images look like what they are images of?”

38. Conceptual & Methodological Problems in the study of Visual Perception • An important distinction that I will address only briefly (but is covered in Chapters 2 and 3 of S&V) is between seeing and coming to have a belief about what one is seeing. This is crucial to the claim of Cognitive Impenetrability of Vision or the view of vision as an architectural module. • The advent of Signal Detection Theory alerted us to the fact that signal detection involves at least two stages: a detection stage and a response selection or decision stage, the latter of which is the only stage thatis cognitively penetrable.