Images Chapter 6

1. ImagesChapter 6 Paul Thagard (2005). Mind: An Introduction to Cognitive Science. 2nd Edition. MIT Press FCAC, University of Hyderabad

2. Outline • Imagistic Mental Representations • Evaluation of Images as a Representation scheme • Representational power • Computational power • Problem Solving: Planning, Decision, Explanation • Learning • Language • Psychological Plausibility • Neurological Plausibility • Practical Applicability FCAC, University of Hyderabad

3. Introduction • What are (mental) images? • In the SRK LH complex, how many rooms do we cross before we get to our room (on Tuesdays)? • Is the turn-stile nearer to the gate at the Main Entrance or at the Small gate of University? • Many people remember such things by constructing mental images and doing visual counting and estimation! FCAC, University of Hyderabad

4. Introduction … • The general theme: • People have mental images of situations • Processes such as scanning and rotation operate on those images • The processes for constructing and manipulating images produce the intelligent behaviour FCAC, University of Hyderabad

5. Introduction … • Cognitive Scientists interested in imagery concentrated on visual imagery, but • We can not ignore images connected to non-visual modalities • Auditory imagery • Taste and Smell imagery • Does Rasmalai taste like Malpau? • Tactile imagery • Does growth of a beard feel like sandpaper? • Motor imagery (mirror neurons or video neurons!) • How do you execute a square drive? Or How do you execute drop-serve in badminton? • Emotional imagery • How did you feel when you first heard that you got admission into University of Hyderabad? FCAC, University of Hyderabad

6. Brief History • Ancient Greeks • Aristotle: no thought without images. • Modern philosophers: • Rationalists (e.g. Descartes) and empiricists (e.g. Locke) agreed that images are important for thinking. • Descartes argued that some thought was non-imagistic, distinguishing imaging and conceiving. • Berkeley also pointed out that it is difficult to image 'triangularity' (as opposed to some specific triangle). • In the 19th century (beginnings of psychology – Wundt and James) • strong claims about the importance of imagery again became popular with the introspectionists. • Beginning of the 20th century • Behaviorism: imagery was ignored, if not considered irrelevant (because it was exceedingly 'internal' in nature). FCAC, University of Hyderabad

7. Modern Views • 1950s-60s (post-cognitive revolution) • focus was on language-like, not picture-like processing for understanding cognition. • the applicability of the computer metaphor to imagistic processing wasn't immediately clear • 1970s-Present • No one has argued that imagery replaces verbal thought or accounts for all aspects of cognition. • The question is whether there is anything special about mental images, or if/how we can/should talk about mental images at all. • Most cognitive scientists take it that some processing is best understood in terms of mental images (though this is not unanimous). • They are interested in understanding: 1. the relation between images and perception; and 2. the functional structure of images themselves FCAC, University of Hyderabad

8. Visual Representation • For people with normal vision, seeing things seems automatic and easy. • Computer vision deals with computers understanding images, a very tough problem, considering • Changes in lighting conditions across the scene • All the occlusions • Enormous amount of processing done, before our brains can interpret an image • Edge detection, figure-ground separation, depth and distance perception from binocular images, colour, motion, etc. • Finally, mental images do not require that objects are present on the eye (retina)! • We can retrieve and manipulate images to accomplish many mental tasks FCAC, University of Hyderabad

9. Necker Cube FCAC, University of Hyderabad

10. Representational Power • A picture is worth a thousand words! FCAC, University of Hyderabad

11. Representational Power • Not everything is representable visually • Abstract sentences • Justice is fairness. • General (universally quantified) sentences • All dinosaurs are extinct. • Awkward sentences • Smoking causes cancer • Some causal statements • If you get a cold, then you will have cough. • So, visual images complement but do not replace verbal representations. • What is the Structure of Mental images? • Array-like • Neural networks FCAC, University of Hyderabad

12. Computational Power • Visual thinking is useful for problems that depend on visual appearance or spatial relationships. • Computational procedures • Inspect • Imagine a plate that has a knife to the left of it and a fork to the right of it. Is the knife to the left or right of the fork? • Find • Where is switch located for retracting the projection screen in DCIS seminar room? • Zoom • Does a frog have a tail? • Rotate • What does “E” look like when it is flat on its back? • Transform • Imagine the letter “B”. Rotate it 90 to the left. Put a triangle the same width as the rotated “B” directly below it and pointing down. Remove the horizontal line. What do you see? FCAC, University of Hyderabad

13. Computational Power (contd.) • Planning: How to accomplish a goal from an initial or current state? • Visual plan using mental maps that encode spatial relations of the places / objects. • Ex: Suppose you have many errands to do: checking scholarship status in admin, changing clothes, attending class in LHC, dropping assignments in the mail box, etc. • Problem solvers often employ diagrams as an external aid to supplement the mental images. FCAC, University of Hyderabad

14. Computational Power (contd.) • Decision: Selection of best means to get to the goal. • Little research done in imagery-based decision making • Ex: Suppose you are deciding whether to wear your blue or red shirt, you might imagine how you look and then decide. FCAC, University of Hyderabad

15. Computational Power (contd.) • Explanation: In this, you are trying to understand why something happened. • Visual reasoning may be useful in generating explanations. • People (Nikola Tesla) use mental imagery to diagnose machine faults • Explanation of how Brazil fits into and under West Africa, by visualizing the South America and Africa on the map and bringing them together: Continental Drift hypothesis. • Visual explanation complements verbal reasoning. FCAC, University of Hyderabad

16. Computational Power (contd.) • Learning: Ability to use experience to improve performance. • Athletes are often coached to improve their performance by using imagery • People are asked to watch videos of maestros to perfect their skills • Images are useful for generalization • Category of elephant formed from mental pictures • Abductive learning can also be visual • You find a long scratch on your car, you can generate various verbal explanations but you might also construct a mental movie in which someone drives up beside you in the parking lot and opens a door that scrapes along your car just where the scratch appears. • Archaeologists use visual abduction when they generate explanations of ancient objects. FCAC, University of Hyderabad

17. Computational Power (contd.) • Language: How do concepts underlie our ability to use language? • Language is essentially verbal, how does imagery help! • Visual metaphors! • Ex: He’s up today. She’s on top of her job. • Metaphors can also tie more than one kind of sensory representation • Ex: “loud clothes” • Some researchers (Langacker) propose cognitive grammar that takes metaphor and imagery as central to mental life, including language processing. • Ex: The meaning of “trumpet” may be tied in part to an auditory image of the sound a trumpet makes. FCAC, University of Hyderabad

18. Psychological Plausibility • Psychological plausibility assessed through experiments on how people use imagery • Several experiments conducted to show that visual imagery is part of human thinking. FCAC, University of Hyderabad

19. FCAC, University of Hyderabad

20. FCAC, University of Hyderabad

21. Is this transformed from Normal or Rotated R? FCAC, University of Hyderabad

22. Is this transformed from Normal or Rotated R? FCAC, University of Hyderabad

23. Is this transformed from Normal or Rotated R? FCAC, University of Hyderabad

24. Is this transformed from Normal or Rotated R? FCAC, University of Hyderabad

25. Psychological and Neurological Plausibility • Some Evidence for Perception/imagery have shared mechanisms • Brain damage effects (Case Study) • Same areas ‘light up’ (Kosslyn, 1995) (Auditory imagery) FCAC, University of Hyderabad

26. Case Study • Case study: Farah, Beauvois and Saillant published a paper describing their intensive study of one patient (1985, 265-6). This patient had "verbal-visual disconnection syndrome” • could not associate colour names with visual impressions of colours • no problems working with colours purely visually (i.e. could see when two colour samples were identical) • could use colour terms alone (e.g. answering "What colour is associated with envy?") • when shown a black and white picture of object could select a colour sample that matches the object's actual colour • if asked what colour a particular object was, relying only on his powers of mental imagery, failed. • Therefore, had the same problem with mental images that he had with visual perceptions: he could associate both with other visual perceptions, but he could associate neither with verbal representations. • The patient was injured in a brain area known to be active during visual perception, the study suggests that mental imagery and visual perception are controlled by the same brain areas. FCAC, University of Hyderabad

27. Auditory Imagery FCAC, University of Hyderabad

28. Structure of Images • Functional structure of mental images • What characteristics do pictures/models and images share? • Mental rotation (Shepard and Metzler, 1971) • Mental folding (Shepard and Feng) FCAC, University of Hyderabad

29. FCAC, University of Hyderabad

30. FCAC, University of Hyderabad

31. Structure of Mental Images • Kosslyn claims we 'can manipulate these models much like we do actual objects' ('mental model'). • Scanning experiments (Kosslyn, et al., 1973, 1978): • 1) Imagined map (slide) • 2) Focus on the front or back of an animal determine if the animal has a property. • Mental resolution (Kosslyn, et al., 1978, 1980): • 1) Imagined animals at different relative sizes (two slides) • 2) Imagined animals at the same size. • Mental screen size (Kosslyn, et al., 1983): • Large objects ‘overflow’ screen at closer imagined dist. FCAC, University of Hyderabad

32. FCAC, University of Hyderabad

33. FCAC, University of Hyderabad

34. Practical Applicability • Imagery is very important in education • It is important to educate students how to use images better and effectively. • Mathematical intuition sometimes depends on visual and spatial representations (Dehaene et al., 1999). Important in teaching mathematics. • Memory enhancement programs rely on imagery. • Engineering design • Architects, engineers, product designers use visual representations such as blue prints, sketches – perhaps mental imagery is part of their creative mental processes. • Content Based Image Retrieval (CBIR) is an emerging area in image processing and computer vision. FCAC, University of Hyderabad

35. References • Paul Thagard (2005). Mind: An Introduction to Cognitive Science. 2nd Edition. MIT Press. • Chris Eliasmith’s Phil/Psych 256 PPTs from University of Waterloo, Canada FCAC, University of Hyderabad