Exploring Synesthesia: Cognitive Neuroscience Insights from the DVF Project

1. Cognitive Neuroscience Talk • Wednesday, 10/8 @ 7:30 pm • Weyerhaeuser Boardroom • Wilma Koutstaal, University of Minnesota • Enhanced Mental Agility in Individuals and Groups

2. DVF Project • Email me a brief description of your question and method by midnight the night before your meeting • This will help me prepare for our meeting • No class on Thursday, 10/9 • On Friday, 10/10, we will discuss what you need to put together in order to run your experiment • Email me materials by Tuesday, 10/14

3. Synesthesia • Sensations from one modality (e.g., sound) are experienced in another modality (e.g., vision) • Perception of a form (e.g., a letter) induces an additional perception (e.g., a color)

4. Types • Grapheme-color • ABCDE12345 • Most common • “When I see this 6, I know it is black. I see it as black, but I recall it and internalize it as green. I get very snippy when I see a red 6 because only a 3 can be red. My friend experiences 6 in a deep blue.”

5. Types • Sound-color • Experience colors when hear sounds • “Beethoven? Mostly sage and gold, that is, if it’s in a major key. Dave Matthews, without a doubt, I hear in shades of brown.” • “My friend sees stars—very yellow ones—when cymbals are played. I find this absurd. Only music in the key of F is yellow, and it’s harvest gold, at that.”

6. Types • Tactile-taste • When touch something, experience a taste • “Velvet isn’t bad, that is, if it is velvet on both sides. And the associations aren’t usually unpleasant, except for terry cloth—but that’s obvious.”

7. Types • Sequence-spatial • Spatial placement of numbers, months of the year, and/or days of the week

8. Types • Sequence-spatial • Spatial placement of numbers, months of the year, and/or days of the week

9. Types • Sequence-spatial • Spatial placement of numbers, months of the year, and/or days of the week

10. Types • Sequence-spatial

11. Synesthesia • Sensations from one modality (e.g., sound) are experienced in another modality (e.g., vision) • Perception of a form (e.g., a letter) induces an additional perception (e.g., a color) • About 1% of population • Mostly female • http://synesthete.org/

12. http://synesthete.org/

13. http://synesthete.org/

14. Nunn et al. (2002) • Sound-color synesthetes(all female) • Age, IQ, and gender-matched controls • 2 tasks • Hear words (vs. tones) • View colors (vs. grays) • Measure brain activity using fMRI

15. Nunn et al. (2002) Controls Synesthetes V4 R L R L Hearing words Seeing colors Overlap

16. Nunn et al. (2002) • Sound-color synesthetes(all female) • Age, IQ, and gender-matched controls • 2 tasks • Hear words (vs. tones) • View colors (vs. grays) • Measure brain activity using fMRI • Same region (V4) active when synesthetes hear words and view colors • Synesthetic perception of color produces activity in color processing region (V4)

17. Brang et al. (2010) • Grapheme-color synesthetes • Age, IQ, and gender-matched controls • Locate V4 and grapheme area (posterior temporal lobe) using MEG • View color patches • View achromatic graphemes

18. Brang et al. (2010) Control Synesthete

19. Brang et al. (2010) Control Synesthete

20. Brang et al. (2010) • Grapheme-color synesthetes • Age, IQ, and gender-matched controls • Locate V4 and grapheme area (posterior temporal lobe) using MEG • View color patches • View achromatic graphemes • V4 responds to achromatic graphemes in synesthetes

21. Brang et al. (2010) • Is grapheme-color synesthesiaperceptual? • Altered cortical connectivity • V4 should activate at same time as grapheme area • Is grapheme-color synesthesiaconceptual? • Over-learned associations persisting since childhood • V4 should activate after grapheme area

22. Brang et al. (2010) Grapheme area V4

23. Brang et al. (2010) • Is grapheme-color synesthesiaperceptual? • Altered cortical connectivity • V4 should activate at same time as grapheme area • Is grapheme-color synesthesiaconceptual? • Over-learned associations persisting since childhood • V4 should activate after grapheme area • V4 activates at same time as grapheme area • Grapheme-color synesthesia is perceptual not conceptual

24. Rouw and Scholte (2007) • Use diffusion tensor imaging (DTI) • Compare anisotropy in grapheme-color synesthetes and controls

25. Rouw and Scholte (2007)

26. Rouw and Scholte (2007) V4

27. Rouw and Scholte (2007) • Use diffusion tensor imaging (DTI) • Compare anisotropy in grapheme-color synesthetes and controls • Greater anisotropy in synesthetes than controls in • Inferior temporal cortex • Connections between V4 and grapheme area? • Parietal cortex • Important for feature binding; binding of shape and color?

28. Daniel Tammet • Born on a Blue Day (2006) • Embracing the Wide Sky (2009) • Thinking in Numbers (2012) • Asperger’s Syndrome • High functioning autism • Savant • Languages • Numbers • Synesthesia • Number-color, shape, texture, emotion,…

29. Nunn et al. (2002) Controls Synesthetes V4 R L R L Hearing words Seeing colors Overlap

30. Weak Synesthesia • We all have to certain degree • “Loud shirt” • “Sharp cheddar”

31. Marks (1974) • Match tones of varying pitch to colors of varying brightness

32. Marks (1974) Tone Pitch (hertz) Color

33. Marks (1974) • Match tones of varying pitch to colors of varying brightness • Higher pitched tones matched to brighter colors

34. Marks (1987) • Discriminate tones of varying pitch • High pitch and low pitch • Light of varying brightness presented simultaneously with tones • Measure response time to discriminate

35. Marks (1987)

36. Marks (1987) • Discriminate tones of varying pitch • High pitch and low pitch • Light of varying brightness presented simultaneously with tones • Measure response time to discriminate • Low pitches discriminated faster when light was dim than bright • High pitches discriminated faster when light was bright than dim • Cross-talk between senses in “normal” subjects

37. Weak Synesthesia • We all have to certain degree • “Loud shirt” • “Sharp cheddar” • A type of imagery that enriches our sensory experience