Visually-induced auditory spatial adaptation in monkeys and humans

1. Visually-induced auditory spatial adaptation in monkeys and humans Norbert Kopčo Center for Cognitive Neuroscience, Duke UniversityHearing Research Center, Boston UniversityTechnical University of Košice, Slovakia (Frequent flier # OK10509496)

2. Way to go Red Sox! Way to go Red Sox! Introduction • Visual stimuli can affect the perception of sound location e.g. the Ventriloquism Effect • But does effect persist? BU HRC

3. Way to go Red Sox! Introduction • Visual stimuli can affect the perception of sound location e.g. the Ventriloquist Effect • But does effect persist? • - barn owls: prism adaptation (Knudsen et al.) • - monkeys: “ventriloquism aftereffect” (Woods and Recanzone, Curr. Biol. 2004) • Why does the effect persist? • Calibration of auditory perception • (on different time scales): • - to new environments (rooms) • - to anatomical changes (head size) BU HRC

4. GOALS • Ventriloquism “aftereffect” in saccade task, in monkeys and humans? - well-defined sensory-motor paradigm - bridge to barn owl prism adaptation studies (on different time scale) • Reference frame of plasticity? - Visual, auditory, or oculomotor reference frame? BU HRC

5. Methods • Basic idea: • 1. Pre-adaptation baseline: Measure auditory saccade accuracy • 2. Adaptation phase: Present combined visual-auditory stimuli, with visual location shifted • 3. Compare auditory saccade accuracy pre- and post-adaptation BU HRC

6. Methods • Initial experiment: Does it work? • Design: • Monkey • Pre-adaptation baseline – ~100 Auditory-only trials • Adaptation phase – • 80% V-A stimuli, visual stimulus shifted 6 deg. Left or Right • 20% Auditory-only • Compare Auditory-only trials from adaptation phase to pre- • adaptation phase Sounds: Loudspeakers Visual stimuli: LEDs BU HRC

7. RESULTS BU HRC

8. RESULTS BU HRC

9. RESULTS BU HRC

10. RESULTS BU HRC

11. QUESTION • How does vision calibrate sound perception in primates? • - monkeys and humans • Unlike barn owls, monkeys and humans make eye movements. With every eye movement, the relationship between visual space and auditory space changes. • Visual and auditory spatial information are different! BU HRC

12. Visual and auditory spatial information are different! • VISION: • Retina provides “map” of object locations • Locations shift when eyes move • Frame of reference is “eye-centered” BU HRC

13. Visual and auditory spatial information are different! • AUDITORY: • Sound location calculated from interaural timing and level differences • Cue values do NOT shift when eyes move • Frame of reference is “head-centered” BU HRC

14. ? Eye-centered? ? Oculomotor? Head (ear) -centered? Goals • Perform behavioral experiments to answer the following questions: • Does visual calibration of auditory space occur in eye-centered, head-centered, or a hybrid coordinate system? • Are humans and monkeys similar? BU HRC

15. Experimental Setup • Audiovisual display as viewed by the subject • 9 speakers in front of listener • (~1 m distance), separated by • 7.5° (humans) or 6° (monkeys) • Light-emitting diodes (LEDs) • at three center speakers: • - aligned with speakers, or • - displaced to the left • or to the right • (by 5°-humans, 6°-monkeys) • 2 LEDs below speaker array • used as fixation points (FP) • Stimuli: • Auditory stimulus: • 300-ms broadband noise burst • Audio-Visual stimulus: • Same noise with synchronously lid LED. • Vertical location (degrees) • Horizontal location (degrees) BU HRC

16. Expected behavior • Magnitude (°) • Stimulus Location (°) Method • Audiovisual display • Induce shift: • - in only one region of space • - from a single fixation point • Test to see if shift generalizes • to the same sub-region in: • - head-centered space • - eye-centered space BU HRC

17. Experiment: Procedure • Audiovisual display as viewed by the subject • One trial consists of: • 1. Fixation point (FP) appears. • 2. Subject fixates FP. • 3. Target stimulus is presented • (Audio-Visual or Auditory-only). • 4. Subject saccades to perceived location of stimulus (humans instructed to always saccade to sound). • 5. Monkeys only: Reward for responding within a criterion window (+- 10° from speaker). • Vertical location (degrees) • Horizontal location (degrees) BU HRC

18. AV stimuli • 50 % of trials • A-only stimuli • shifted FP: 25% of trials • A-only stimuli • trained FP: 25% of trials • Speakers • LEDs • FP Experiment: Procedure • Experiment divided into 1-hour blocks. • AV stimulus type kept constant within a block (left, right, or no displacement). • 12 blocks for humans, 16 for monkeys. • Subjects: 7 humans, 2 monkeys. • Within a block three types of trials, randomly interleaved: • AV FP on left and right. In presentation collapsed to right. BU HRC

19. Speakers • LEDs • FP • or Results: Humans • Audiovisual display • Trained FP A-only • responses: • - Shift induced in • trained sub-region • - Generalization to • untrained regions • (asymmetrical) • Induced Shift Magnitude (M+SE °) • Shifted FP A-only • responses: • - Shift reduced in • center region • Stimulus Location (°) • Expected Responses • Head-centered representation, • modulated by eye position BU HRC

20. Speakers • LEDs • FP • or Results: Monkeys • Audiovisual display • Trained FP A-only • responses: • - Shift in trained sub- • region weaker • - Generalization to • untrained regions • stronger • (asymmetry oppo- • site to humans) • Magnitude of Induced Shift (°) • Shifted FP A-only • responses: • - Shift decreases on • the right • - Shift increases on • the left • Stimulus Location (°) • Expected Responses • Humans: • Representation • more mixed • than in humans BU HRC

21. Summary • The main results are consistent across species: • Locally induced ventriloquist effect results in short-term adaptation, causing 30-to-50% shifts in responses to A-only stimuli from trained sub-region. • The induced shift generalizes outside the trained sub-region, with gradually decreasing strength (However, the pattern of generalization differs across the species) • The pattern of induced shift changes as the eyes move. But, overall, it appears to be in a representation frame that is more head-centered than eye-centered. BU HRC

22. Discussion (almost done) • Posterior • Parietal • Cortex • Neural adaptation could have been • induced at several stages along the • pathway (IC, MGB, AC, PPX, MC, • SC). • In humans, multiple effects • observed at different temporal • scales  likely adaptation at • multiple stages • Future work • Examine temporal and spatial • factors influencing the eye- • centered modulation. • Look at other trained sub-regions. • Cerebrum • Thalamus • Thalamus • Midbrain • Midbrain • Pons • Pons BU HRC • (Kandel, Schwartz, Jessel) and (Purves)

23. Speakers • LEDs • FP Humans: temporal profile In humans, multiple effects observed at different temporal scales  likely adaptation at multiple stages BU HRC

24. Speakers • LEDs • FP Humans: ipsilateral shift Eye-centered modulation does not occur. Possible explanation: the modulation is specific to the eye-centered hemisphere in which the audiovisual shift is induced (I-Fan currently testing) BU HRC

25. Speakers • Speakers • LEDs • FP • LEDs • FP Monkeys: central vs. ipsi shift Are the monkeys really adapting in an eye-centered co-ordinate frame? BU HRC

26. Summary • The main results are consistent across species (when shift induced in CENTER):Shift induced in the center • Shift generalizes to non-trained sub-regions • Shift changes with eye movement • The consistency across species is less obvious when the trained sub-region shifts to IPSI: • Humans: The relatively small eye modulation disappears • Monkeys: Eye movement induces shift that is almost purely eye-centric BU HRC

27. Collaborators Jennifer Groh Center for Cognitive Neuroscience, Duke University I-Fan Lin Barbara Shinn-Cunningham Hearing Research Center, Boston University Support NIH grants to Barb and Jenni Slovak Science Grant Agency