290 likes | 417 Vues
Multisensory Interplay. Jon Driver and Toemme Noesselt. Sensory Research. Sensory Research. Behavioral Consequences of Multimodality. Joint estimates of single property Spatial Ventriloquism Auditory Driving McGurk Effect
E N D
Multisensory Interplay Jon Driver and Toemme Noesselt
Behavioral Consequences of Multimodality • Joint estimates of single property • Spatial Ventriloquism • Auditory Driving • McGurk Effect • Can modalities affect each other without creating a single unified percept? • Touch at a location can help perception of color • Sound-induced Illusory Flash • Orientation discrimination improves with multiple beeps
Convergence Zones • Superior Colliculus • inputs from somatosensory, auditory, and visual areas • Super or Sub-additive responses found for combination
Additivity in SC • most likely with weak unisensory inputs • Ceiling effect? • Neural limitations? • Late onset in development • Depends on multisensory cortex
Testing for Convergence • Anatomical studies: • Direct connections between different sensory areas • Single-cell studies: • Response to stimulation from different modalities • Neuroimaging: • Large-scale responses based on BOLD signal
Influences on 'Sensory-Specific' Areas • Growing body of research shows that sensory-specific areas might be an artifact of the studies done with them • Examine studies that use: • fMRI • EEG • Invasive recording in animal models
fMRI • Caveat: • fMRI has been shown to respond to attention and imagery • For example, speech may be imagined when viewing lip movements
fMRI analysis • Inspired by Stein (SC), some look for sub-, super-additive responses • Maybe linearity is normal, though, so some use max or mean criteria • Difficult because of spatial resolution
Convergence in V1? Amedi, Jacobsen, Hendler, Malach, and Zohary, 2002
ERP results example • Tactile stimulation • Visual cue • ERP extracted
ERP studies • visual N1 enhanced when tactile stimulation occurred at same location as a visual event • visual P1 modified by task-irrelevant sound • P1 modified by attend-visual relative to attend-tactile conditions
ERP studies • ERPs show early multi-sensory effects (~30 ms) • Poor localization • Potential methodological confounds
Invasive Studies • Current-source densities (CSD) reflect local PSPs • Region of auditory association cortex • Location and timing of stimulation consistent with auditory feed-forward, visual feed-back
Invasive Studies • Posture may affect responses to auditory signals in A1 • Tactile stimuli modulate initial response to auditory signals in A1
Multisensory Interplay • Examples of converging zones of multi-sensory input • Examples of interplay: one modality affects another • What frameworks does this evidence suggest?
Possible Frameworks • A) All Multisensory • B) Bimodal Brain Areas • C) Critical Feedback Circuitry
All Multisensory • Unlikely to be completely undifferentiated • Even primary sensory areas responsive to multiple modalities
All Multisensory • Thalamus might be source of multisensory interplay • Tactile stimulation can affect first neural response in A1, hypothesized from thalamus • Found in gerbils, hard to study in humans
All Multisensory • Direct coritco-cortical influences • Anatomical evidence: single synapse from AC-VC and AC-SSC, AC-OC • However, not as many as to conventional Multi-sensory areas • Role still unclear
All Multisensory • Still overwhelmingly "sensory-specific"
Bimodal Brain Areas • Less extreme version of account A • Similar to current account, with more multi-sensory regions • Parallel multi- and single-modality processing could explain early EEG modulation
Bimodal Brain Areas • Different areas in auditory areas may be connected to distinct visual areas • Bimodal interplay would be affected by transduction time, explaining BOLD response time differences
Feedback Circuitry • Effects in primary areas might be feedback from convergence zones • Evidence from effective connectivity in fMRI, tactile stimulation increases visual response • Evidence from EEG source-localization: STS - VC
Feedback Circuitry • Evidence from invasive recordings: late A1 stimulation from vision (speculation?) • Feedback can be tested directly, but very little has been done
Remarks • Perhaps the rival frameworks are all valid for certain situations • Perhaps primary cortex responses in the blind and deaf can help tease out what the multisensory roles are • New techniques will allow testing causal interplay