Stein, B., Meredith, M., The merging of the senses , Cambridge, Mass., MIT Press, 1993

1. Stein, B., Meredith, M., The merging of the senses, Cambridge, Mass., MIT Press, 1993 Part III, IV

2. Representation of the sensory space in the superior colliculus Alignement of sensory and motor maps

3. Sensory and motor representations are distributed in map-like form Segregation • Map-like recreation of the receptor epithelium, and thus of the sensory space it serves • In the cns the visual, auditory, somatosensory representations occupy spatially distinct regions, functionally and anatomically defined • In the sensory cortex and thalamus the representations of the different sensory modalities are organized in maps which are segregated one from another (segregation between modalities in order to avoid confusion between modalities ) • Submodality features separate maps within each modality (segregation within each modality to facilitate recognition of certain stimulus characteristics Integration • No distinct regions for single sensory representation or single submodality feature in the superior colliculus

4. Superior colliculus Anatomical organization: • Cerebral tissue organized in layers, with a significant distinction between superficial and deeper ones • superficial layers: dense visual innervation (the entire colliculus was long considerd an exclusively visual structure) • deeper layers receive inputs from different sensory modalities (visual, auditory, somatosensory; ascending and descending) and from motor-related structures • deeper layers send their outputs to areas of the brain stem and spinal cord involvend in positioning the peripheral sensory organs and in the transformation of incoming sensory information into motor commands (sensorimotor transduction) • most descending output neurons are the sites of multisensory convergence (multimodal efferent neurons): each of the sensory representations has access to at least some of the same efferent circuitry  the different sensory systems can initiate the same behaviors via some of the same neurons Consequences of ablation: • disturbances in visual attention (visual neglect) and orientation behaviors (« where » system), lost of the ability to respond appropriately to contralateral (to the lesion) touch and auditory stimuli • bilaterally symmetrical lesions are far less disruptive to visual auditory and somatosensory behavior than unilateral ones

5. Behavioral function: • Attentive and orientation roles Specific role: • Integrating the modalities: • associates the sensory inputs to redirect the organs the input originates from • in order to localize (capture-avoid) the source of the stimulus • Sensorimotor transduction: • transforms incoming sensory inputs into motor commands • by virtue of the convergence and intermixing of sensory inputs and motor circuits (different sensory modalities access to the same output motor circuits) • many neurons have sensory and motor properties and are involved in a variety of circuits and functions Realization: • alignement of the different sensory and motor maps (receptive fields and movement fields in register) • sometimes with use of multisensory neurons which can have also a premotor role

6. Sensory maps in the colliculus • sensory neurons of the colliculus are organized in visuotopic, somatotopic, auditory maps • visual maps in superficial layers: nasal-temporal meridians (horizontal medians) run rostral-caudal, vertical meridians run medial-lateral; in deeper layers: similiarity of the overall pattern, with close alignement of the representation of central visual space, but larger receptive fields, including far periphery of the visual space : maps are not the simple extension of the superficial ones • somatosensory neurons in the deeper layers have large receptive fields and are organized in maps which show a regular relationship with visual maps: the front of the animal is represented rostral while the hindparts are caudal, the upper surface is represented medial and its lower aspects lateral. The blocks of tissues devoted to regions of the body surface are not exclusive: considerable overlap among the representations • auditory neurons are organized in a computated spatial map oriented very much like the visual and somatosensory maps: the auditory horizontal meridian is represented rostrally-caudally and the vertical one median-laterally • register between superficial visuotopic and deeper visual, auditory, somatosensory maps • the same axes are used to represent all three sensory modalities: • the fact that the visual maps in the superficial layers and the visual, auditory and somatotopic maps of the deeper layers are aligned indicate an intimate interaction. But there is no evidence that superficial neurons influence deeper visual, somatosensory or motor neurons

7. spatial characteristics of the visual somatosensory and auditory systems In both the visual and somatosensory systems each peripheral nerve fiber responds to a stimulus in a restricted (generally contralateral) spatial domain, regardless of stimulus intensity, nd this defines the cell’s receptive field. For this reason is easy to understand how the cns constructs a spatial map of the contralateral visual field and the contralateral body surface In contrast, there is no spatial map at the peripheral receptors of the auditory system, which is organized both at the thalamic and cortical levels according to frequencies or tones (tonotopic, not spatiotopic). The contruction of spatial auditory maps in the superior colliculus is the result of a computation based on the differences in intensity and timing of suond as it reaches the two ears

8. Motor maps in the colliculus • Premotor or motor efferent neurons of the colliculus are connected through many circuits with other motor areas in the the brain stem and spinal cord • these motor connections are organized in maps • motor maps overlap sensory maps: • the electrical stimulation of a site representing superior temporal visual space will move the eyes temporally and superiorly, as if to center the fovea on the visual location represented at the stimulation site; stimulation of the medial aspects of the structure where upper visual field is represented will elicit upward eye movements, while lateral stimulation downward movements • the representation of a region of sensory space and the representation of the signals required to move the eyes toward that region are in the same colliculus location: sensory and motor maps covary • eye and ear movement maps are in register

9. interspecies constancy of register between sensory maps • Details presented above are drawn from data gathered in the cat • There are important differences between animal species • What remains constant is the general plan of overlapping (register) of the different sensory representations, of the movement representations and of sensory and movement representations • The differences reflect the sensory modality a species depends on most for exploring and responding to environmental stimuli • From reptiles, to birds to mammalians, the register of sensory and movement representations in the midbrain is a constant presence, presumably because it is an efficient solution to the problem of using different sensory cues to move one and the same body

10. multisensory integration at the superior colliculus level topographic register and convergence on multisensory neurons

11. From the examination of individual sensory maps to the examination of individual multisensory neurons • The integration of a multisensory multimotor maps can be done in two ways: • stimuli originating from the same locations in sensory space activate neighboring unimodal neurons from different modalities which have acces to at least one of the output neurons • different modalities converge on the same neurons and these same multisensory neurons produce a coordinated series of premotor signals • We have just seen that multisensory integration in the superior colliculus depends on the alignement of sensory representations • But there is evidence that the interrelationship among different sensory representations is more intimate than a simple parallel among individual organizations • The largest group of the sensory neurons in the deep layers is multisensory  an effective sensory stimulus activates many of the same neurons, enhancing its salience • Visual-multisensory and somatosensory-multisensory neurons are not clustered in one region of the superior colliculus representing one region of visual or somatosensory space: they cover the entire structure, so that the the majority of neurons from which the visual map, or the somatosensory map, is constructed are multisensory

12. multisensory neurons • there are many areas in the brain in which multiple sensory afferents converge • there are colliculus neurons that respond vigorously to low intensity auditory stimulus, but if the animal can’t see the visual stimulus the response is suppressed • the colliculus is an apt structure to study interactions between sensory modalities, and in particular multisensory integration at the level of a single neuron

13. Multisensory integrated maps • Since the different sensory maps in the colliculus are composed of many of the same neurons, it is more appropriate to consider them as three components of a multisensory integrated map, rather than three parallel independent maps • And since this « multisensory map » is in register with motor maps, we should speak of a  « multisensory-multimotor  map » which coordinates th movement of eyes, ears, head, body • It is not to be forgotten for the understanding of the functioning of the superior colliculus that i ntegrated multisensory maps coexist with unimodal maps (unimodal neurons) that preserve modality specificity and modality specific activation of premotor neurons

14. multisensory neurons in the colliculus Method: • estimates are based on all neurons ecountered in the superior colliculus, including those that doesn’t respond to sensory stimuli • quantitative evaluation of the effects of presenting two stimuli from different modalities independently and concertedly in every neuron ecountered • permits to evaluate the effect of a stimulus (auditory) in biasing one other (visual), even if it itself doesn’t generate impulses Results (cat): • over ½ of the the neurons of the deeper layer of the colliculus are influenced by stimuli from more than one sensory modality = are multisensory • neurons with visual inputs predominate (somatosensory in rodents) • the most frequent category is visual-auditory multisensory (visual-somatosensory in rodents) • the ¾ of the neurons having descending efferent projections are activated by multisensory stimuli; tha majority of neurons that fail to demonstrate a descending projection are unimodal • multisensory neurons are the most significant contributors to the behavior mediated by the superior colliculus • the outputs of the superior colliculus are mostly the products of the synthesis of different sensory inputs • it is the presence of integrate multisensory messages that determine colliculis mediated behaviors

15. colliculus multisensory neurons characteristics • unimodal and multimodal neurons are very similar for what regards the properties of their receptive fields, but for their dimension (bigger receptive field for multisensory neurons): unimodal and multisensory neurons have acces to the same modality specific information and synthetize it in the same way • multisensory convergence takes place directly in the colliculus (doesn’t derive from multisensory neurons somewhere else)  corticotectal and peripheral inputs are unimodal and converge on a single colliculus neuron

16. Assembling the stimuli: enhancement and depression • Since sensory channels separate the stimuli, the brain must then relate stimuli one to another • In assembling the stimuli from different modalities the brain acts on the base of the significance of the stimuli, determined by: • intrinsic circuitry • postnatal experience • Some combinations of stimuli enhance the activity of neural responses and so they become more salient; other combinations depress the activity and they remain less salient • Enhancement and depression in activity produced by combinations of stimuli is characteristic of superior colliculus neurons • Enhancement and depression generally signal the presence or absence of meaningful relations among the stimuli

17. Enhancement: the potent effect of combinations of sensory stimuli • Responses evoked by combinations of stimuli are stronger (at all levels: response reliability, number of impulses evoked, peak impulse frequency, duration of the discharge train) than responses evoked by a single sensory cue • Response enhancements are present in every multisensory combination • The multisensory activity of a neuron sometimes reveals only when multiple sensory cues are present (ie: two apparently uneffective stimuli produce action potentials only as a combined stimulus)

18. Depression • Responses evoked by combined stimuli are sometimes weaker (fewer impulses, shorter discharge train duration, lower peak frequencies, lower response reliability) than one stimulus alone • Response depression is less common than enhancement; it depends on some specific properties as spatial inhibition, inhibitory surrounds, inhibitory inputs that are not common to all multisensory receptive fields • The inhibitory effect of the ineffective stimulus becomes apparent only when coupled with another stimulus

19. Rules for multisensory integration: causality • Multisensory enhancement and depression are determined by the spatial and temporal characteristics of the stimuli • Multisensory assembly depends on whether or not the stimuli are like to have common causality • stimuli that occur at the same time in the same place are likely to be interrelate because they are likely to have common causality • so this combination of stimuli is likely to produce enhancement • stimuli that occur at different places and times are unlikely to be related and they will product depression • Experience is not effective in changing the interaction of a stimulus combination from enhancement to depression • The role of experience is in aligning the different sensory maps in the superior colliculus • Once the maps are aligned neuronal enhancement or depression becomes dependent on the spatial and temporal relationship among stimuli

20. Rules for multisensory integration: space • Multisensory enhancement and depression are determined by the spatial and temporal characteristics of the stimuli: • Space: spatially coincident multisensory stimuli tend to produce response enhancement; spatially disparate stimuli produce depression or no interaction: • varying the position of the different sensory stimuli, it is evident that the enhancement is present only if the visual and auditory stimuli fall in the visual and auditory receptive field of the multisensory neuron • the different unimodal receptive fields of a multisensory neuron overlap • it is in virtue of this overlap that stimuli located near one another in space enhance one another’s effects • it is not question of an identity of position in the external space (or visual and somatosensory stimuli would never be integrated: it is a question of representational axes, which are common (see register, alignement of the maps) • the alignement of the maps is then a prerequisite for multisensory integration and enhancement • enhancement and depression depend on the falling of a stimulus in the zones of excitation or of inhibition of the receptive field of a multisensory neuron: the receptive field and not space is the proper referent for multisensory integration

21. receptive fields • Receptive fields are the regions in which limits a stimulus produces an excitation • Sometimes these regions are bordered by inhibitory areas the excitation of which produces the suppression of the excitation in the entire receptive field • A stimulus is never excitatory or inhibitory in itself • Receptive fields characteristics are robust

22. Rules for multisensory integration: time • Multisensory enhancement and depression are determined by the spatial and temporal characteristics of the stimuli: • Time: multisensory stimuli within a certain “temporal window” interact in a way that enhances responses • Overlapping the peak activity (excitatory or inhibitory) periods of two unimodal stimuli maximizes their interactions

23. Other rules for multisensory integration • Response enhancement is not effective when two stimuli come from the same modality (despite the fact that multisensory intgration depends on unimodal receptive field properties) • Combinations of weak unimodal stimuli produces the greatest results

24. Summary • The construction of a multisensory space at the superior colliculus level • parallel alignement (register) • alignement of the sensory representations (sensory maps register) • alignement of the sensory and movement representations (sensory and motor maps register) • convergence on a single neuron (integrated multisensory maps) • multisensory neurons • multisensory and premotor neurons • coordinating different modalities (different spaces) and movement (sensorimotor transduction) • in order to position the sensory organ in relation with the stimulus to reach-avoid (attention-positioning)

25. Questions • Enhancement-depression paradigm : • ROLE OF SPACE AND TIME FOR SENSORY INTEGRATION • CAUSALITY: IS IT TRUE THAT TWO STIMULI BEING ASSEMBLIED FOR TEMPORAL AND SPATIAL REASONS (WHERE TIME AND SPACE ARE RELATIVE TO THE RECEPTIVE FIELD CHARACTERISTICS, AND NOT TO THE OBJECT THEY ORIGINATE IN) ARE LIKELY TO HAVE A COMMON CAUSALITY? AND THAT COMMON CAUSALITY IS THE RELEVANT PARAMETER? • Existence of unimodal informations: • EXISTENCE OF UNIMODAL SPATIAL INFORMATIONS IN THE FORM OF RECEPTIVE FIELD EXCITATIONS • Alignement of the sensorimotor maps as a prerequisite for integration