Linguistic Neuroscience: Extending Perceptual Neuroscience to Language

1. Ling 411 – 12 Linguistic Neuroscience:Extending Perceptual Neuroscience to Language

2. “Linguistic Neuroscience”? • Applying the findings of perceptual neuroscience to language • Perceptual neuroscience as in Mountcastle’s 1998 book • Mountcastle doesn’t say anything about language • But his findings can be applied

3. Findings relating to columns(Mountcastle, Perceptual Neuroscience, 1998) The column is the fundamental module of perceptual systems probably also of motor systems This columnar structure is found in all mammals that have been investigated The theory is confirmed by detailed studies of visual, auditory, and somatosensory perception in living cat and monkey brains

4. Adjacency and the Proximity Principle • Neighboring areas for closely related functions • The closer the function the closer the proximity • Consequences • Members of same category will be in same area • Why? • Same category because similar functions • Competitors will be neighbors in the same area • Why? • Neighbors in same area have same general function along with additional differentiating function • They compete w.r.t. the differentiating function

5. Inhibitory connections Based on Mountcastle (1998) • Columnar specificity is maintained by pericolumnar inhibition (190) • Activity in one column can suppress that in its immediate neighbors (191) • Inhibitory cells can also inhibit other inhibitory cells (193) • Inhibitory cells can connect to axons of other cells (“axoaxonal connections”) • Large basket cells send myelinated projections as far as 1-2 mm horizontally (193)

6. Extrapolation to Language? Our knowledge of cortical columns comes mostly from studies of perception in cats, monkeys, and rats Such studies haven’t been done for language Cats and monkeys don’t have language That kind of neurosurgical experiment isn’t done on human beings Are they relevant to language anyway? Relevant if language uses similar cortical structures Relevant if linguistic functions are like perceptual functions

7. Perception and Language • Why haven’t such studies been done for language? • That kind of neurosurgical experiment isn’t done on human beings • Cats and monkeys don’t have language • Are they relevant to language anyway? • Relevant if language uses similar cortical structures • Relevant if linguistic functions are like perceptual functions

8. Relevance to Language These studies of perception are relevant if Perceptual structure and functions are basically the same across modalities Including associative areas (higher-level) Linguistic comprehension is basically a perceptual process

9. Objection • Cats and monkeys don’t have language • Language (as we know it) is a unique human faculty • Therefore language must have unique properties of its structural representation in the cortex • Answer: Yes, language is different, but • The differences are a consequence not of different (local) structure but differences of connectivity • The neurocognitive network does not have different kinds of structure for different kinds of information • Rather, different connectivities

10. Justifying extrapolation • Hypothesis: Extrapolation of findings about cortical columns can be extended to • humans • linguistic and conceptual structures • Why? • Summary of the argument • Cortical structure, viewed locally, is • uniform across mammalian species • uniform across different cortical regions • Exceptions in primary visual and primary auditory areas • Different cortical regions have different functions • because of differences in connectivity • not because of differences in structure

11. Essence of the argument • Cortical structure and function, locally, are essentially the same in humans as in cats and monkeys and rats • Moreover, in humans, • The regions that support language have the same structure locally as other cortical regions

12. Uniformity of cortical function Claim: Locally, all cortical processing is the same The apparent differences of function are consequences of differences in larger-scale connectivity Conclusion (if the claim is supported): Understanding language, even at higher levels, is basically a perceptual process

13. Argument for local uniformity of representation Different types of cortical information Perceptual Conceptual Grammatical Phonological How are they different? Two possibilities They differ in their structural form They differ based on their connections Claim: Possibility #2 is the correct one The “connectionist claim”

14. Support for the connectionist claim Lines and nodes (i.e., columns) are approximately the same all over Uniformity of cortical structure Same kinds of columnar structure Same kinds of neurons Same kinds of connections Conclusion: Different areas have different functions because of what they are connected to

15. Linguistic Information in the Cortex • Problem: Linguistic information is usually described symbolically • In the symbolic mode of description, different kinds of linguistic information appear to have different kinds of structure • Phonology • Morphology • Regular and irregular inflections • Syntax • Semantics • Claim: If the information is viewed as connectional instead of symbolic, it turns out to have a high degree of uniformity

16. Uniformity of cortical structure • Six layers throughout, with similar structure • Columns throughout • Same neuron types everywhere – pyramidal most frequent, spiny stellate in layer IV, etc. • Inhibitory and excitatory connections throughout • Same neurotransmitters everywhere • Excitatory: glutamate • Inhibitory: GABA • But: What about the different Brodmann areas? • The differences are relatively minor • They may be based on experience

17. Structural Uniformity?A closer look Differences are found at lower levels Primary sensory areas have specialized structures These are called heterotypical areas Properties of columns depend on afferent inflow More uniformity in higher-level areas “Homotypical” areas (i.e., same type) Relatively uniform structure Makes them flexible, adaptable Properties depend on intracortical processing Different homotypical areas differ not because of their structures but because of their connections

18. A heterotypical area: Visual motion perception An area in the posterior bank of the superior temporal sulcus of a macaque monkey (“V-5”) Albright et al. 1984 400-500 μ

19. Auditory areas in a cat’s cortex(Heterotypical) A1 AAF – Anterior auditory field A1 – Primary auditory field PAF – Posterior auditory field VPAF – Ventral posterior auditory field

20. Exceptions: Diversity in cortical function Perception vs. production Back brain vs. front brain Sharpness of contrast Phonology and morphology require sharp contrasts Conceptual categories have fuzzy definitions Left vs. right hemisphere RH may have.. Larger minicolumns Less lateral inhibition

21. Uniformity in LH Associative Areas The associative areas are homotypical The structure that subserves language understanding is the same as perceptual structure Columns of similar structure With similar kinds of connections Claim: Understanding language is the same process as perception To support this claim, must look more closely at cortical function Subclaim: Locally, all columns function alike

22. Primary areas and higher-level areas (LH) These are homotypical

23. The uniformity of information? Different types of cortical information Perceptual Conceptual Grammatical Phonological How are they different? Two possibilities They differ in their form of representation They differ based on their connections Claim: Possibility #2 is the correct one The “connectivity claim”

24. Uniformity of cortical function If cortical function is uniform across mammals and across different cortical areas, then the findings presented by Mountcastle can be extended to language Claims: Locally, all cortical processing is the same The apparent differences of function are consequences of differences in larger-scale connectivity Conclusion (if the claim is supported): Understanding language, even at higher levels, is basically a perceptual process

25. Testing the claim • Claim: • The apparent differences of function are consequences of differences in larger-scale connectivity • To test, we need to understand cortical function • That means we have to better understand the function of the cortical column

26. Columns do not store symbols! • They only • Receive activation • Maintain activation • Inhibit competitors • Transmit activation • Important consequence: • We have linguistic information represented in the cortex without the use of symbols • It’s all in the connectivity • The Challenge: • How? • This claim goes against most of the history of linguistics

27. Operation of the Network • The linguistic system operates as distributed processing of multiple individual components – cortical columns • Columnar Functions • Integration:A column is activated if it receives enough activation from other columns • Can be activated to varying degrees • Can keep activation alive for a period of time • Broadcasting:An activated column transmits activation to other columns • Exitatory – contribution to higher level • Inhibitory – dampens competition at same level • Columns do not store symbols!

28. Why the usual approach won’t work • Let us suppose that words are stored in some kind of symbolic form • What form? • If written, there has to be.. • something in there that can read them • something in there that can write them • something in there that can move them around, from one place to another • something in there to compare them with forms entering the brain as it hears someone speaking – otherwise, how can an incoming word be recognized?

29. Why the usual approach won’t work (cont’d) • If not written, then represented in some other medium • Doesn’t solve the problem • You still need whatever kind of sensory detectors can sense the symbols in whatever medium you choose • Plus means of performing all those other operations

30. Compare imagery • Visual images • Little pictures? • If so, what is in there to see them? • Auditory images • Little sounds vibrating in the brain? • If so, what is in there to hear them? • There has to be another way!

31. There must be another way • Visual imagery (e.g. of your grandmother) • Reactivation of some of the same nodes and connections that operate when actually seeing her • Auditory imagery (e.g. of a tune) • Reactivation of some of the same nodes and connections that operate in actually hearing it

32. Another way, for language • A syllable • Activation of the nodes and connections needed to recognize or produce it • A word • Activation of the nodes and connections needed to recognize it • A syntactic construction • Activation of the nodes and connections needed to recognize or produce it

33. Hjelmslev’s view The postulation of objects as something different from the terms of relationships is a superfluous axiom and consequently a metaphysical hypothesis from which linguistic science will have to be freed. Louis Hjelmslev Prolegomena to a Theory of Language (1943: 61)

34. Columnar Functions: Integration and Broadcasting Integration: A column is activated if it receives enough activation from Other columns Thalamus Can be activated to varying degrees Can keep activation alive for a period of time Broadcasting: An activated column transmits activation to other columns Exitatory Inhibitory Learning: adjustment of connection strengths and thresholds

35. Integration and Broadcasting • Broadcasting • To multiple locations • In parallel • Integration

36. Integration and Broadcasting Now I’ll tell my friends! Broadcasting Integration Wow, I got activated!

37. What matters is not ‘what’ but ‘where’ What distinguishes one kind of information from another is what it is connected to Lines and nodes are approximately the same all over Hence, uniformity of cortical structure Same kinds of columnar structure Same kinds of neurons Same kinds of connections Different areas have different functions because of what they are connected to

38. Operations in neurocognitive networks • Activation moves along lines and through nodes • Integration • Broadcasting • Connection strengths are variable • A connection becomes stronger with repeated successful use • A stronger connection can carry greater activation

39. Can language be accounted for by such simple operations? • Phonology • Words and their meanings • Syntax and morphology • Conceptual relationships

40. Sequence • In language, sequence is very important • Word order • Order of phonological elements in syllables • Etc. • Also important in many non-linguistic areas • Dancing • Eating a meal • Can cortical columns handle sequences?

41. Lasting activation in minicolumn Cell Types Pyramidal Spiny Stellate Inhibitory Recurrent axon branches keep activation alive in the column – Until is is turned off by inhibitory cell Connections to neighboring columns not shown Subcortical locations

42. Notation for lasting activation > Thick border for a node that stays active for a relatively long time > Thin border for a node that stays active for a relatively short time

43. Recognizing items in sequence This node recognizes the sequence ab c This link stays active a b Node c is satisfied by activation from both a and b If satisfied it sends activation to output connections Node a keeps itself active for a while Suppose that node b is activated afternode a Then c will recognize the sequenceab

44. Recognizing stop consonants • Consider stop consonants, e.g. t, d • At the time of closure • For voiceless stops there is no sound to hear • For voiced stops, very little sound • The stops are identified by transitions • To following vowel • From preceding vowel

45. Demisyllables [di, de, da, du] F1 and F2 For [a] It is unlikely that [d] is represented as a unit in perception

46. Recognizing a syllable and its demisyllables Cardinal node for dim Functional subweb for dim dim Just labels di- -im Auditory features of [di-] Auditory features of [-im]

47. Another syllable and its demisyllables Cardinal node for bill Subweb for bill bil bi- -il

48. Multiple connections of -il bil hil kil Bill hill mill kill etc. bi- -il One and the same /-il/ in all of them

49. Multiple connections of -il bil hil kil Bill hill mill kill etc. bi- -il Similarly for multiple connections of bi- bit, bib, bid, etc.

50. Multiple connections of -il bil hil kil Bill hill mill kill etc. bi- -il To lower level nodes, for phonological features