Looking for a ‘gold standard’ to measure language complexity : What psycholinguistics and neurolinguistics can (and can

1. Lise Menn & Jill Duffield Linguistics Dept/Institute of Cognitive Science University of Colorado, Boulder lise.menn@colorado.edu cecily.duffield@colorado.edu Looking for a ‘gold standard’ to measure language complexity:What psycholinguistics and neurolinguistics can (and can’t) offer to formal linguistics

2. Background for our stance • Cross-linguistic work on basic morphosyntax in aphasia, and on the earliest stages of child phonology shows that these areas are loaded with individual and with language-specific differences. • ‘Markedness’ keeps vanishing into the mist of unverifiability. It’s no guide to complexity. • So, the issue of what’s simple and why, especially in those domains, has been a constant undercurrent and a frequent topic of discussion. Grammatical Complexity Workshop

3. What we’re going to say – 1 to 4 1. What our brains find difficult is not always what grammars consider complex, partly because what’s hard for our brains is not constant; it depends on many factors. (It’s complicated.) 2. Proposed metrics for language or grammar complexity should correspond in some way to the ‘gold standard’ of what’s hard for our brains to process. 3. Language complexity measures will have to go beyond a single measure of grammar complexity, because complexity for speakers/writers is not the same as for hearers/readers. 4. Psycho- and neurolinguistics can’t provide a royal road to measuring the complexity of a grammar or a language, but they do provide tools to measure processing complexity of individual sentences/utterances for speakers vs. listeners or readers, and learners vs. skilled language users. Grammatical Complexity Workshop

4. What we’re going to say – 5 to 8 5, Psycho- and neurolinguistic studies indicate that a valid measure of complexity will have to integrate across many linguistic levels (including semantics) and take frequency into account. 6. This implies that construction-based and usage-based approaches to grammar can provide insights into how grammars can come closer to reflecting what our brains do. 7. But: Complexity measures must handle competition and how it gets resolved in both comprehension and production: the paradigmatic axis also plays a role in complexity. 8. Pragmatics/real-world knowledge are involved in resolving this competition. (Implications for practical applications are clear; for comparison of languages, much less so.) Grammatical Complexity Workshop

5. Problematic examples to start from • Pourquoi l’aphasique peut-il dire: "Je ne peux pas le dire" et pas "Elle ne peut pas la chanter''? (Nespoulous & Lecours 1989) [Why can the aphasic person say I can’t say it but not She can’t sing it ?] • Possible culprits: lexical frequency, collocation frequency (formula status?), emotional weight… • Dressler’s (1991) work on Breton: Aspeaker with fluent aphasia tends to name pictures or examples of a single object using the plural form if the object itself is most frequently found in quantity (leaves, potatoes); using the dual if the object is usually found in pairs (eyes, hands). • Goes against all notions of markedness. Relative frequency of particular inflected form of particular word must be the explanation. • What brains find difficult is not always what grammar and linguists consider complex. Grammatical Complexity Workshop

6. 1. What our brains find difficult is not always what grammars consider complex, partly because what’s hard for our brains is not constant; it depends on many factors. Why is there a problem? We cannot equate ‘simpler’ with what is learned earlier, and the reason we cannot do it is that the neural networks change over the course of development. Grammatical Complexity Workshop

7. Another example from aphasia • Aphasic verb tense production errors are often, as one might expect, substitutions of present tense for past tense. • But the reverse seems to be true for at least some agrammatic aphasic speakers of Arabic (Mimouni & Jarema, 1997), Polish (Jarema & Kądzieława, 1990), and Korean (Halliwell, 2000) Grammatical Complexity Workshop

8. What brains find difficult is not always what grammars consider complex • This is not just about speakers with brain damage. ‘Difficulty’ is tied to particular circumstances. • Production/comprehension asymmetry: • most obvious case: ambiguity. Speakers, who know what they intend to say, often produce utterances that are difficult for hearers because of ambiguity in their referring expressions (He did it!) and elsewhere • Long history of studies of ambiguity resolution in psycholinguistics that demonstrates & relies on the processing difficulty caused by hearer’s or reader’s need to resolve ambiguity on-line. • Other studies showing speakers have to put effort into being simple for their hearers (any teacher knows this!) Grammatical Complexity Workshop

9. What brains find difficult is not always what grammars consider complex • Learning changes the brain (how could it fail to?), creating a learner/skilled user asymmetry • So, relative ‘simple-to-complex’ rankings must shift as we learn our first languages (OT calls this constraint (re-)ranking). • Phonotactics provides many uncontroversial examples • Blevins’ (1995) illustrations of syllable types: Spanish and Sedang permit CCVC but not CVCC, while the reverse is true for Klamath and Finnish. • Japanese speakers struggle with English /tr/ but routinely produce [tstʃ], e.g. in place name ‘Tsuchiura’) Grammatical Complexity Workshop

10. Before we go on: An essential distinction • The ‘grammar of a language’ as an abstraction across speakers – the ‘patterns out there’ to be learned (E-language) • The ‘grammar of a language’ as an abstraction across speakers isn’t directly testable by psycholinguistics/neurolinguistics. If that grammar is your main concern, what we have to say has to be mediated by your idea of the relationship between the grammar of a language and the grammar of each speaker. • the grammar internal to a given speaker, which should be that speaker’s internal approximation to the ‘patterns out there’ (I-language). • This is what we’re concerned with in this presentation. Grammatical Complexity Workshop

11. But focusing on speaker-internal grammar is only a start • Making this distinction can handle (some) differences between learners (who have a cruder approximation to that abstract grammar) and skilled users (who have a better one). • But there are more problems to deal with. One that we’ll keep coming back to: If there’s only one internal grammar, a single measure of its complexity can’t handle the the fact that what’s difficult for comprehension (e.g. ambiguity, unclear reference) is not necessarily difficult for production. Grammatical Complexity Workshop

12. Back to our first three points, slightly elaborated: What our brains find difficult is not always what grammars consider complex, partly because what’s hard for our brains is not constant and depends on many factors. Proposed metrics for language or grammar complexity – at least for speaker-internal language or grammar – should correspond in some way to the empirical ‘gold standard’ of what’s hard for our brains to process. So language complexity measures that claim to be valid metrics for what’s in human minds will also have to go beyond a single measure of grammar or language complexity. it’s Complicated Grammatical Complexity Workshop

13. And we still have the problem cases we started with: • Some aphasic people can say I can’t say it. but not She can’t sing it. • A Breton speaker with fluent aphasia tends to name pictures or examples of a single object using the plural form if the object itself is most frequently found in quantity (leaves, potatoes) using the dual form if the object is usually found in pairs (eyes, hands). Grammatical Complexity Workshop

14. What can (or can’t) psycholinguistics & neurolinguistics offer? Grammatical Complexity Workshop

15. What’s a ‘gold standard’? Why do linguists need one for complexity? • A rigorous standpoint, outside of particular formalisms and levels of language, to inform proposed measures of complexity • Needed in order to test whether a proposed metric corresponds to measures of what our brains find effortful to process …just as a proposed metric of color must correspond to some psychophysical measure of human responses to color if it’s going to be useful in accounting for perception. A metric that is useful for calibrating printers may not do well at accounting for what colors people find similar. Grammatical Complexity Workshop

16. Computational measures of utterance complexity need to be validated against processing measures –i.e., measures of performance • Validating a particular formal analysis of processing (e.g., an analysis that can take the number of competing antecedents for a relative pronoun into account, an analysis that can take various aspects of frequency into account) puts us into the domains of psycholinguistics and neurolinguistics, as other speakers have already made clear. Grammatical Complexity Workshop

17. Quotes on taking a psycholinguistic approach to grammatical complexity …the emerging correlation between performance and grammars exists because grammars have conventionalized the preferences of performance, in proportion to their strength and in proportion to their number, as they apply to the relevant structures in the relevant language types. (Hawkins 2004) In order to test the hypothesis that typological distributions reflect processing complexity, an independently motivated, well-defined, and empirically assessable notion of processing difficulty is essential. (Jaeger & Tily 2010) …not only [should] grammatical theorists …be interested in performance modeling, but also …empirical facts about various aspects of performance can and should inform the development of the theory of linguistic competence.” –Sag & Wasow, 2011 …the competence-performance distinction acknowledges the value of the sort of work linguists do in their day-to-day research, while recognizing that this work eventually must be placed in a broader psychological context. (Jackendoff 2002) Grammatical Complexity Workshop

18. Complexity measures must predict processing effort for different levels, and their interactions Psycholinguistic experiments with normal speakers Study of cognitive constraints and island effects (Hofmeister & Sag, 2010) • Results: Island constraints interact with other features to affect processing effort, correlating with grammaticality judgments • WH-island violations are processed more easily when the extracted element is complex (a WH-phrase) • Which employee did Albert learn whether they dismissed after the annual performance review? processed more easily than • Who did Albert learn whether they dismissed after the annual performance review? Grammatical Complexity Workshop

19. Measuring complexity for speakers:Error rate as an example Studies of formation of grammatical dependencies: even subject-verb agreement can be complex! • Producing subject-verb agreement: a “local noun” embedded in a subject noun phrase may interfere with the production of agreement - but structure constrains that interference. (Bock & Cutting 1992) • The key [PP to the wooden cabinet]… is / are… • The key [PP to the wooden cabinets]… is / are… • The key [RC that ___ opened the cabinet]… is / are… • The key [RC that ___ opened the cabinets]… is / are… attraction effect no attraction effect Grammatical Complexity Workshop

20. And there’s a glitch: Complexity measures must be compatible with different performance metrics • Speakers and listeners show different sensitivities to certain structures in processing tasks • While error data Bock & Cutting (1992) showed that relative clauses isolate information interfering with agreement for speakers while prepositional phrase modifiers do not (i.e., a clause boundary effect), Tanner (2012) uses ERP and reading times to show no interaction between structure x local noun number x grammaticality (no clause bounding effect) • in production • The key [PP to the wooden cabinet(s)]… • The key [RC that ___ opened the cabinet(s)]… • in comprehension • The key [PP to the wooden cabinet(s)]… • The key [RC that ___ opened the cabinet(s)]… effect of number of embedded noun differs across structure here, but not here Grammatical Complexity Workshop

21. Glitch: Complexity measures must be compatible with different performance metrics • Speakers and listeners show different sensitivities to certain structures in processing tasks • Speaker-hearer asymmetries aren’t ‘just’ matters of discourse ambiguity…. • which means that there can’t be just one measure of complexity of a sentence – comprehension and production may give different complexity rankings Grammatical Complexity Workshop

22. Glitch: Different sensitivities to structures in different processing tasks Study of discourse and weight-based factors on relative clause extraposition: Francis & Michaelis (2012) used two different tasks to measure the combined effects of: • Definiteness: e.g., (Some/The) research… • VP length: e.g. (…was conducted/…has been conducted fairly recently) • RC length: e.g., (….that refutes the existing theories/…that refutes the existing theories with very clear and convincing evidence) • Judgment task: readers saw two versions of a relative clause sentence (e.g., "Further research that indicates...." vs. "Further research has been conducted...") • Elicited production task: speakers were given three constituents - a subject NP, arelative clause, and a verb phrase, and asked to order those constituents in a full sentence. Grammatical Complexity Workshop

23. Glitch: Speakers and listeners may show different sensitivities to structures in particular processing tasks • In both experiments, indefinite subjects (e.g. "Some research" vs. "The research"). as opposed to definite ones, were more likely to be used with extraposed relative clauses. BUT: • In the judgment (comprehension) task, readers preferred the extraposed version for longer relative clauses • VP length, however, didn't matter • In the production task, VP length did matter: shorter VPs were more likely to predict extraposed relative clauses. • Relative clause length, however, didn't matter in production. No explanation for this particular pair of comprehension/production discrepancies yet… Grammatical Complexity Workshop

24. So: We don’t have a ‘royal road’ to offer; how about a set of road-building tools? What tools do we have to measure utterance complexity? • Complexity for the listener (comprehension) • Neural correlates of relative effort: one current measure is Event-Related Potential (ERP); • Errors, reaction time and eye-tracking can also be used. (Imaging tools. though glamorous, don’t have good enough spatiotemporal resolution yet) • Complexity for the speaker (production) • Error-based measures of relative effort comprise the majority of production research. • Reaction time measures and eye-tracking measures also are useful. Grammatical Complexity Workshop

25. Things you probably know about measuring performance • Available performance measures for both comprehension and production can only look at one word, utterance, or short passage at a time. • Performance always has a large random element – minds differ, and they are simultaneously busy with many things besides the task set by the experimenter (wishing for coffee, worrying about politics or the weather…) • So most measures have to be averaged over fairly large sets of similar items, and sometimes over speakers as well. Grammatical Complexity Workshop

26. Measuring complexity for listeners: Event-Related Potential as an example An ERP study of metaphor comprehension(Lai, Curran, & Menn 2009) • Comprehending conventional and novel metaphors: Lexical semantics affects comprehension effort when structure is held constant. • Sense/nonsense judgment task, comparing listeners’ ERPs for these four semantic groups: • Literal: Every soldier in the frontline was attacked. • Conventional: Every point in my argument was attacked. • Novel: Every second of our time was attacked. • Anomalous: Every drop of rain was attacked. (assignment of sentences to groups checked by naïve subject ratings) Grammatical Complexity Workshop

27. Measuring complexity for listeners: Event-Related Potential as an example • Result: Conventional metaphors required a short burst of additional processing effort when compared with literal sentences. Novel metaphors required a more sustained effort, similar to the effort observed in anomalous sentences. • Literal: Every soldier in the frontline was attacked. • Conventional: Every point in my argument was attacked. • Novel: Every second of our time was attacked. • Anomalous: Every drop of rain was attacked. • Comprehension of metaphors involves an initial stage of mapping from one concept to another; such mappings are cognitively taxing, implying that complexity (as processing effort) involves more than structure. • ERP matches our intuitions about complexity – and elaborates them. Grammatical Complexity Workshop

28. Measuring complexity for speakers:more about agreement errors Studies of formation of grammatical dependencies: even subject-verb agreement can be complex! • Producing subject-verb agreement: a “local noun” embedded in a subject noun phrase may interfere with the production of agreement - but structure constrains that interference. (Bock & Cutting 1992) • The key [PP to the cabinet]… is / are… • The key [PP to the cabinets]… is / are… • The key [RC that ___ locks the cabinets]… is / are… Grammatical Complexity Workshop

29. Measuring complexity for speakers:more about agreement errors • Producing subject-verb agreement: “Local nouns” that are embedded more deeply are less likely to interfere with agreement production (Franck, Vigliocco & Nicol, 2002) • The threat [PP to the president [PP of the companies]] … is / are… • The threat [PP to the presidents [PP of the company]] … is / are… • so syntactic structure can directly affect production complexity – and here, the more complex structure has made one component of producing this sentence easier! Grammatical Complexity Workshop

30. Measuring complexity for speakers:Other measures: • Onset latencies – how long before speaker starts to respond • Show difficulty in processing (e.g., competition between verb forms in subject-verb agreement: Haskell & MacDonald, 2003; Staub, 2009) • Eye tracking • Shows the interface between high-level message formulation and sentence planning (Brown-Schmidt & Tanenhaus, 2006) • Directed elicitation of alternate forms • Provides a measure of accessibility (e.g., production of optional complementizers allows speakers time to access upcoming constituents: Ferreira & Firato, 2002) Grammatical Complexity Workshop

31. So that was the fourth point: • Psycho- and neurolinguistics can’t provide a royal road to measuring the complexity of a grammar or a language, but they do provide tools to measure processing complexity of individual sentences/utterances for speakers vs. listeners or readers, and learners vs. skilled language users. Grammatical Complexity Workshop

32. 5. A valid measure of complexity will have to integrate across many linguistic levels (including semantics), and take frequency into account. • What does that mean, and what kinds of data support it? Let’s break it up into sub-claims that we can examine one at a time. Grammatical Complexity Workshop

33. What valid complexity measures would have to do: 5. Complexity measures must predict processing effort for multiple levels, and for interactions among levels, taking frequency into account O’Grady: “… the interaction of simple elements and phenomena can yield systems and effects of a qualitatively different and more complex nature.” Grammatical Complexity Workshop

34. First, let’s talk about processing effort for different levels and their interactions • A testable general complexity measure would have to be able to make predictions about the effort for processing short individual utterances or passages, and correctly predict the relative effort needed. • We’ve already seen that this effort depends partly on the choice of individual lexical items within those utterances or passages • She can’t sing it; Every minute of our time was attacked. Grammatical Complexity Workshop

35. 5. Complexity measures must predict processing effort for different levels and their interactions • While linguists may analyze a particular utterance into several components, the mind may store it and process it as a whole, or as both whole and analyzed. • Work on idiom blends suggests that idioms are both analyzable and stored as lexical items (Cutting & Bock, 1997) Observed speech error: “Help all you want”, blended from idioms/formulas ‘Help yourself!’ and ‘Take all you want!’ - idiom blends like this respect the internal structure of each component, because their surface syntax remains well-formed, so the speaker must have access to those internal structures. As many theorists now argue, • The ‘whole vs. analyzed’ opposition is much too crude, as many linguists have argued on theoretical grounds (Culicover 1999, among many others.) Grammatical Complexity Workshop

36. 5. Complexity measures must predict processing effort for different levels, and their interactions Start with some intuitive evidence for constructions • Particular verbs may be specified or preferred by a particular construction, such as mind if in the politeness formula Do/will [person A] mind if [event X]? (Do you mind if I sit here?). Easy to process, and not completely fixed lexically. (This construction is nested in more general patterns, cf. Would your mother have a fit if I…) • Conversely, when an unexpected word is used in a familiar construction – especially if it evokes a different construction - it can make the construction relatively harder to process (Would you care if I sit here?). • so ‘effort for different levels’ needs to include ‘mixed levels’, e.g. structures with some lexical items specified Grammatical Complexity Workshop

37. 5. Complexity measures must predict processing effort for different levels and their interactions Psycho- and neurolinguistics also tell us that the ‘whole vs. analyzed’ opposition is too crude.We even have to go beyond constructions and talk about collocations. • Collocations with high transitional probabilities, even when the collocations don’t form constructions (e.g., subject + aux collocations, he has or I am), are easier for speakers to produce. (That’s why we can have contractions across the NP-VP boundary!) • Smooth flow across this boundary is well exemplified in fluent aphasias, e.g. French jargon aphasia (Lecours et al. 1981) • Learning probabilities is a subconscious (procedural) and gradual process. Expectations that A will be followed by B, or that A will occur in structure α, become stronger over time, rather than clicking from ‘nothing’ to ‘all’.. Grammatical Complexity Workshop

38. ‘Agrammatic’ aphasic speakers show the effect of high sequential probabilities • … forgot the wash the dishes ‘forgot that she was washing the dishes’ • … I like the go home. ‘I’d like to go home.’ Once they have chosen the definite article to follow forget or like, these speakers are in trouble; both plug in familiar phrases (wash the dishes, go home) with appropriate semantic content, but in forms that cannot follow the. These utterances are difficult to explain in grammatical terms, because they show the article being substituted for the infinitive marker, and, even more strikingly, because the collocation ‘V+the’ goes across the major syntactic boundary between the verb and what should be the start of its NP object. Grammatical Complexity Workshop

39. 5. Complexity measures must predict processing effort...taking frequency into account • We can’t escape dealing with usage and frequency. • In the limit, there may be no empirically testable difference between ‘being stored as a unit’ and having very strong, predictable links from one sub-unit to another. So if your theory doesn’t permit multi-level or other kinds of complex ‘units’ and/or doesn’t recognize collocations that aren’t constructions, that’s not necessarily a problem. • What you do need is – at least - a way to incorporate item frequencies and transition probabilities into the representation of a structure after the structure has its words filled in (or during the process of getting them filled in). Grammatical Complexity Workshop

40. 5. Complexity measures must predict processing effort for different levels, and their interactions We’ll shortly consider some more evidence for constructions, as structures lying between the extremes of lexical items and full clauses. • But: the complexity of deploying a construction is not determined by construction frequency alone (e.g., Dutch word order in aphasia, Bastiaanse, Bouma & Post 2009) • For example, the interpretation or deployment of a construction, such as the English subject or object cleft, may be made better or worse by the existence of similar constructions (Dick et al. 2001:772): the ‘paradigmatic axis' is relevant to processing complexity – more on that soon. Grammatical Complexity Workshop

41. 5. Complexity measures must predict processing effort for different levels, and their interactions, taking frequency into account • Purely structural complexity does affect normal and aphasic language processing (e.g., Thompson & Shapiro 2007 showed that practice on structurally more complex clause generalizes to improvement on less complex clauses, but practice on simple clauses doesn’t generalize to more complex ones). • But in general: processing effort is a function of the interaction of structure and frequency at multiple levels, • Let’s look at a psycholinguistic study of normal speakers and a related one of aphasic speakers which demonstrates this. Grammatical Complexity Workshop

42. An example of frequency/structure interaction: Relative verb-(subcategorization) frame frequencies create a bias(reader’s expectation) that affects readers’ processing patterns and comprehension. ‘Shrink’, for example, has a syntactic bias towards the undergoer-subject argument structure – it is more frequently used in the ‘unccusative’ frame than in any other. The sweater shrank two sizes They shrank the sweater two sizes 5. Complexity measures must predict processing effort for different levels, and their interactions – taking frequency into account

43. The sweater shrank two sizes They shrank the sweater two sizes Eye movements during reading: verbs with similar syntactic biases pattern together, whereas verbs that are similar only in meaning do not, Garnsey et al. 1997. Comprehension: Clauses that conform to a verb’s bias, - They proposed X, They suggested that Y- comprehended faster and more accurately than bias-violating sentences with the same structure - They proposed that X,They suggested Y (ibid.) Production studies supporting this: Gahl & Garnsey 2004, 2006. 5. Complexity measures must predict processing effort for different levels, and their interactions – taking frequency into account

44. 5. …taking both structure and frequency into account • Gahl et al. 2003: People with aphasia comprehend sentences better when the verb is in its preferred frame. • Clauses with unaccusatives (undergoer-subjects) can be hard or easy, depending on how typical it is for the verb to be used in the unaccusative construction. • This supports a processing model that has relatively direct semantic construal of the verb frames of simple clauses, rather than indirect construal involving, e.g., traces.

45. The interaction of levels and the effect of frequency on complexity also support our sixth point: 6. Construction-based and usage-based approaches to grammar can provide insights into how grammars can come closer to reflecting what our brains do. Finally, let’s look at evidence for our last two points Grammatical Complexity Workshop

46. a valid measure of complexity will have to integrate across many linguistic level, and furthermore… Complexity measures must be compatible with different performance metrics There can be no single performance metric. Grammatical Complexity Workshop

47. Complexity measures must be compatible with different performance metrics There can be no single performance metric. Why not? • What is effortful for speakers does not always match what is effortful for hearers • Speakers and hearers have been shown to be sensitive to different structural features the same utterance types • Therefore: The goal of an overall complexity measure needs to be split into sub-goals; several roughly commensurable but sometimes incompatible measures are required. • Let’s look at some experimental evidence Grammatical Complexity Workshop

48. Complexity measures must be compatible with different performance metrics Identifying referents--Production Cognitive demands can swamp informationthat would improve referential success (Wardlow Lane & Ferreira, 2008) • Speakers are asked to name target objects (in the common ground) so that listeners can identify them, while being faced with privileged objects of varying saliency. • Speakers could name the target object by only using information that is common ground (e.g., “the heart”) or by also using privileged information (e.g., “the small heart”) that is useless and possibly confusing to the hearer. Grammatical Complexity Workshop

49. Complexity measures must be compatible with different performance metrics Identifying referents—Production experiment Cognitive demands swamp information that would improve referential success (Wardlow Lane & Ferreira, 2008) • Results: When the saliency of privileged information was increased, speakers made more reference to it, (e.g., identifying the target object as “the small heart”) than when such information was less salient, despite the risk of confusing the listener. • In this task, cognitive demands result in a lower processing load for the speaker when producing more (complex) descriptions. Grammatical Complexity Workshop

50. Complexity measures must be compatible with different performance metrics Identifying referents—Comprehension experiment using eye-tracking (following the listener’s successive visual fixations) Visual context affects ambiguity resolution (Spivey, Tanenhaus, Eberhard & Sedivy, 2002) Four conditions: Listeners heard either Put the apple on the towel in the boxor Put the apple that’s on the towel in the box Sometimes what they saw was this: A. ‘on the towel’ is redundant: there’s only one apple Grammatical Complexity Workshop