PDP Models and American Health Care Reform

1. PDP Models and American Health Care Reform Mark S. Seidenberg NCPW13 BCBL San Sebastian 2012

2. Observation: Many core concepts of the PDP approach have been broadly assimilated into cognitive science/neuroscience But the modeling , not so much (present distinguished company notwithstanding) skepticism about relevance/adequacy in areas such as language acquisition arising from close analyses of specific PDP models availability of alternative approaches Seidenberg NCPW13 talk

3. Observation II: People will endorse PDP concepts as long as you call them something else Like health care debate in US: Click here for movie controls Seidenberg NCPW13 talk

4. Obamacare vs. PDP • maintain current insurance • promote access to affordable health care • no denial based on pre-existing conditions • individual mandate (the thing that pays for the good stuff) • distributed representations • interactive processing • computation of best fits • PDP models Seidenberg NCPW13 talk

5. Why? Let us look. Three case studies, all alike 1. Diagnosis of a fatal problem models behave differently from people broad implications, widely repeated attention heads elsewhere 2. Diagnosis turns out to be wrong critiques don’t support broader implications less widely known (like getting an audience for a failure to replicate) 3. There is, however, a related problem of considerable interest producing exciting work but need more; have to overcome (1) Seidenberg NCPW13 talk

6. Case 1: Catastrophic interference 1. Diagnosis of problem McCloskey & Cohen, 1989 Unlike people, simple feedforward nets exhibit unwanted retroactive interference Based on close analyses of models of simple arithmetic task 2. Solution is interleaving Hippocampus in complementary systems model (MMO, 1995) Life (in which experience is massively interleaved) The type of catastrophic interference that McCloskey-Cohen focused on does occur occasionally certain verbal learning experiments unusual circumstances like Korea  France émigrés (Pallier et al.) Seidenberg NCPW13 talk

7. 3. The interesting related problem Massive entrenchment! Reduction in plasticity associated with expertise Example: Critical periods in language learning “Paradox of success” (Seidenberg & Zevin, 2006), Expertise with L1 makes it difficult to absorb L2 Some models in this area (Ping Li, others). Haven’t gone that far. A recent example: , Seidenberg NCPW13 talk

8. Impact of Dialect Variation in the USon Learning to Read Achievement gap in reading 1. African Americans (and other minorities) perform less well on tests of reading and other subjects compared to whites; 2. “gap” has been persistent for many years 3. poor reading skills a problem for individuals and society Seidenberg NCPW13 talk

9. Why? Not just poverty, school/teacher quality Possibly related to language experience? Major US dialects: Standard American English African American English These dialects overlap more than 2 languages But also differ a lot: phonology, morphology, syntax, discourse Seidenberg NCPW13 talk

10. Dialect mismatch effects Home dialect AAE vs. school dialect SAE When schooling starts, child has to learn more of the second dialect learn in less familiar dialect, in noisy environment using books written SAE Dialect differences make learning a more difficult task than for child who uses same dialect at home and in school. But all are judged against same achievement milestones. “Gap” ensues Other factors like SES may exacerbate further Seidenberg NCPW13 talk

11. We wanted to examine impact on reading. Obvious area: how differences in pronunciation affect acquiring basic decoding skills Seidenberg NCPW13 talk

12. GOLD, FLOOR, and LOW rhyme in AAE Pronunciation differences Many words pronounced the same (at phonemic level) Many words pronounced differently Percentage varies with dialect density 30% of words and higher Seidenberg NCPW13 talk

13. Teacher: G-O-L-D, that’s “gold” [child searches spoken language vocabulary for “gold”] Child: Ohhh, “gole” Seidenberg NCPW13 talk

14. Thus: Spelling-sound correspondences are more complex for AAE speakers. We have models for that…. Seidenberg NCPW13 talk

15. Contrastive words: different pronunciations in SAE, AAE bound old toast Non-contrastive: same pronunciation in both dialects brush air stage Latencies do not differ in ELP data base. Seidenberg NCPW13 talk

16. Naming latencies as a function of AAE density Children (N =22, M age =11.4 years old) Adults (N = 32, M age = 35.5) Seidenberg NCPW13 talk

17. Modeling Once you see the set-up, effects are obvious orthphon model Learns phonology first Then learns to map spellings onto phonology SAE learn map spellings onto known SAE pronunciations AAE learn to pronounce words in SAE whilecontinue using AAE phonology in speech Seidenberg NCPW13 talk

18. Model (based on Harm & Seidenberg, 1999) Training corpus: 1700 words from 2nd grade norms SAE version AAE version: about half the pronunciations are different Seidenberg NCPW13 talk

19. SAE match: SAE-SAE AAE match: AAE-AAE Mismatch: AAE-SAE Seidenberg NCPW13 talk

20. Training on both dialects Seidenberg NCPW13 talk

21. Summary About achievement gap: dialect mismatch slows learning Playing field is not level Models suggest ways to fix this. About models: entrenchment, proactive interference Seidenberg NCPW13 talk

22. Case 2: Language acquisition 1. Diagnosis of the problem Language has properties that can’t be captured by NNs Rules (Pinker), algebraic rules (Marcus), procedural knowledge (Ullman) Demonstrations: Marcus et al. Lather, rinse, repeat 2. Second opinions: Plenty of people have taken issue with these claims rule-governed only under idealization of data “competence theory of performance”: Seidenberg & Plaut (in press?) semantic-phonological theory of the past tense (not rules-exceptions) improved models (Altmann, others) Seidenberg NCPW13 talk

23. 3. The interesting related problem: What is “Statistical learning”? Language learners learn from statistics of the input Process starts in infancy Many studies examining what kinds of statistics are learned Little of the research makes contact with PDP/connectionist models/concepts Newport (2010) sees progress in “the movement in many parts of psycholinguistics from rules to connectionism to statistical learning” (p.  369).    “Statistical learning” is not Obamacare! Seidenberg NCPW13 talk

24. Irony: Linguists’ early criticism of connectionist/PDP models languages exhibit lots of regularities depending on how you count models are too powerful; can learn any arbitrary association can’t explain why languages exhibit some regularities and not others why people can learn some things and not others Current research on “statistical learning” in language acquisition same issues! lots of different statistics can be studied in artificial language studies what are the general principles? why are some regularities learnable and not others? Seidenberg NCPW13 talk

25. They’ve thrown the theory of how the child learns out with the connectionist bathwater. Need more models, not fewer Recent example: Willits (2012) thesis, UW Seidenberg NCPW13 talk

26. Here’s what Jon did • Studies of non-adjacent dependencies • which are everywhere in NL • drink, drank, drunk • was cooking • The The womangavethebookto the boy • The key(s)to the cabinets is/are on the table. • S -> NP+ (S) +VP • Challenging learning problem. • Many recent behavioral studies of infants, toddlers using • artificial grammar methods • Not much connection to earlier AGL research Seidenberg NCPW13 talk

27. Representative studies: learning an AxB pattern (auditory presentation) PelWadimRud PelKiceyRud PelPuserRud VotWadimJic VotKiceyJic VotPuserJic Pel Wadim Rud Pel Kicey Rud Pel Puser Rud Vot Wadim Jic Vot Kicey Jic Vot Puser Jic Vary number of As, Bs, Xs Surprisingly hard to learn Gomez, Maye, Newport & Aslin Seidenberg NCPW13 talk

28. Willits (2012) Used SRNs to address 4 phenomena: 1. Learning distance-invariant nonadjacent dependencies AxB with 0-3 intervening items 2. Impact of correlated semantic cue (AB are both animals or both foods) 3. Impact of consistent but semantically-unrelated cue (A animal, B food) 4. abstract rule-like knowledge (Marcus) Learn test ABA ABA’ (same pattern, new items) ABB ABB’ Key change: let model learn during test phase (like babies do). Then model can learn test pattern with new items--with savings. Seidenberg NCPW13 talk

29. Conclusions 1. Overcoming purported limitations of SRNs, yes. Behavior is similar to humans, yes. 2. More important: Analysis shows reasons why models work. Implications re: learnability of other abstract, “rule-like” properties of language un-learnability of some types of problems which should be unlearnable for people too Seidenberg NCPW13 talk

30. Case 3: Linking Brain and Behavior Problem: PDP models motivated by linkage to brain, “neurally inspired”, etc. But, most models have not been very constrained by brain data (PDP, neuroimaging developed in parallel at about the same time) 1. Diagnosis: poor fit because the brain doesn’t work that way, e.g., backprop, units ≠ neurons, etc. 2. Second opinion: things are moving along fine Recent models that are more closely tied to brain Plaut, Lambon Ralph, Taiji Ueno, McClelland, others here Seidenberg NCPW13 talk

31. 3. Interesting related problem: more please! Integrate PDP models with brain data Otherwise differences in activation for words vs. nonwords = word level representations Grain of neuroimaging data is like grain of behavioral data Models can indeed apply to both Seidenberg NCPW13 talk

32. Recent example from our group Jeff Binder (Medical College of Wisconsin Will Graves (now at Rutgers) Me, Tim Rogers (Wisconsin) Seidenberg NCPW13 talk

33. How many ways are there to be a skilled reader? Do skilled readers (e.g., of English) read the same way? Old question: Baron & Strawson (1976) “Chinese” vs. “Phoenician” readers “visual” “phonological” orthsem orthphonsem Seidenberg NCPW13 talk

34. Maybe different division of labor? Computing a code depends on input from various parts of the system Efficiency arises from division of labor between sources Affected by type of word, type of writing system Plaut et al., 1996: computing phonology Harm & Seidenberg, 2004: computing semantics Individual differences could be related to reading skill, experience Seidenberg NCPW13 talk

35. New work looked at impact of semantics on reading words aloud In principle words can be read without using semantics (as in the DRC-CDP+ models) However, in our model, orthsemphon is available, and could facilitate performance for some words or readers Semantic effects on naming: are there any? YES: Strain et al., 1995; Hino & Lupker, 1996; Lichacz et al., 1999; Strain & Herdman, 1999; Hino et al., 2001; Shibahara et al., 2003, and several others. NO: Monaghan & Ellis, 2001; Brown & Watson, 1987; de Groot, 1989; Baayen et al., 2006). Seidenberg NCPW13 talk

36. Perhaps there are individual differences… Study: examined use of semantics in reading aloud among skilled readers (college graduates, med students) Determine if individual differences are associated with neuroanatomical variation in relevant parts of reading network. Seidenberg NCPW13 talk

37. 1. Graves et al. (2010): 18 subjects read 465 words aloud in scanner 2. Effect of semantics on naming indexed by impact of imageability. Also looked at freq, consistency, bigrams, number of letters, other factors. 3. Graves et al. (2012): Left hemisphere semantic and phonological ROIs based on results of 2010 study semantic: AG ITG/ITS phonological: pSTG pMTG 4. DTI tractography to measure volumes of pathways Seidenberg NCPW13 talk

38. Seidenberg NCPW13 talk

39. Semantic effects on naming correlated with white matter volume in sem-phon pathways Anatomy, not strategy Seidenberg NCPW13 talk

40. Seidenberg NCPW13 talk

41. Everything A-OK? Some reasons why models get a bad name 1. we take credit for good behavior, and discount the bad behavior implementations limited, etc etc like: model learns something that people learn but takes 10 million trials heads I win, tails you lose Properties that hold over many models? Requires doing a lot of models. Like doing replication experiments. Takes lots of time, analysis. Could be hard to build a career around. Seidenberg NCPW13 talk

42. 2. What about taking learning seriously? Problem wasn’t that backprop wasn’t neurally realistic It isn’t behaviorally realistic. what is learning really like? conditions vary: explicit extrenally provided teacher external or self-generated error signals that are noisy, partial, inconsistent, wrong general rather than specific etc. Can be addressed (h/t O’Reilly). Maybe models would learn on the human order of magnitude. Seidenberg NCPW13 talk

43. So, there is progress, there are obstacles, there are future directions. Why is this important to recognize? In the famous words of the philosopher, Those who fail to remember history are doomed to fail to remember repeating it. Carlos Santana Seidenberg NCPW13 talk

44. Thanks for listening! Thanks also to collaborators: Dialect research: Julie Washington GSU Daragh Sibley Haskins Acquisition research Jon Willits Indiana Jenny Saffran Wisconsin Reading brain Jeff Binder MCW Will Graves MCW And Jay for introducing me to PDP. Seidenberg NCPW13 talk