Connectionist Models of Development Jeff Elman Department of Cognitive Science UCSD

1. Connectionist Models of Development Jeff Elman Department of Cognitive Science UCSD

2. The issue: Nature vs. Nurture • Today’s class: • What biology can do • What learning can do

3. Two possible ways to control development. . .

7. Less DNA More DNA

11. mossy cells muscle cells pyramidal cells Purkinje cell sperm cells

14. Genetic conservatism

17. from butterfly host from alderfly host Trichogramma (wasp)

18. from butterfly host from alderfly host Trichogramma (wasp)

19. Lesson 1: no “special purpose” genes • Lesson 2: genetic conservatism • Lesson 3: change through “tweaks & twiddles” • Lesson 4: the importance of the environment

20. Modeling learning with “neural networks”

21. The first computers. . .

22. Walter Pitts

23. e1 en i1 in 

24. John von Neumann • “Every digital computing device contains certain relay like elements with discrete equilibria. Such an element has two or more distinct states in which it can exist indefinitely...It is worth mentioning that the neurons of the higher animals are definitely elements in the above sense... Following W. Pitts and W.S. McCulloch...we ignore the more complicated aspects of neuron functioning.”

25. The Colossus(one of ten) Bletchley Park, 1944

26. Is the human brain a computer?

27. CPU and memory are separate • the processor is very fast • serial operation • very reliable parts; but the whole computer may break easily • computers are programmed • the memory and CPU are the same • the parts (neurons) are very slow • parallel; enormous interconnectivity • unreliable parts; but the whole brain is very reliable • brains learn

28. Things digital computers do well • Numeric computation • Symbolic computation • Repetitive operations • Follow instructions Things human brains do well • Find subtle patterns (intuition?) • Find novel solutions by analogy • Learn

29. ELIZA and the problem of natural language processing the problem with SCRIPTS

30. Eliza Eliza: Hello. I am Eliza. How can I help you? Me: You can’t help me at all. Eliza: You like to think I can’t help you, do you not? Me: I am sure of it. Eliza: How long have you been sure of it? Me: Ever since I was 10 years old. Eliza: Why do you tell me you were 10 years old just now? Me: You are not a real person Eliza: Does it please you to believe I am not a real person?

31. Scripts, Schemata, and Frames John went to McDonald's with Mary. They ordered a hamburger and fries. John gave the woman at the counter $3.49. They left. • What is McDonalds? • Why did John and Mary go there? • Why did John give the woman money? • Did they eat anything?

32. John went to McDonald's restaurant with Mary. They ordered a hamburger and fries. Suddenly, Mary's husband Lenny burst in with a shotgun. John hid under a table. • Why did John and Mary go to McDonalds? • Why did John hide under the table? • Did John get to eat his Big Mac? • What were they doing before going to McDonalds? • Where will Mary’s husband spend the rest of his life?

33. fragile (vs. graceful degradation) • fixed repertoire • limited to well-defined domains • how to get the knowledge in the system

36. What is a neural network?

38. Learning vs. Programming

39. InputOutput 2 4 7 49 4 16 13 ?

40. InputOutput fever, runny nose, muscle ache -> flu no fever, runny nose -> allergies no fever, skin rash -> staphylococcus infection fever, runny nose -> ?

41. How do you pronounce “ou”? “aw” “au” “uh” “oo” “oh” “ought” -> “ouch” -> “tough” -> “through” -> “though” -> “plouty” -> ? “plough” -> ?

42. “The Voringian binx glorphed the Knappoboor.”

43. Learning to read out loud • Discovering where the words are • Discovering categories

44. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother. My grandmother lives near us. I like to visit my grandmother.

45. [first stages of learning]

46. “I like to go to my grandmother’s house. Well…because she gives us candy. Well ... and we eat there sometimes. Sometimes we sleep overnight there. Sometime when I got to go to my cousin’s ...”

47. A (surprisingly) hard problem: Where are the words?

48. Whereareth es il ens es b et w eew or d s Whereareth es il enc es b et w eenw or d s

49. “Many years ago, a boy and girl lived in a castle by the sea. They played with a dragon….”

50. manyyearsagoaboyandgirllivedinacastlebytheseatheyplayedwithadragonmanyyearsagoaboyandgirllivedinacastlebytheseatheyplayedwithadragon 00011 10111 00010 11011 11011 00100 10111 01111 01000 10111 11000 10010 10111 . . .