Conexionism

1. Conexionism

2. Conexionism (C) analog creier sau nu? • Retea larga neuroni: Computari + Rs • Computarile = Proiectare vector input pe un vector output • Diferenta paradigma simbolica si C - Natura reprezentarilor • C: Rs = Vectori (interpretati) • C: Reprezentari distribuite/locale + Procese in paralel

3. McCulloch & Pitt (’43), Selfridge (’59) Rosenblatt (‘62) - Perceptron Convergence Procedure, Doua niveluri cu regula Hebb • Minsky and Papert (’69): - SI, SAU = Linear separabile - Exclusive SAU = Nonlinear separabil Elman (pp. 61-63) • Solutia: Rs interne (prin unitati ascunse) (Elman, p. 64-65) • Reteaua invata ce legaturi are nevoie!!

4. McClelland (’81) (Jets and Sharks) Clark - 3 generatii Generatia unu: • “Unitatile input trimit semnal la unitatile ascunse. Fiecare unitate ascunsa computeaza iesirea proprie si trimite semnalul la unitatile output.” (Garson 2007)

6. Diferete tipuri de retea Elman p. 51

7. Paternul de activare a unei retele - Determinat de legaturile intre noduri • “Cunostiintele retelei sunt construite progresiv prin schimbarea legaturilor.” (Elman et al. ‘96) • Fiecare unitate input primeste input extern • Fiecare unitate ascunsa calculeaza valoarea proprie in functie de valorile de activare primite de la unitatile input. (Garson 2007)

8. “Fiecare unitate ascunsa este sensitiva la regularitati complexe = microtrasaturi.” • Each layer of hidden units - provinding a particular distributed encoding of input pattern (= an encoding in terms of pattern of microfeatures).” (Bechtel & Abrahamsen 2002 or Hinton, McClleland and Rumelhart 1986) → Nivelul subsimbolic (Smolensky ‘88)

9. Acelasi fenomen - intre unitatile ascunse si cele output • Activarea retelei = Legaturi pozitive (excitatorii) sau negative (inhibitorii) • Valoarea de activare - fiecare unitate: “Functia aduna contributiile de la toate unitatile care trimit, unde contributia unei unitati este definita de legatura intre unitatile de trimitere si cele de primite inmultit cu valoarea de activare a unitatii care trimite.”

10. aj = activarea nodului j care trimite output la nodul i • Legatura dintre aj si ai este wij • Inputul j la i este ai = wij aj • Ex: Nodul jare output = 0.5; legatura j cu i = -2.0 → (0.5 x -2.0) = -1 • Un node primeste inputuri alte noduri: Net input Neti = ∑wij aj Input total primit de nodul i

11. Outputul ≠ inputul (ca si neuronii) • Ce “face” un nod (functia raspuns) = Valoarea de activare a nodului = Functii liniare, mai ales nonliniare (functia sigmoid, etc.) (Elman, p. 53) • Nonliniar = “Valoare numerica a outputului nu este direct proportionala cu suma inputurilor.” (Clak)

12. Rumelhart and McClleland (‘81): 3 niveluri - cuvant, litera, trasaturi ortografice Ex: Nodul “trap” primeste input pozitiv de la nodurile literelor “t”. “r”, “a”, and “p”; celelalte noduri – inhibitate (Elman, p. 55) • Principiu: Similaritatea → Generalizarea - Dincolo de cazurile intalnite! • Asociationism – Regularitati statistice • Reteaua clasifica pattern (1100), tendinta clasifice un nou pattern (1101) in mod similar (vezi Elman, p. 91)

13. Similaritatea → Generalizari vs. “tirania similaritatii” (Mcleod,Rolls,Plunkett ‘96) • Telul: Legaturile pentru o functie • Regula de invatare “Hebb’s rule” (doar corelari-pereche – “obisnuinta”…) • Metoda de propagare intoarsa (backpropagation rule) • Nu corespunde procesului uman de invatare! • Alte metode: self-supervised learning, unsupervised learning) (Elman)

14. Backpropagation rule • Pasul I: Eroarea = Diferenta output actual si output target • Pasul II: Ajustarea fiecarei legaturi → Descreste eroarea (Churchland p. 43) • “Feedback-ul local - supervisor - crestere sau scadere usoara – legaturi → Imbunatatirea performantei retelei • Repetata legatura cu legatura + nivel cu nivel → Panta descresterii erorii” (Clark)

15. Algoritmul: “Propagam informatia erorii in spate, de la unitatile output la cele ascunse [si la cele input]” (Elman) • “Abilitatea - invata - schimba in timp - NU functie a unei schimbari explicite in mechanism, ci consecinta intrinseca a invatarii insasi. Reteaua invata ca si copii.” • Invatarea ca gradient descent in spatiu legaturii (Elman, p. 72)

16. Memorie de superpozitie (“Superpositional storage”) • “2 Rs superpozitionate daca resursele ce reprezinta itemul 1 - coextensiv cu cele ale itemul 2” • “Incodeaza informatia pt. item 2 prin amendarea setului de legaturi originale -prezerva - necesar functional(anumite patternuri input-output) sa reprezinte item 1 - simultan cu necesitatea functi-la sa reprezinte item 2.” (Clark)

17. Superpozitie - 2 trasaturi combinate: • Folosirea Rs distribuite • Folosirea regulii invatare - impune metricasemantica asupra Rs • “Semantically related items - represented sintac-lly related (partially overlapping) patterns of activation.” • (Ex: cat and panther vs. fox sau Elman, p. 91) • Implicit (tacit)–explicit or potential-actual → “Recreation” of a R at t1 (Horgan ‘97)

18. “Semantic similarity R-l contents - echoed R-l vehicle.” (Clark) → Prototype extraction (category/concept) + Genera-on • Rs distribuite = Clustere in spatiul stare a unitatilor ascunse (unele retele) (Shea ’07) • Rs = Vehicles of content - individuable non-semantically (Shea) • Reguli – antrenament/invatarea → “Metrica semantica” a Rs dobandite (Clark)

19. B. Sensitivitate de context intrinseca(“Paradigma subsimbolica”, Smolensky) • Computationalism (Sistem Fizic Simbol) = Transparenta semantica • LOT: Compositionalitate, sistematicitate, productivitate • Rs simbolice: Sintaxa + semantica combinatoriala

20. Fodor & P: C NU are sintaxa + semantica combinatoriala • C = Doar implementare a sistem simbolic vs. • C - “fine grained context sensitivity” (Clark) • o R pt un item data de “activitatea patternului distribuit - contine subpatternuri pt. trasaturi … o retea poate reprezenta instantieri a unui astfel de item, care poate sa difere referitor la o trasatura sau mai multe.”

21. “Such ‘near neighbors’ [clusters] - represented by similar internal R-al structures - vehicles of several Rs (activation patterns) will be similar to each other in ways that echo semantic similarity of cases – semantic metric in operation” (Clark) • “Continutul elementelor in program subsimbolic nu recapituleaza direct conceptele” (Smolensky) si “unitatile nu au continut semantic precum cuvintele in limbajului natural” (Clark)

22. Diferentele de activare la fiecare unitate oglindesc detalii a diferitelor functii mentale in interactiune cu “contextul lumii reale” • Cunoasterea dobandita prin antrenare - analizata prin “postraining analysis” (statistical analysis and systematic interference) sau “cluster analysys”

23. 1 stare conex. = 1 pattern activare (in spatiu activare) constituita din patt-nuri constituiente • Pattern activare - NU descompus in constituienti conceptuali (ca la simbolic) (Smolensky ‘88) • Descompunerea conex. = Aproxim - Patternul complex (subp-uri constit.) nu poate fi definit; depinde de context • Structura const-enta a subp-lor - puternic influentata de structura interna inclusa in system (Ex. Ceasca de cafea)

24. Constituientii conceptuali a starilor mentale = Vectori de activare • Rs conex. are constituienti = Parti functionale a Rs complexe (NU parti effective a schemei concatenative, relatiile constituiente, instatieri de tipul relatiei parte-intreg) • Reteaua – invatare asociationista – nu are nevoie de structura compozitionala

25. 1) Simbolic: Compozitiona-te concatenativa 2) C: Compozitiona-te functionala 1) Concatenarea = “Unind constituienti succesivi fara alterarea lor” = Rs “tb. sa preserve tokens unei expresii din constituienti si relatia lor secventiala” 2) Compo-tea func-la = A avea R prin recuperarea partilor in anumite procese

26. 1) “Symbolic: Context of a symbol is manifest around it and consist of other symbols 2) Subsymbolic: Context - symbol - manifest inside it, and consist of subsymbols” Ex: “Ceasca cu cafea”: • Structura compoz. = Sens aproximativ • “Not equivalent a context-independent R of coffee + a context-independent R of cup … sticking together - symbolic structure concatenating them together → Syntactic comp-nal structure like “with (cup, coffee)”

27. Nets - “Not involve computations defined over symbols. … … accurate (predictive) picture of systems processing - at numerical level of units + weights + activation-evolution equation” → No syntactically identifiable elements • Nets: Same elements for syntax + semantics” (Smolensky 1991)

28. “Mental Rs + Processes: Not supported by the same formal entities • Nets - 2 levels: 1) Formal, algorithmic specification of processing mechanisms 2) Semantic interpretation → Must be done at 2 levels of description (Smolensky 1991)

29. “1 nivel: Procesele mentale reprezentate de nivelul de descriere numerice a unit-lor, legat-lor, ecuatii de evolutie activarilor (NU interpretare semantica) (Smolensky) • “2 nivel: Activitati la nivel larg permit interpretari, dar patt-le astfel fixate nu sunt descrieri clare a procesarii” • “Metrica semantica a sistemului impune o similaritate pt. continut acolo unde exista o similaritate pt. vehicul (=patter-ri similare).” (Clark)

30. Sistemele de codare exploateaza “more highly structured syntactic vehicles than words” → Economical use of R-l resources • “Free” generalization - a new input if it resembles an old one… will yield a response rooted in that partial overlap → Sensible responses to new inputs • Graceful degradation = Sensible response given some systemic damage, pattern completion, damage tolerance

31. Fodor and McLaughlin ‘90, McLaughlin ‘93 vs. Smolensky, Elman Generatia a doua • Structura temporala • Retelele recurente (SRN) → Memoria termen scurt (Elman ’90, ’91, ’93) (Elman, p. 75) • Elman ’90: Predictie cuvinte/litere

32. Time in language-sentences, 2 features: (1) Processed sequentially in time (2) Exhibit long-distance dependencies - form of one word - depend on another that is located at an indeterminate distance.” (Verbs agree subjects, a relative clause - between subject and verb) (Bechtel &A) • Net - incorporate such relationships without explicit Rs of linguistic structures

33. SRN - Prezice cuvintele succesive: • Inputul: 1 cuvant/litera = 1 nod (Localist) • Outputul: Prezicerea cuvantului urmator • Backprop: Adjustarea fiecarei legaturi la fiecare eroare, apoi urmatorul cuvant • Proces repetat (mii propozitii) • Unitatile ascunse = Spatiu 150-dimensiuni • “Retea – Invata sa reprezinte cuvinte care se comporta in moduri similare (= proprietati distribuite similare) cu vectori care sunt apropiati in spatiu intern R-l”

34. Spatiu = “Hierarchical clustering tree” a patt-lor activare unit. ascunse (cuvinte) • “Capturing hidden unit activation pattern corresponding each word, measuring distance betw. each pattern and every other pattern. These pattern distances = Euclidian distance betw. vectors in activation space” → Tree “placing similar patterns close and low on tree, and more distant groups on different branches.” = VERBS, animates, NOUNS, inanimates (Elman, p. 96) • Context-sensitivity = “Tokens of same type are all spatially proximal, and closer to each other than to tokens of any other type.” (Elman)

35. Nets: “Descopera” categorii (verbe, etc.) ”properties that were good clues to grammatical role in training corpus used.” (Clark) = “Discovering lexical classes from word order” (Elman ’90) • “R-I form and r-l content often can be learned simultaneously.” (Elman ’90) • “Cluster analysis” (= mapping distribution of activation points in state space) network learned a set of static distributed symbols → Relations of similarity + difference between static states) (Clark)

36. Training tends to cause activation vectors to cluster together in hidden layer state space → A network’s R-l structure • Clusters (regions in state space) = Vehicles of content • Internal mechanism → Operations on vehicles: a presented sample activates a hidden layer cluster, which in turn activates an output layer cluster • Activation cluster = Activation particular pattern of distributed activation (Shea ‘07)

37. Activation produced by classified novel samples (NSs) falls into existing clusters • Classifying NSs = Proximity to existing points in hidden layer state space of activation produced by old Ss (A common property) • Viewpoint of individual patterns of activation: NSs – Not same as old Ss (Shea ’07) • Viewpoint of clusters: NSs produce activation in same hidden layer clusters as old Ss → NSs are being treated in the same way as some of old Ss

38. No separate stage of lexical retrieval • No Rs of words in isolation • Internal states following input of a word - reflect input taken together with prior state • Rs = Not propositional; their information content changes over time depending on current task (Elman ‘96) • SRN - No semantics; net learns to group encoding of animate objects together only because they were distributed similarly in training corpus! (Bechtel & Abrahamsen ‘02)

39. Cotrell – Face recognition • 64 X 64 input units (256 levels of activation/ “brigthness”); 80 hidden; 8 output • Training set: 64 photos for 11 faces + 13 photos nonfaces • 8 output units: first for faces-nonfaces; second for male-female; third for name; 5 for name • Each hidden unit – links to all input units: holism -------------------------- Recunoaste o foto; recunoaste o persoana in poza necunoscuta (= categorii (prototip) = clusturs) (98%); discrimineaza femeie sau barbat din persoana/poza noua (81%); 1/5 lipsa din foto (71%) (Churchland ‘95, pp. 40- 49)

40. Artificial Neural Networks: A tutorial by A. K. Jain, J. Mao, K. Mohiuddin, 1996

41. C. Strong Representational Change • Retea - “Cunoastere” prin mecan-le invatare (“Past tense learning” - R & McC) • Arhitectura (unitati, niveluri) – inascuta, nu si simbolurile. Legat-le = Continutul - Dependente de antrenare • Bates, Elman: Invatate 90%, innascute 10% • Antrenarea → Cunoastere si procesare = Modularitate functionala • Antrenare - Schimbari calitative (U-curve shape, Plunkett and Marshman ‘91, ‘93)

42. “Cunoasterea si procesarea puternic amestecate!” (Clark) • “New knowledge - to be stored superpositionally - amending existing weights” (Clark) • “Classicist thinks of mind - Static, recombinable text • Connectionist - Highly fluid environm-lly coupled dynamic process” (Clark)

43. Rumelhart, McClelland (‘86) – Prediction of past tense English verbs”: Regular vs. irregular verbs • 2 layers → Problema: Past tense = Nonlininar 1) Simbolic: 2 mecanisme (vb-s regulate/ neregulate 2) C: 1 mechanism = Retea de conexiuni pt. vb-s regulate/neregulate • Retele - 1 nivel rezolva probleme liniar separabile • Past tense = Problema nonlineara

44. Plunkett, Marchman (‘91, ‘93) – Unitati ascunse → The “U-shape form” pt. reproducerea patternuri de eroare observate la copii (Elman) • “Differentiation at behavioral level imply NO differentiation at level of mechanism • Regular and irregular verbs behave differently even though represented and processed similarly in same device.” (Elman ‘96)

45. Dezbateri Pinker & Prince (‘88) – Nets: Good making associations + matching patterns • Limitations in mastering general rules such as formation of regular past tense. (Garson or Bechtel & Abrahamsen) • ‘’91, ’93 Plunkett & Marchman: U-Shape form

46. Generatia a treia -“C dynamic” • Trasaturi neurobiologice realiste la noduri + legaturi • Unii filosofi: Retelele – distrib, Rs similare cu structura creierului → Retelele = Implementare a mintii • Altii: Retelele = Minte • Nets: For motor control, pattern recognition (vs. planning and sequential logical derivation)

47. C vs. Simbolic • C elimina homunculusul • “Connectionist models - attractive - computational framework for - emergentproperties” (Elman) • R. distribuita = Computational – Incodeaza informatia privitor la similaritati si diferente (Clark) • NU modularitate - Incepe antrenarea → “Huge difference betw. starting modular and becoming modular.” (Elman)

48. (a) Sistematicitatea (S) (Fodor & P) • Fodor’s LOT • Marcus (2001, “The Algebraic Mind: Integrating Connectionism and Cognitive Science”); Mintea: • Reprezinta relatii abstracte intre variabile • Are un mecanism pt. Rs recursive • Sistematicitate • Distinge intre individual si general

49. Replici • Simbolic – NU e singura directie pt S • S – Dedusa din structura gramaticala a limbajului (Clark) • Retele pt. S: Smolensky’s tensor product, Chalmers’s net, recursive autoassociative memory (RAAM)

50. Elman (2005) • Explaining the shape of change • New views on how knowledge may be represented • The richness of experience Language acquisition (nonliniar) + Perception (new mechanisms vs. same mechanism); U-shape form (past tense) – Aquistion of a rule vs. Item-based Learning)