Brain Code and Brain Reading

1. Brain Code and Brain Reading Prof.dr. Jaap Murre University of Maastricht University of Amsterdam jaap@murre.com http://neuromod.org

2. What principle should it use? Storage capacity Resistance to damage Access speed Serve necessary calculations What constraints can be identified? Neurons are very noisy They can do only simple calculations The are severely limited in how they can communicate How does the brain represent information?

3. Localized versus distributed coding • Distributed coding • 1010111000101100110101000110111000 • Extremely localized coding • 0000000000000000010000000000000000 • Sparse (or semi-distributed) coding • 0000100000100000010000000010000000

4. Distributed coding • Maximizes storage capacity • Resistant to damage • Able to implement complex calculations • May need high metabolism (energy, O2) • Overlap of codes can cause severe interference in processing and learning

5. Localized coding • Is able to store only a few representations but also forces category formation • Sensitive to loss (causes very specific disorders) • Mainly useful for theoretical purposes

6. How can codes come into existence? Example: Competitive learning • Competitive learning is a form of unsupervised learning • Needs a large degree of localization (or very sparse codes) to work • Forms categories on the basis of regularities (is also called regularity learning) • These networks are able to signal novelty

7. Example of competitive learning:Stimulus ‘at’ is presented 1 2 a t o

8. Example of competitive learning:Competition starts at category level 1 2 a t o

9. Example of competitive learning:Competition resolves 1 2 a t o

10. Example of competitive learning:Hebbian learning takes place 1 2 a t o Category node 2 now represents ‘at’

11. Presenting ‘to’ leads to activation of category node 1 1 2 a t o

12. Presenting ‘to’ leads to activation of category node 1 1 2 a t o

13. Presenting ‘to’ leads to activation of category node 1 1 2 a t o

14. Presenting ‘to’ leads to activation of category node 1 1 2 a t o

15. Category 1 is established through Hebbian learning as well 1 2 a t o Category node 1 now represents ‘to’

16. Kohonen self-organizing map • Winner-take-all competition • Network has 2D (or 3D etc.) layout • Weights of neighbors of the winner are updated with the winner • The map tries to mirror the input space • This works even with semantic spaces

17. Kohonen map: winner-take-all

18. Kohonen map: winner-take-all Initial weights are random Initial winners will be random as well As time goes by, winners to similar inputs will tend to be located close together

19. Weights are updated to the winner and to its neighbors Weights are moved towards input For example: Inputs are: .9 .9 .7 Weights are: .5 .3 .9 New weight are: .6 .4 .8 .9 .9 .7

20. Kohonen self-organizing map • The size and shape of the neighborhood determines the final layout • The size of the neighborhood is usually diminished with training • The learning rate is typically also slowly diminished • Training may take a long time

21. Example: 2D map tries to capture 3D color space (by Simon Lucas) • Input vectors are drawn randomly from RGB (red, green, blue) color space • Node color reflects the node’s incoming weights (it will respond best to that color) • Depending on the learning rate and neighborhood different organizations emerge • No single fixed organization is best in this case • Other example:

22. Sparse coding • Forms a good middle ground between fully distributed and extremely localized coding • Is computationally sound in that it allows very large numbers of representations with a small number of units

23. Brain codes in the motor system Codes for action and movement

24. Cortical anatomy of the motor system: lateral view

25. Medial view

26. Schematic overview of the motor system

27. Basic questions regarding motor control can nowadays be answered • How are motor movements represented in the brain? • How are they used in the production of movement? • Which brain areas are involved?

28. How to be precise with noisy components Area 5 neuron during repeated reaching movements: each individual trial gives a rather imprecise signal

29. Population coding • Population coding allows precise representations on the basis of (very) noisy or even damaged components • Population coding is based on the statistics of averages • They rely on coarse-coded neural representations

30. Coarse coding • If a neuron’s representation responds to ‘many’ inputs, this is called coarse coding • The advantage is that more accurate representations can be formed by suitable combination of the coarse representations

31. Why coarse coding works • If we move along a straight line, each time we cross a receptive field boundary one neurons changes its activation: • the representation changes.

32. In primates abundant evidence exists for coarse coding

36. Georgopoulos shows that movement is coded in population vectors

37. Population vectors give accurate movement direction signals

38. Motor cortex sets up the signal, but execution is dependent upon other areas

39. Response competition

40. Spinal cord

41. Coarse maps of limb movements in the frog • Spinal cord of frogs does significant motor processing • Frog can still ‘clean’ itself after severing of cord (dogs can also still scratch themselves) • The data suggest that even at a spinal level coarse coding is used • It is likely that similar types of coding are used in mammals

42. Cats with severed spinal cord could still walk on a treadmill

43. Method followed by Emilio Bizzi Based on the idea of ‘muscles as springs’ by Feldman

44. Limb movements in frog spinal cord are coded with respect to their end-positions

45. The interactions of force fields can be described by vector calculus Fields A and B combined predict field <AB> (see C). When A and B are stimulated the resulting field (see D) corresponds to the theoretical field <AB>

46. Brain codes in the visual system Is our knowledge represented in a localized or distributed fashion?

47. Many (395) single cell recordings in monkey pre-frontal cortex Trying to find categories in the brain 82 out of 395 were category selective Freedman, Riesenhuber, Poggio, & Miller (Science, 2001)

49. Cells predict the correct category (single-cell, lateral pre-frontal in monkey)

50. Brain Reading? Jim Haxby’s (2001) study