Image Stabilization by Bayesian Dynamics

1. Image Stabilization by Bayesian Dynamics • Yoram Burak • Sloan-Swartz annual meeting, July 2009

2. What does neural activity represent? • In Bayesian models:probabilities Accumulated evidence in area LIP Shadlen and Newsome (2001) Direction of motion: single, static variable

3. What does neural activity represent? • In Bayesian models:probabilities Accumulated evidence in area LIP Shadlen and Newsome (2001) Direction of motion: single, static variable What about multi-dimensional, dynamic quantities?

4. Foveal vision and fixational drift

5. Foveal vision and fixational drift Fixational drift is large in the fovea: - between micro-saccades - ~20 receptive fields - between spikes (100 Hz) - ~2-4 receptive fields ! Image from: X. Pitkow cone separation: 0.5 arcmin

6. Foveal vision and fixational drift Fixational drift is large in the fovea: - between micro-saccades - ~20 receptive fields - between spikes (100 Hz) - ~2-4 receptive fields ! Image from: X. Pitkow cone separation: 0.5 arcmin Downstream areas require knowledge of trajectory to interpret spikes

7. Joint decoding of image and position Bayesian: vs. N x 2probabilities Discrimination task: X. Pitkow et al, Plos Biology (2007) # positions

8. Joint decoding of image and position Bayesian: vs. N x 2probabilities Discrimination task: X. Pitkow et al, Plos Biology (2007) # positions 30 x 30 binary pixels Unconstrained image N x 2900probabilities

9. Joint decoding of image and position Bayesian: vs. N x 2probabilities Discrimination task: X. Pitkow et al, Plos Biology (2007) # positions 30 x 30 binary pixels Unconstrained image N x 2900probabilities Can the brain apply a Bayesian approach to this problem?

10. Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?

11. Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?

12. Decoding strategy Factorized representation: Discards information about correlations

13. Decoding strategy Factorized representation: Discards information about correlations Update dynamics: minimize DKL evidence, diffusion Exact if trajectory is known.

14. Decoding strategy Factorized representation: Discards information about correlations Update dynamics: minimize DKL evidence, diffusion Exact if trajectory is known. Retinal encoding model: evidence - Poisson spiking (rate λ1 for on pixels, λ0 for off) diffusion - Random walk (diffusion coefficient D)

15. Decoding strategy Factorized representation: Discards information about correlations Neural Implementation - Two populations: where , what For 30 x 30 pixels: N × 2900 → N + 900quantities.

16. Update rules Update of what neurons: Ganglion cells multiplicative gating What nonlinearity Where

17. Update rules Update of what neurons: Ganglion cells multiplicative gating What nonlinearity Where Update of where neurons: Ganglion cells multiplicative gating Where + diffusion What

18. Demo m x m binary pixels image retina 2d diffusion (D) Poisson spikes: 100 Hz (on), 10 Hz (off) Decoder

19. Demo

20. Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?

21. Performance Performance degrades with larger D (and smaller λ) accuracy Convergence time [s] D D

22. Performance Faster and more accurate for larger images m = 5, 10, 30, 50, 100 accuracy Convergence time [s] D D

23. Demo

24. Performance Faster and more accurate for larger images m = 5, 10, 30, 50, 100 accuracy Convergence time [s] D D

25. Performance Faster and more accurate for larger images m = 5, 10, 30, 50, 100 accuracy Convergence time [s] D D

26. Performance Faster and more accurate for larger images m = 5, 10, 30, 50, 100 accuracy Convergence time [s] D D

27. Performance scales with linear image size m accuracy Convergence time [s] D/m D/m m x m pixels

28. Performance scales with linear image size m D* accuracy Convergence time [s] D/m D/m m x m pixels Analytical scaling:

29. Performance Performance improves with image size. Success for images 10 x 10 or larger Prediction for psychophysics: Degradation in high acuity tasks when visual scene contains little background detail.

30. Temporal response of Ganglion cells Common view: fixational motion important to activate cells, due to biphasic response f(t) 50 ms t Temporal response makes decoding much more difficult. Non-Markovian: Need history

31. Temporal response of Ganglion cells Approach: Choose decoder that is Bayes optimal if the trajectory is known. Ganglion accuracy Convergence time [s] Where What D D history dependent decoder / naive decoder “filtered trajectory”

32. Temporal response of Ganglion cells Is fixational motion beneficial? Known trajectory , perfect inhibitory balance Convergence time [s] D Optimal D - order of magnitude smaller than biological value

33. Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?

34. Network architecture Each ganglion cell innervates multiple what & where cells (spread: ~10 arcmin) Reciprocal, multiplicative gating Ganglion What Where

35. Activity: What neurons Slow dynamics, evidence accumulation Where neurons Fewer. Highly dynamic activity Tonic, sparse in retinal stabilization conditions.

36. Activity: What neurons Slow dynamics, evidence accumulation Where neurons Fewer. Highly dynamic activity Tonic, sparse in retinal stabilization conditions. Where in the brain? Monocular If so, suggests LGN or V1 LGN? Modulatory inputs to relay cells (gating?) V1? Lateral connectivity in where network, Increase in number of neurons.

37. Summary Strategy for stabilization of foveal vision Factorized Bayesian approach to multi-dimensional inference

38. Summary Strategy for stabilization of foveal vision Factorized Bayesian approach to multi-dimensional inference Explicit representation of stabilized image “What” and “where” populations

39. Summary Strategy for stabilization of foveal vision Factorized Bayesian approach to multi-dimensional inference Explicit representation of stabilized image “What” and “where” populations Good performance at 1 arcmin resolution Problem is easier for large images, for coarser reconstruction

40. Summary Strategy for stabilization of foveal vision Factorized Bayesian approach to multi-dimensional inference Explicit representation of stabilized image “What” and “where” populations Good performance at 1 arcmin resolution Problem is easier for large images, for coarser reconstruction Network architecture: Many-to-one inputs from retina, multiplicative gating (what/where)

41. Acknowledgments Uri Rokni Haim Sompolinsky Markus Meister Special thanks - the Swartz foundation