Neuromorphic Image sensors

1. Neuromorphic Image sensors Eugenio Culurciello Yale University EENG427 Lesson 8

2. Biomimetic Circuits Taking hints from nature: • How does nature solve everyday problems • Can we implement nature’s solutions? … in Silicon?

3. Biomimetic Circuits Human Eye: a wonderful machine • Small and light: 1 inch, 7 grams • Retina: neural sensor network, rods and cones • Optic nerve carries ‘digital’ signals to the brain http://webvision.med.utah.edu/anatomy.html

4. Structure of the Eye

5. The Retina

6. Biomimetic Circuits • Dynamic Range: 10+ orders of magnitude • Bandwidth: 100M sensors, 1M fibers in optic nerve • Specialization: • Cones in color, high resolution - fovea • Rods in the dark / motion http://webvision.med.utah.edu/anatomy.html

7. Digital Cameras and Si Eye Everyone wants silicon eyes! • Small • Light • Acute: now > 1Mpixel • Must work in: • Dim restaurant • Outside BBQ Long life = Low power Almost like a human eye! http://www.panasonic.com.au/product_pdf/EB-X70.pdf

8. Neuromorphic Image Sensors Smart-sensors are those devices in which the sensors and circuits co-exist, and their relationship with each other and with higher-level processing layers goes beyond the meaning of transduction. Smart-sensors are information sensors, not transducers and signal processing elements. Smart sensors are not general purpose devices. Everything in a smart sensor is specificaly designed for the application targeted for.

9. Neuromorphic Image Sensors When compared to a vision processing system consisting of a camera and a digital processor, a vision chip provides many system level advantages. Speed: The processing speed achievable using vision chips exceeds that of the camera-processor combination. A main reason is the information transfer bottleneck between the imager and the processor. In vision chips information between various levels of processing is processed and transferred in parallel. Large dynamic range: Many vision chips use photodetectors and photocircuits which have a large dynamic range over at least 7 decades of light intensity. Many also have local and global adaptation capabilities which further enhances their dynamic range. Conventional cameras are at best able to perform global automatic gain control.

10. Neuromorphic Image Sensors Size: Using single chip implementation of vision processing algorithms, very compact systems can be realized. The only parts of the system that may not be scalable are the mechanical parts (like the optical interface). Power dissipation: Vision chips often use analog circuits which operate in subthreshold region. There is also no energy spent for transferring information from one level of processing to another level. System integration: Vision chips may comprise most modules, such as image acquisition, and low level and high level analog/digital image processing, necessary for designing a vision system. From a system design perspective this is a great advantage over camera-processor option.

11. Neuromorphic Image Sensors Although designing single-chip vision systems is an attractive idea, it faces several limitations: Reliability of processing: Vision chips are designed based on the concept that analog VLSI systems with low precision are sufficient for implementing many low level vision algorithms. The precision in analog VLSI systems is affected by many factors, which are not usually controllable. As a result, if the algorithm does not account for these inaccuracies, the processing reliability may be severely affected. Vision chips also use unconventional analog circuits which may not be well characterized and understood. Resolution: In vision chips each pixel includes a photocircuit which occupies a large proportion of the pixel area. Therefore, vision chips have a low fill-factor and a low resolution. The largest vision chip reported has only 210 230 pixels, for a photocircuit consisting of six transistors only.

12. Neuromorphic Image Sensors Difficulty of the design: Vision chips implement a specific algorithm in a limited silicon area. Therefore, often off-the-shelf circuits cannot be used in the implementation. This involves designing many new analog circuits. Vision chips are always full custom designed, and full custom design is known to be time consuming and error-prone. Programming: None of the vision chips are general purpose. In other words, many vision chips are not programmable to perform different vision tasks. This inflexibility is particularly undesirable during the development of a vision system.

13. Mahowald, Mead's silicon retina Mahowald's silicon retina chip is among the first vision chips which implemented a biological facet of vision on silicon. The computation performed by Mahowald's silicon retina is based on models of computation in distal layers of the vertebrate retina, which include the cones, the horizontal cells, and the bipolar cells. The cones are the light detectors. The horizontal cells average the outputs of the cones spatially and temporally. Bipolar cells detect the difference between the averaged output of the horizontal cells and the input.

14. PASIC sensor from Linköping University The ``Processor ADC and Sensor Integrated Circuit'' (PASIC) as the name suggests consists of a sensor array, A/D converters, and processors. Each column has its own ADC and processor.

15. Andreou and Boahen's silicon retina This silicon retina is an implementation of the outer-plexiform of retinal processing layers. The design has a distinctive feature that separates it from all other silicon retinas. The implementation uses a very compact circuit, which has enabled the realization of a 210 x 230 array of image sensors and processing elements with about 590,000 transistors, which is the largest among all reported vision chips. This silicon retina uses a diffusive smoothing network. The function of this one-dimensional network can be written as dQn/dt is the current supplied by the network to node n, and D is the diffusion constant of the network, which depends on the transistor parameters, and the voltage    .

16. Andreou and Boahen's silicon retina The function of the network can be approximated by the biharmonic equation where g and h are proportional to the the diffusivity of the upper and lower smoothing layers, respectively. More details about the function of the circuit can be found in relevant references. All the 2D chips use a hexagonal network with six neighborhood connection. The largest chip occupies an area of 9.5x9.3mm, in a 1.2um CMOS process with two layers of metal and poly. A cell size of about 40x40um has been achieved for this implementation. Under typical conditions the chip dissipates 50mW.

17. Andreou and Boahen's silicon retina Andreou and Boahen have encapsulated the model of the retina in a neat and small circuit (below). This circuit includes two layers of the diffusive network. The upper layer corresponds to horizontal cells in retina and the lower layer to cones. Horizontal N-channel transistors model chemical synapses.

18. Biomimetic Circuits What would it take to reproduce the human eye in Si? 3D Fabrication Process High Connectivity http://www.nips.cc/Web/Groups/NIPS/NIPS2000/00papers-pub-on-web/KurinoNakagawaLeeNakamuraYamadaParkKoyanagi.pdf

19. Biomimetic Circuits And with a conventional process? NEURONS: Advantage IN SPACE Neurons in the human brain make up to 105 connections with their neighbors CIRCUITS: Advantage IN TIME Integrated circuits handle communication cycles six orders of magnitude smaller than the inter-event interval for a single neuron or cell

20. Conventional Image Sensors • Integrate light on a capacitor for a fixed time • Sample the analog capacitor voltage • Pixels are synchronously scanned

21. Events: digital pulses Time Address-Event Image Sensors • Measure the time to integrate to a fixed voltage • Light triggers a digital event • Integrate (to threshold) and fire Event Driven!

22. Pixel Operation • Photocurrent is integrated on a 0.1pF capacitor. Slew Rate of 0.1V/ms in typical indoor light of 0.1mW/cm2 • Pixel is reset to ‘Vdd_r’ • While integrating light, the voltage on the capacitor will decrease down to the threshold of the inverter

23. Pixel Operation • The switching current of the inverter is fed back by a current mirror to sharpen the transition. The integrating capacitor is disconnected to minimize power consumption during reset. • Reduced power consumption when compared to an inverter • Slew rate gain

24. Pixel Operation

25. Pixel Operation • Equation of the switching point (voltage): • In time domain:

26. Address-Event • Address-Event Representation: asynchronous protocol for communication between large arrays • The AER model trades complexity in wiring of the biological systems for processing speed of integrated circuits

27. Address-Event Architecture

28. Address-Event Architecture

29. 1/Ti t t N/Th SampleImages from Sensor 10k samples 100k samples Inter-Event Image Histogram Image

30. Chip layout E. Culurciello, R. Etienne-Cummings, K. A. Boahen, ``A Biomorphic Digital Image Sensor'‘, IEEE Journal of Solid-State Circuits, Vol. 38, No. 2, February 2003.

31. Sensor Performance