See.ColOr: Experiments in Enhancing Perception for the Blind

1. “It's always incumbent on a researcher to make the effort to reach out to the community to engage with the population they're serving.”

2. Experiments with the See ColOr Interface Guido Bologna CVML, University of Geneva, Switzerland

3. Outline • Introduction • Local Perception Module • Experiments in the past • New experiments • Conclusion

4. Introduction • Blind people 40 millions in 2020 (90% after the age of 1 year) 4 millions in OCDE countries • Long term goal : sensing the environment with sounds Action is key to perception • A number of pixels are directional sound sources

5. Local Perception Module

6. See ColOr Seeing Colours with an Orchestra head-centric colour encoding 2D sound spatialization distance stereovision video piano saxophone 25 points headphones

8. Colour encoding • Colour system: Hue-Saturation-Luminance • Hue: musical instrument • Saturation: 4 instrument notes • Luminance: 2 instruments with different notes

9. Oboe Viola Saxophone Violin pizzicato Piano Flute Trumpet Hue

10. Saturation and luminance • 4 levels of saturation • 8 levels of luminance Low High Double bass Human voice

11. Prototypes • Stereoscopic or Kinect camera • Sonification of a row containing 25 points • CIPIC Kemar HRTFs (2D spatialization) • HSL colour coding • Two depth coding schemes • Lower peripheral resolution [15 12 9 7 5 3 3 2 2 1 1 1 1 1 1 1 2 2 3 3 5 7 9 12 15]

12. Depth coding (first mode) For all sonified pixels (25) : • 90 ms for undetermined depth (D); • 160 ms for 0 <D<1 • 207 ms for 1<D<2 • 254 ms for 2<D<3 • 300 ms for D > 3

13. Depth coding (second mode) • The strategy here is to sonify the dominant colour of the row with only 1 sound source (providing average depth) • The centroid of the first red area is laterally spatialized [R R R R R B B J B B B R R G G GJ J B B B G G G G] [R R R R R ] • The volume V after 3 meters starts to decrease: F(V) = V * exp(-k*D)

14. Experiments in the Past

16. Left picture : the 6 experiment participants interpreted the major colours as the sky the sea and the sun; clouds were more difficult to infer (2 subjects); instead of ducks, all the subjects found an island (with yellow sand) or a ship. Right picture : 4 participants correctly recognised the tree with the sky and the grass; one subject qualified the tree as a strange dark object and finally the last individual imagined a nuclear explosion !

17. All individuals found one of the red doors in a time range between 4 and 9 minutes.

18. Pairing colored socks

19. Participant Time (mn) Success rate (pairs) P1 12 5 P2 24 5 P3 75 P4 6 3 P5 16 5 Average13.0 (7.3) 4.6 (0.9) Participant Time (mn) Success rate (pairs) P6 5 5 P7 10 5 P8 45 P9 4 5 P10 18 5 Average8.2 (6.0) 5.0 (0.0)

20. Following a red path • In this experiment the goal was to follow a coloured path in an outdoor environment.

21. Participant Path Length (m) Speed Average (m/h) P1 M+C = 88 723 P2 M = 84 710 P3 M+B = 110 485 P4 M+A = 93 656 P5 M = 84 484 P6 M+A+C = 97 600 P7 M+A+C = 97 451 P8 M = 84 869 P9 M = 84 1326 P10 M = 84 1096 Average 90.5(8.7) 740.0(284.6)

22. Find a red tee shirt

23. New Experiments

25. Alerting System Object Recognition Local Perception Text reading Current Prototype with Kinect Global Perception

27. See ColOr – CVML (2013)

28. EXPERIMENT ONE: The Spinning Chair

30. EXPERIMENT THREE: Finding Juan

32. EXPERIMENT TWO: Walking Towards a Wall

33. EXPERIMENT FOUR: Grasping Objects

34. Conclusions

35. Sonification of two very important vision features (colour and depth) made it possible to achieve a number of tasks related to mobility and recognition. • We will pursue to define new challenging experiments requiring the activation of several See ColOr modules. • We hope that new small 3D RGB cameras will be in the market soon (like the new Kinect Capri) !