Università degli Studi di Milano

1. Università degli Studi di Milano A Multimodal Interactive System to create and explore graph structures Cristian Bernareggi, Loredana Parasiliti Provenza (joint work with C. Comaschi, A. Marcante, P. Mussio, S.Vanzi)

2. N H H H The problem • Scientific reasoning widely employs spatial representations of concepts having an underlying graph structure. Different contexts: • Automata theory • Chemistry • Visually impaired or blind people meet with difficulties to perceive and reason on such representations.

3. Related work • Tactile & audio-tactile systems for exploring 2D diagrams [Way&Barnek 1997], [Vanderheiden 1989] • Haptic and audio systems for exploring line graph (e.g. function diagrams): • to analyse and manipulate bar graphs [McGookin &Brewster 2006] • to compare multiple graphs [McGookin &Brewster 2006] • TeDUB: an audio-haptic system for exploring 2D UML diagrams [Petrie et al. 2002]

4. Our goal • To enable visually impaired or blind people • to perceive and reasonon graph structures • to manipulate them • to communicate their reasoning on graph structure and experiences also with sighted people

5. Our solution • Creation of a virtual environment for producing multimodal experiences of graph structures through a direct manipulation paradigm

6. The MIS: a first experience • To explore the virtual space, shaped as a 3D discretegrid, where the user can experience: • Haptic modalities: the space is an haptic grid (when the user moves from one element to the next one s/he perceives the overcoming of a gentle threshold); • Audio modalities: the virtual space informs the user on her/his position within the grid • Visual modalities: the virtual space is a visual grid (a proxy represents the position of the haptic device in the space)

7. The MIS: a second experience • To create, explore and manipulate graph structures. Visually impaired or blind users are able to • perceive a graph structure through haptic and auditive signals • manage and interact with the graph structure through direct manipulation operations Visual feedback supports communication on graph structures among sighted and not-sightedusers

8. How to interact with the MIS MIS to create and explore graph structures FORCE FEEDBACK

9. The current prototype A user aims at creating a graph structure that represents the ammonia molecular structure NH3

10. System development and evaluation • The current prototype has been developed as a third step in a usability life cycle Evaluations performed with 2 types of blind users: • experts • not-experts in computer science and in the use of haptic device;

11. Evaluation experiments • A first experiment with 2 expert blind users to assess: • the representation of the virtual space to better support the perception and navigation of the space and graph structures • the movement in the virtual space and within the graph structure • A second experiment (*) with 5 not-expert blind users to assess • the movement in the virtual space and graph nodes perception • How easily graph nodes can be found in the grid. • How useful speech messages are. (*) OBSERVED VARIABLES: the number of errors, the execution time for each considered tasks

12. Some evaluation results • Virtual space representation & perception as: • Grid consisting of haptic points and lines better support the user than spheres and cylinders in navigating the 3D virtual space • Graph manipulation and perception by not-experts blind users • Users learned in few seconds (10-15 secs) how to move in the virtual space and place nodes • Node search: nodes inserted by the user found in about (20-25) sec, pre-existing nodes found in a longer time (45-60 sec); • Vibrational effect was rather misleading.

13. Future Work • The system needs further enhancements: • To support more advanced exploration strategies based on guided movements (e.g. forces) rich audio descriptions and visual feedback • further evaluation experiments with both sighted and not-sighted users

14. References • Way, T. P., and Barner, K.E. Automatic Visual to Tactile Translation, Part I: Human Factors, Access Methods and Image Manipulation. IEEE Trans. on Rehabilitation Engineering, 5, 1 (1997). • Vanderheiden, G. C. Nonvisual alternative display techniques for output from graphics-based computers. Journal of Visual Impairment and Blindness 83, 8 (1989), 383-390. • McGookin, D. K., and Brewster, S.A. MultiVis: Improving Access to Visualisations for Visually Impaired People. In extended Proc. CHI 2006, ACM Press (2006). • McGookin, D. K. and Brewster, S.A. 2006. Graph builder: Constructing non-visual visualizations. In ICAD 2006 (London, UK). • McGookin, D.K. and Brewster, S.A. 2006. Contextual Audio in Haptic Graph Browsing. In Proc. ICAD 2006 (London, UK), ICAD, 91-94 • Petrie, H., Schlieder, C., Blenkhorn, P., Evans, G., King, A., O’Neill, A.-M., Ioannidis, G.T., Gallagher, B., Crombie, D., Mager, R., and Alafaci, M. TeDUB: a system for presenting and exploring technical drawings for blind people. In K. Miesenberger, J. Klaus and W. Zagler (Eds.), LNCS 239: Computers Helping People with Special Needs. Heidelberg, Springer Verlag, 2002.