Mercator

1. Mercator Mapping GUIs to Auditory Interfaces

2. Goals • Provide transparent access to X Windows applications for computer users who are blind or severely visually impaired. • Build a framework to monitor, model, and translate graphical interfaces of X Window applications without modifying the application. • Develop a methodology for translating graphical interfaces into nonvisual interfaces.

3. Mercator Design Considerations • Modality for nonvisual interface -- auditory or tactile. • Substantial research in auditory interfaces (Gaver, Bly, Blattner et al) • Users’ ability to monitor multiple auditory signals - the “cocktail party effect” (Cherry) • Active versus passive interaction • Low cost, standard audio devices • Users’ possible lack of tactile sensitivity due to diabetes

4. Modeling Visual Interfaces

5. Auditory Widgets • Correct level of abstraction • Convey objects … and attributes

6. Navigation • Support two activities • Allow users to “scan” interface • Allow users to operate interface • Mouse navigation is unsuitable • Map interface structure into hierarchical tree structure • Based on widget hierarchy • Users walk tree structure to navigate • Works with existing keyboard short-cuts

9. Mercator Architecture • Three Main Goals: • Capture -- high-level semantically meaningful information from X applications. • Store -- a semantic, off-screen model of the graphical interface. • Translate -- the stored information to present a consistent alternative interface.

10. Spectrum of solutions • Modify the applications • Modify the toolkit • Interpose between application and window server

11. First Architecture

12. Second Architecture

13. How this worked • Xt hooks, RAP, Xlib safety net • Control flow • Interface modeling • Interpreted interfaces • Simulating input

14. How to do this now...