Alan L. Liu, Harlan Hile, Gaetano Borriello Computer Science & Engineering

1. Informing the Design of an Automated Wayfinding System for Individuals with Cognitive Impairments Alan L. Liu, Harlan Hile, Gaetano Borriello Computer Science & Engineering University of Washington, USA Pat A. Brown, Mark Harniss, Kurt Johnson Rehabilitation Medicine University of Washington, USA Henry Kautz Computer Science University of Rochester, USA

2. Wayfinding Concerns Limit Independence of Persons with Cognitive Disabilities Learning new routes training from job coaches / caregivers burden on caregivers and Access transit Adapting to changes in routes missing a bus encountering an unexpected barrier Stress of being lost lowers meta-cognition can become overwhelmed and “stuck”

3. Convergence of Technologies Location-sensing technologies Geographic information Mobile computing Online photo collections User modeling Intelligent wayfinding system

4. Our Earlier Results on Wayfinding Wide variation in user preference and expectation of directions modality (image, audio, text) complexity (e.g., direction length) visual distinctiveness timing Direction ambiguity problematic nonstandard turns, especially outdoors Attention demand problematic being distracted or becoming unaware of surroundings ` Liu et al. ASSETS 2006, PervasiveHealth 2009

5. Needs for More Intelligent Wayfinding Assistance Map and/or turn representation not suitable for all Using good landmarks can beneficially augment directions Current systems employ static navigation strategies Ignore health conditions, error behavior, desired detail, safety concerns, place familiarity, and other individual differences Need to support customization and adaptation

6. Contributions • Automatic landmark selection system • Markov-Decision Process based system for optimizing routes and cues • User study to establish further customization / adaptation needs

7. Landmark selection system Leverage collections of geo-tagged photos Select image of landmark based on proximity to user location and near user orientation Heuristic: landmarks useful for wayfinding tend to be more popular • Further enhancements and alternative views are ongoing

8. Modeling Wayfinding with Markov Decision Processes MDPs have been used in robot navigation planning MDP defined by states, actions, transitions, costs state: user position and history of recent events action: direction cue transition: next states and their likelihood given action cost: notion of effort or preference for route A solution to a MDP is a policy that minimizes expected costs The MDP framework can support adaptation using reinforcement learning

9. Modeling Wayfinding using Markov Decision Processes Observation Next state transitions State State (location, orientation, recent history) SystemAction (direction cue) Transition probabilities State Pr(St+1|St,A) State Cost / Reward

10. Modeling customization Alter costs based on individual preferences Preference of direction types Route concerns (distance, accessibility, complexity) Alter transition probabilities based on user ability Does user move in correct direction in response to cue? Study: Does system customization affect user experience? Model 1: Naïve model: Use any direction that guides user along shortest path Model 2: Landmark cues more costly (require more cognitive resources) than turn-based directions, but can take user farther with each cue Policy will use landmarks for long segments, turns for short segments

11. Location Wizard-of-Oz interface

12. Wayfinding application user interface Image, audio (TTS), text, vibration output Repeat audio and Help input Help button requests alternative direction (and possibly route)

13. Participants 7 participants (6 male) with cognitive impairments Age ranging from 21-49 (mean 34) Due to limited size of study, focus on notable user reaction to wayfinding experience

14. Results Participants split on usefulness of landmark directions Participants 1, 2, 3*, 5 expressed affinity for landmarks Participants 4, 6, 7 preferred turn-based directions Difference in location between photo and participant problematic P1: When you have an actual picture of what's in front of the person, that's excellent. P2: The towers [of the Campanile] were behind trees, but I trusted myself to go forward.

15. Results cont'd Some participants indifferent to unfamiliar landmark names, others distracted by them Suggestions included only referring to landmark name when it was destination, and in other important situations Compound directions, meant to give user knowledge of upcoming turns, caused confusion Participants often thought follow-up turn direction was separate Participant 6 sometimes reversed the directions

16. Results cont'd Attentiveness Most participants stated that they paid attention to surroundings, while others commented that familiarity would change their interaction mode I was thinking for a while, maybe I should stick this in my pocket and react when the thing goes off. I didn't do it because [I thought], “Well, by the time I take it out, will I miss the instruction?” but obviously not, the instruction stayed on which was good, so I should have.

17. Current steps Learning user-specific cost function Use of help button Time required to understand cue Experience sampling Learning transition probabilities Predict success or failure of cue based on cue type and features of the state Better landmark-based images.

18. Learning Transition Probabilities by Linear Regression Difficulty incidence rate Model's predicted difficulty

19. Better Landmark-Based Images

20. Conclusion Wayfinding support for individuals with cognitive disabilities should be customized – no one strategy is optimal for all users The Markov Decision Process framework provides a way to generate custom strategies that take into account user's needs (abilities) and preferences Effectiveness of a strategy is inseparable from details of presentation Provided initial results on learning custom models (success probabilities)

21. Thank You Alan L. Liu, Harlan Hile, Gaetano Borriello Computer Science & Engineering University of Washington, USA Pat A. Brown, Mark Harniss, Kurt Johnson Rehabilitation Medicine University of Washington, USA Henry Kautz Computer Science University of Rochester, USA This material is based upon work supported by the National Institute on Disability and Rehabilitation Research (NIDRR) Grant #H133A031739, Microsoft Research Intelligent Systems for Assisted Cognition Award, and Nokia Research.