Understanding Human Behavior from Sensor Data

1. Understanding Human Behavior from Sensor Data Henry Kautz University of Washington

2. A Dream of AI • Systems that can understand ordinary human experience • Work in KR, NLP, vision, IUI, planning… • Plan recognition • Behavior recognition • Activity tracking • Goals • Intelligent user interfaces • Step toward true AI

3. Plan Recognition, circa 1985

4. Behavior Recognition, circa 2005

5. Punch Line • Resurgence of work in behavior understanding, fueled by • Advances in probabilistic inference • Graphical models • Scalable inference • KR U Bayes • Ubiquitous sensing devices • RFID, GPS, motes, … • Ground recognition in sensor data • Healthcare applications • Aging boomers – fastest growing demographic

6. This Talk • Activity tracking from RFID tag data • ADL Monitoring • Learning patterns of transportation use from GPS data • Activity Compass • Learning to label activities and places • Life Capture • Kodak Intelligent Systems Research Center • Projects & Plans

7. This Talk • Activity tracking from RFID tag data • ADL Monitoring • Learning patterns of transportation use from GPS data • Activity Compass • Learning to label activities and places • Life Capture • Kodak Intelligent Systems Research Center • Projects & Plans

8. Object-Based Activity Recognition • Activities of daily living involve the manipulation of many physical objects • Kitchen: stove, pans, dishes, … • Bathroom: toothbrush, shampoo, towel, … • Bedroom: linen, dresser, clock, clothing, … • We can recognize activities from a time-sequence of object touches

9. Application • ADL (Activity of Daily Living) monitoring for the disabled and/or elderly • Changes in routine often precursor to illness, accidents • Human monitoring intrusive & inaccurate Image Courtesy Intel Research

10. Sensing Object Manipulation • RFID: Radio-frequency identification tags • Small • No batteries • Durable • Cheap • Easy to tag objects • Near future: use products’ own tags

11. Wearable RFID Readers • 13.56MHz reader, radio, power supply, antenna • 12 inch range, 12-150 hour lifetime • Objects tagged on grasping surfaces

12. Experiment: ADL Form Filling • Tagged real home with 108 tags • 14 subjects each performed 12 of 65 activities (14 ADLs) in arbitrary order • Used glove-based reader • Given trace, recreate activities

13. Legend General solution Point solution Results: Detecting ADLs Inferring ADLs from Interactions with Objects Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz IEEE Pervasive Computing, 4(3), 2004

14. Experiment: Morning Activities • 10 days of data from the morning routine in an experimenter’s home • 61 tagged objects • 11 activities • Often interleaved and interrupted • Many shared objects

15. Baseline: Individual Hidden Markov Models 68% accuracy11.8 errors per episode

16. Baseline: Single Hidden Markov Model 83% accuracy9.4 errors per episode

17. Cause of Errors • Observations were types of objects • Spoon, plate, fork … • Typical errors: confusion between activities • Using one object repeatedly • Using different objects of same type • Critical distinction in many ADL’s • Eating versus setting table • Dressing versus putting away laundry

18. Aggregate Features • HMM with individual object observations fails • No generalization! • Solution: add aggregation features • Number of objects of each type used • Requires history of current activity performance • DBN encoding avoids explosion of HMM

19. Dynamic Bayes Net with Aggregate Features 88% accuracy6.5 errors per episode

20. Improving Robustness • DBN fails if novel objects are used • Solution: smooth parameters over abstraction hierarchy of object types

22. Abstraction Smoothing • Methodology: • Train on 10 days data • Test where one activity substitutes one object • Change in error rate: • Without smoothing: 26% increase • With smoothing: 1% increase

23. Summary • Activities of daily living can be robustly tracked using RFID data • Simple, direct sensors can often replace (or augment) general machine vision • Accurate probabilistic inference requires sequencing, aggregation, and abstraction • Works for essentially all ADLs defined in healthcare literature D. Patterson, H. Kautz, & D. Fox, ISWC 2005 best paper award

24. This Talk • Activity tracking from RFID tag data • ADL Monitoring • Learning patterns of transportation use from GPS data • Activity Compass • Learning to label activities and places • Life Capture • Kodak Intelligent Systems Research Center • Projects & Plans

25. Motivation: Community Access for the Cognitively Disabled

26. The Problem • Using public transit cognitively challenging • Learning bus routes and numbers • Transfers • Recovering from mistakes • Point to point shuttle service impractical • Slow • Expensive • Current GPS units hard to use • Require extensive user input • Error-prone near buildings, inside buses • No help with transfers, timing

27. Solution: Activity Compass • User carries smart cell phone • System infers transportation mode • GPS – position, velocity • Mass transit information • Over time system learns user model • Important places • Common transportation plans • Mismatches = possible mistakes • System provides proactive help

28. Technical Challenge • Given a data stream from a GPS unit... • Infer the user’s mode of transportation, and places where the mode changes • Foot, car, bus, bike, … • Bus stops, parking lots, enter buildings, … • Learn the user’s daily pattern of movement • Predict the user’s future actions • Detect user errors

29. Technical Approach • Map: Directed graph • Location: (Edge, Distance) • Geographic features, e.g. bus stops • Movement: (Mode, Velocity) • Probability of turning at each intersection • Probability of changing mode at each location • Tracking (filtering): Given some prior estimate, • Update position & mode according to motion model • Correct according to next GPS reading

30. Dynamic Bayesian Network I mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k Time k-1

31. Mode & Location Tracking Measurements Projections Bus mode Car mode Foot mode Green Red Blue

32. Learning • Prior knowledge – general constraints on transportation use • Vehicle speed range • Bus stops • Learning – specialize model to particular user • 30 days GPS readings • Unlabeled data • Learn edge transition parameters using expectation-maximization (EM)

33. Predictive Accuracy How to improve predictive power? Probability of correctly predicting the future City Blocks

34. Transportation Routines Home A B Workplace • Goal: intended destination • Workplace, home, friends, restaurants, … • Trip segments: <start, end, mode> • Home to Bus stop A on Foot • Bus stopAto Bus stopB on Bus • Bus stop B to workplaceon Foot

35. Dynamic Bayesian Net II gk-1 gk Goal tk-1 tk Trip segment mk-1 mk Transportation mode xk-1 xk Edge, velocity, position qk-1 qk Data (edge) association zk-1 zk GPS reading Time k Time k-1

36. Predict Goal and Path Predicted goal Predicted path

37. Improvement in Predictive Accuracy

38. Detecting User Errors • Learned model represents typical correct behavior • Model is a poor fit to user errors • We can use this fact to detect errors! • Cognitive Mode • Normal: model functions as before • Error: switch in prior (untrained) parameters for mode and edge transition

39. ck-1 ck gk-1 gk tk-1 tk mk-1 mk xk-1 xk qk-1 qk zk-1 zk Time k Time k-1 Dynamic Bayesian Net III Cognitive mode { normal, error } Goal Trip segment Transportation mode Edge, velocity, position Data (edge) association GPS reading

40. Detecting User Errors

41. Status • Major funding by NIDRR • National Institute of Disability & Rehabilitation Research • Partnership with UW Dept of Rehabilitation Medicine & Center for Technology and Disability Studies • Extension to indoor navigation • Hospitals, nursing homes, assisted care communities • Wi-Fi localization • Multi-modal interface studies • Speech, graphics, text • Adaptive guidance strategies

42. Papers Liu et al, ASSETS 2006 Indoor Wayfinding: Developing a Functional Interface for Individuals with Cognitive Impairments Liao, Fox, & Kautz, AAAI 2004 (Best Paper) Learning and Inferring Transportation Routines Patterson et al, UBICOMP 2004 Opportunity Knocks: a System to Provide Cognitive Assistance with Transportation Services

43. This Talk • Activity tracking from RFID tag data • ADL Monitoring • Learning patterns of transportation use from GPS data • Activity Compass • Learning to label activities and places • Life Capture • Kodak Intelligent Systems Research Center • Projects & Plans

44. Task • Learn to label a person’s • Daily activities • working, visiting friends, traveling, … • Significant places • work place, friend’s house, usual bus stop, … • Given • Training set of labeled examples • Wearable sensor data stream • GPS, acceleration, ambient noise level, …

45. FRIEND’S HOME RESTAURANT OFFICE HOME PARKING LOT STORE Learning to Label Places & Activities • Learned general rules for labeling “meaningful places” based on time of day, type of neighborhood, speed of movement, places visited before and after, etc.

46. Relational Markov Network • First-order version of conditional random field • Features defined by feature templates • All instances of a template have same weight • Examples: • Time of day an activity occurs • Place an activity occurs • Number of places labeled “Home” • Distance between adjacent user locations • Distance between GPS reading & nearest street

47. RMN Model s1 Global soft constraints p1 p2 Significant places a1 a2 a3 a4 a5 Activities g1 g2 g3 g4 g5 g6 g7 g8 g9 {GPS, location} time 

48. Significant Places • Previous work decoupled identifying significant places from rest of inference • Simple temporal threshold • Misses places with brief activities • RMN model integrates • Identifying significant place • Labeling significant places • Labeling activities

49. Results: Finding Significant Places

50. Results: Efficiency