“Mapping while walking”

1. Pose Invariant Activity Classification for Multi-floor Indoor Localization Saehoon Yi1, Piotr Mirowski2,3, Tin Kam Ho2,4, Vladimir Pavlovic1 1Computer Science Department, Rutgers University 2Statistics and Learning Research Department, Bell Labs, Alcatel-Lucent3Now at Microsoft Bing 4Now at IBM Watson shyi@cs.rutgers.edu “Mapping while walking”

2. Outline • Related work • Indoor localization • Pedestrian Dead Reckoning • GraphSLAM • Motivation • “Mapping while walking” • Methods • Pose-invariant sensor features • SVM classification of actions • HMM temporal smoothing • Results

3. Indoor Localization • Indoor localization • Various practical use • Radio Frequency (WiFi, 4G cell) maps“where am I in this building?” • Network deployment optimization:“where should we place that 4G cell in the building?” • No GPS • Smartphone sensors • Accelerometer • Gyroscope • Magnetometer • Barometer • WiFi, cell

4. Pedestrian Dead Reckoning • Step detection on vertical acceleration • [Steinhoff et al., Pervasive Computing and Communications 2010] • 3-axis orientation [Madgwick et al., Rehabilitation Robotics 2011] • Trajectory updates • step length d, offset angle β • Subject to drift due to orientation error • Sensor measurement noise • Magnetic field perturbations angle offsetbeta β xt-1 error atposition reset Yaw of the phonew.r.t. horizontalX axis xt-2 directionof walk xt+1 xt xt-3 xt-4 longitude or human/horizontal X(“towards East”) xt-5

5. GraphSLAM • Modify trajectory to minimize estimation error • [Grisetti et al., Transportation Systems Magazine 2010] • Challenges • Requires landmark detection • How do we know that two different observations are actually taken at the same location? • Hand-placed landmarks, e.g.,: QR code or NFC tag • Manually installed and maintained xj xj* zij xi

6. Motivation • Detect and provide natural landmark for GraphSLAM • Stairs and elevators are accurately detected • They are non moving, distinct structures, which is ideal for landmarks • Classify human activities using a smartphone in the pocket • Pose invariant features extracted from smartphone sensors • Jointly infer activity and floor information • Focus on activities that incur floor change • “Mapping while walking” • Facilitate radio-frequency map building for network engineers

7. Methods • SVM: Classify activity at each time point • HMM:Smoothing SVM activity classification and jointly infering floor • Activities • walking • stair up • stair down • stand still • elevator up • elevator down

8. Pose invariant features for IMU sensors • Pose invariant features from A • [Kobayashi et al., ICASSP, 2011] • Invariant to rotation R

9. Statistical features for barometer • Rotation does not affect air pressure • Fluctuate over time • Depend on weatherand temperature • Detects ascending/descending air pressure

10. SVM classification • Features are extracted from sliding window of sensor observations • Linear SVM for 6 activity classes • Fast and storage efficient • Linear classification • Able to implement real time classification in Android OS • Provide activity probability • Platt’s scaling algorithm • Required to obtain HMM observation probability

11. HMM smoothing • 6 activities for each floor • State transit for strong evidence • Smooth sporadic brief misclassification • Augment inference of activity with floor from Viterbi algorithm • Observation probability • from SVM confidence level(Platt’s scaling) • from mixture of Gaussian

12. HMM smoothing • Transition probability • Manually design transition probabilities • Higher probability of transition to the same state • Floor changes only for stair and elevator

13. Experiment set up • Input data • Sensor data recorded at 50Hz • Feature extraction • Sliding window • IMU sensor features • Length: 64 frames • Step size: 35 frames • Barometer features • Length: 192 frames • Train data: 10271 seconds of each class repeatedly performed • Test data: 6160 seconds of 12 natural sequence

14. Activity classification result • For SVM, • Walking is confused to taking stairs • Standing still is confused to taking elevators • Leg dynamics are similar • Air pressure does not change over short period of time • Each sliding window is considered independently

15. Activity classification result • HMM removes sporadic misclassification between walking and taking stairs • Rectification: rectifies stairs to walking when it does not incur floor change

16. Landmarks match • Types of landmark • Stair • Elevator • GraphSLAM requires matching of the same landmark along the trajectory • Training phase • Obtain information from reference landmarks • WiFi access point visibility • Testing phase • Landmarks matching • Get current WiFi AP visibility • Calculate distance to reference landmarks • Take the closest corresponding landmark

17. Initial PDR trajectory • Initial trajectory obtained from PDR • Rotation angle underestimated for every turn • Need to be modified using GraphSLAM

18. Multi-floor GraphSLAMresult

19. Conclusion • Rotation invariant features were able to capture different dynamics of motion activities • Our approach improves classification accuracy and jointly infers activity and corresponding floor information • GraphSLAM successfully modifies multi-floor trajectory using natural landmarks detected by our framework.

20. Q & A Thank you