SurroundSense : Mobile Phone Localization via Ambience Fingerprinting

1. SurroundSense: Mobile Phone Localization via Ambience Fingerprinting Written by Martin Azizyan, IonutConstandache, & RomitChoudhury Presented by Craig McIlwee

2. Motivation • Provide logical localization • Using GPS only isn’t good enough • Doesn’t work well indoors • Doesn’t account for dividing walls • Dedicated hardware is not scalable

3. Approach • Create an ambience fingerprint using sound, light, color, and user movement • Noise signatures specific to type of location/store • Chain stores have color themes • User movement indicative of store type

4. Architecture/Algorithm

5. Architecture/Algorithm • Data is recorded on the phone, preprocessed, and sent to a server • Filter module • Subsets the candidates • Wifi, movement, sound • Match module • Selects the best candidate • Color/sound, Wifi

6. Architecture/Algorithm • No single module needs to be perfect • If each module is ‘good enough’ then all modules combined are sufficient • Being simple reasonably accurate instead of sophisticated and perfect reduces resources required for processing

7. Sound Module • Filter • Sound varies over time • Fingerprints captured from various times of day • Similarity of fingerprints is used to create a threshold for a potential match • Match if within the threshold, discard otherwise • Threshold is generous • More false positives is better than false negatives

8. Motion Module • Filter • Variations in user behavior • Record 4 samples/second, use moving average over last 10 samples • Minor variations suppressed

9. Motion Module • User movement is classified as stationary or mobile • 3 profiles defined • Long stationary – restaurant • Frequent movement with longer stationary – browsing • Frequent movement with shorter stationary – shopping • Some logical locations fit multiple profiles

10. Motion Module

11. Color/Light Module • Match • Images captured from camera while facing downward • Floor themes are consistent • Other orientations introduce noise • Common orientation when checking email, text messages, etc

12. Color/Light Module • Analyze patterns in the image • First attempt was to convert pixels to RGB values • Failed due to shadow and reflection influences • Second attempt was to convert to HSL values • Isolates light on its own axis

13. Color/Light Module

14. Color/Light Module • Same/similar colors result in clusters when graphed • Dominant colors generate larger clusters • Similarity calculated as distance between cluster centroids and size of the clusters • Most similar candidate is the match

15. Wifi Module • Normally a filter, match if camera is not available • Capture MAC address of available access points every 5 seconds • Compare occurrence ratio of currently available access points to known access points

16. Known Issues • Sound varies over time • Split day into 2 hour windows, capture fingerprints during each window • No mention of day of week, time of year • Camera in pocket • All testing done with phone in hand • Expected rise in wearable devices • Mimicking user behavior • Initial data showed artificial behavior • Subsequent attempts shadowed real customers

17. Known Issues • Resource (energy) intensive • Accelerometer fingerprint takes time to capture • Non-business locations may not exhibit enough diversity • Offices, airports, libraries

18. Evaluation • Recorded fingerprints of 51 locations • “War-sensed” by students • 2 different groups during different times of day • Group A’s fingerprints used as database while Group B was at the location collecting their own fingerprints • Accuracy analysis was done on various combinations of sensors types • All sensor types combined yielded 87% accuracy