Overview of Life Logging

1. Overview of Life Logging September 4, 2008 Sung-Bae Cho

2. Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary

3. Advances of Digital Devices • Right now, it is affordable to buy 40 GB (in 2003) • In 3 years 1TB/year is affordable! • It is hard to fill a terabyte/year, but you can: • Look at 9,800 pictures a day (300 KB JPEGs) • Read 2,900 documents a day (1MB files) • Listening to audio or view compressed video 24 hours/day (it takes more than 256 kb/s to fill a TB in a year) • Watch 1.5 Mb/s video 4 hours each day. Source: Microsoft 2003

4. Properties of Mobile Devices Mobility Multi-function Personal Device Large Memory Digital Convergence Advances of Mobile Devices Collect Various Log Data of User’s Everyday Life

5. Everyday Life with Mobile Devices I’m always with you!

6. Available Information from Mobile Devices User Data User Created Contents Scheduler Personal Profile Video Photo Details on daily events Demographical information What is the activity? Preference Who is in the media? Human Relationships Where is the place? How many people in the media? E-mail Address Book Sensor Log Usage Log Bluetooth Bio-sensor Audio Call Log Receiver/Caller Application Usage Time Who is nearby? Noisy level? Duration Sleep? Activity Level? SMS Log Vision Sensor Mp3 Player GPS Contents Time Receiver/Sender Contexts of environments Location

7. Life Logging with Personal Digital Devices Personal Digital Device Context Data Always Bringing Contents Creation Life (Experience) Digital Information Life Record Human Memorize Experience

8. Necessity of Context-aware Computing Episodic Management of Personal Information Summary of Everyday Life Personal Information Retrieval based on Semantic Structure Context-aware Computing Integration of Various Type of Information Data Mining from Personal Information User Modeling from Personal Information

9. Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary

10. Definition of Context-aware Computing • Context : Dey & Abowd (1999) • Any information that can be used to characterize the situation of an entity, where an entity can be a person, place, physical or computational object • Context-aware computing • The use of context to provide task-relevant information and/or services to a user, wherever they may be • Three important context-aware behaviors • The presentation of information and services to a user • The automatic execution of a service • The tagging of context to information for later retrieval • Importance of context • The context of user is changed frequently and drastically in ubiquitous and mobile environment (Pascoe, et al., 1998)

11. Trends on Context Awareness • TEA: EU project for enabling context awareness • 1998 ~ 2000, with TeCo, Starlab, Omega and Nokia • scenarios, technologies, market research and demonstrators • DrWhatsOn concept project • concept and user scenarios for context aware office PDA device • main focus in usability and user interface design • Earlier the focus was in sensor-based context recognition • recognize and utilize information from user's activity and environment properties • Now more directions and possibilities exist • Context-aware computing • Context-aware communication • Context-aware services, connectivity, information retrieval, affective computing, etc. TEA TEA DrWhatsOn

12. Why Context Awareness is Difficult? • Context recognition is never 100% reliable • Contexts are vague, dynamic, overlapping, and ill-defined • Adaptive user interfaces are scary! • May need an adaptive, learning SW • Sensors & algorithms may need constant recalibration • They may also be too CPU intensive • Application development frameworks & support are missing • Security, privacy concerns a lot • Connectivity to environment • Limited I/O through wireless links • What short range connectivity technology should be used? • Productization • Where’s the business? • What everyday problems it really solves? • Where do we get sensors for sensor-based context recognition?

13. Some Technical Challenges • Understanding structure and behavior of context information • context are fuzzy, overlapping, and changing in time • Lack of sensing methods for context information from various sources • Lack of methods for fusing context information • Lack of format of context information • Sensor-based context recognition • hard to obtain reliable data • signal processing and recognition consume memory, energy and processing time • results may be ambiguous • Connectivity to environment and other devices • Are profiles available; how about location information? • Any privacy & security risks?

14. Why do we Need Context Awareness? Maybe avoid disturbing people at wrong times Better social acceptance of technology Increased user satisfaction Right information, right time, right place New services (Location-based services are the obvious ones) Receding to background (Calm computing)

15. Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary

16. MIT Media LAB (1) • Area: Visual Contextual Awareness in Wearable Computing (1998) • Sensor: Vision • Probabilistic object recognition • Probabilistic dependence analysisbased on neighborvector & object recognition • O: object, M: measurement • Task recognition with HMM

17. MIT Media LAB (2) • Activity recognition based on accelerometer (2004) • 20 activities • Sensor: Accelerometer on diverse points of body • Geometric analysis • Mean, energy, frequency-domain entropy, and correlation • Classification method • Decision Tree C4.5, IBL, Naive Bayes

18. MIT Media LAB (3) the number of subjects collected under laboratory (L) or naturalistic (N) settings

19. MIT Media Lab (4) • Aggregate confusion matrix for C4.5 classifier • leave-one-subject-out validation for 20 subjects

20. eWatch Sensor Platform • CMU Computer Science Lab in 2005 • Defense Advanced Research Projects Agency (DARPA) • Activity recognition, improving power consumption, location recognition • Hardware • LCD, LED, vibration motor, speaker,Bluetooth for wireless communication • Li-Ion battery with a capacity of 700mAh • Sensors • a two-axis accelerometer (ADXL202; +/- 2g) • Microphone, light & temperature sensors • Method • multi-class SVMs • HMM based selective sampling

21. Classifier • multi-class SVMs with Gaussian Radial Basis Function kernels • frequency spectrum-based classification • time-domain-based classification with SVM • means, variances, square root of the uncentered second moment, the median absolute differences

22. University of Alberta • National ICT Australia Project: University of Alberta, Canada • Human activity & gesture recognition • Sensor • active, magnetic field, acoustic, laser, camera sensor • Method • Coupled hidden Markov model (CHMM) • Extended HMM model for combining and utilizing concurrent information stream more effectively (By M. Brand et al., 1997) • M. Brand, N. Oliver, and A. Pentland, “Coupled hidden Markov models for complex action recognition,” in IEEE Intl. Conf. Comp. Vis. Pat. Rec., pp. 994-999, 1997.

23. University of Bologna • Micrel Lab: University of Bologna, Italy(2004) • Research • Construction of Ubiquitous environments • Sensor-based gesture recognition • Sensor • Wireless MOCA (Motion capture with integrated accelerometers) sensor • Accelerometer,gyroscope • Small size, small power cosume • Wireless operation • Sticking on diverse points of human body • Method • Hidden Markov Model

24. Activity Recognition Summary • Mostly focused on static classification for pose recognition (relatively easy)

25. Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary

26. Context-aware Mobile Device Trends • Increasing demands for the multi-functional and high-performance phone • Prospects • Increasing demands for Smart Phone • Decreasing basic functional phone • Enlarging role as a computing equipment • Internet surfing • External storage • Increasing investment for adding value of phone • Digitalconvergencewith other functionality • Mp3 player • Personal Media Player • Camera & Camcoder

27. Context-aware Phone on Media The official newsletter of the institute for Complex Engineered Systems (CMU) Sep/Oct 2003 New Scientist, Nov 2004 MIT Sloan Management Review Fall 2004

28. SenSay (Sensing and Say) • Carnegie Mellon University (2003) • A context-aware mobile phone • Adapting to dynamically changing environmental and physiological states • Manipulating ringer volume, vibration, and phone alerts • Providing remote callers with the ability to communicate the urgency of their calls • making call suggestions to users when they are idle • Providing the caller with feedback on the current status of the Sensay user • Sensors • accelerometers, light, and microphones • mounted at various points on the body

29. SenSay (2) • Context recognition by thresholding

30. SenSay (3) • Context classification based on self-organizing map

31. MIT Reality Mining Group • Utilizing Context application from the University of Helsinki (Raento et al., 2005) • Capturing mobile phone usage patterns • from one hundred people (MIT students) • for an extended period of time • Providing • insight into both the users • the ease of use of the device itself • Method: Bayesian inference

32. Light Sensor Proximity Sensor Nokia • Nokia 7650 (2002) • Backlight time adjustment based on proximity & light sensor • Speaker-phone mode change by proximity sensor • Cancellation of speaker-phone mode near ear • Nokia 6230, 6820, 7200 • Presence-enhanced chat service • Presenting and sharing the user’s status • SMS messaging based on other’s status

33. Microsoft Research (1)

34. Microsoft Research (2) • Alarm method for incoming call • Quiet ringing • Volume down with hand touching • Acknowledging and ignoring calls • Calling response: Tilting to body of phone on the pocket • Notification stop: Holding phone with hand without movement on pocket • Target device for notifications • Device selection for alarm: Selecting the recent device if there are several devices of user • Vibration notification • Vibration mode change: Holding phone for a long time

35. Tilt Sensor & User Behavior • Upper:forward/back tilt • Below: left-right tilt

36. VTT Electronics (Finland) • Supported by Nokia • VTT Electronics – Advanced Interaction Systems – Context Awareness • Fuzzy/Bayesian Approach Backlight level & font size adjustment Summary of Context Recognition Procedure (From VTT Publications 511)

37. VTT Technical Research Center of Finland • Context representation Fuzzy logic • Context reasoning  Naive Bayes, Markov Chain • Service  Fuzzy Control Context Ontology Fuzzification

38. VTT Technical Research Center of Finland (2) • Sensor based contexts • Bottom: high-level contexts

39. VTT Technical Research Center of Finland (3) • Audio basedcontext • Bottom: high-level context

40. VTT Technical Research Center of Finland (4) • Naive Bayes based classification

41. TEA Project • High-levelcontext recognition • Method: Rule & SOM http://www.teco.edu/tea

42. Agenda • Life logging • Context-aware computing • Sensory data for activity recognition • Life logging with mobile devices • Summary

43. Summary • Key components for context-aware applications (life logging) • Sensor technology to acquire various contextual information • Intelligence technology to model complex contexts • Agent technology to provide seamless services • Future requirements • Context modeling and cognitive technologies to provide users with more advanced services

44. DrWhatsOn (Laerhoven, 2003) • Concept and user scenarios for context aware office PDA device • Main focus in usability and user interface design • Information fusion: Sensor data, device status, personal preference, schedule • Peripheral attention: supporting appropriate information at appropriate time under prepared conditions • Scenario for a context sensitive phone of Nokia • A day of a Finland student Dude

45. DrWhatsOn Scenario (1) • [Nokia’s DrWhatsOn Concept Video, Urpo Tuomela, Nokia Research Laboratories] • Red mark: Abstracted information from sensor

46. DrWhatsOn Scenario (2)

47. DrWhatsOn Scenario (3)

48. DrWhatsOn Scenario (4)

49. Homework #1: Due 9/9 • Survey the state-of-the-art research on the life logging at KIST, Nokia, Microsoft and MIT (10 min presentation per each institute) • KIST • Nokia • Microsoft • MIT