Fabio Pianesi Massimo Zancanaro

1. Fabio PianesiMassimo Zancanaro FBK-irst Alessandro Cappelletti, Bruno Lepri, Nadia Mana

2. Research questions • Recognition of is happening at a given time slice (mainly) from audio-visual signals Fabio Pianesi & Massimo Zancanaro FBK

3. Data sharing requirements • Indipendent modules for • Visual recognition robust for lighting for • Detecting parts of the body • Detecting and recognizing objects • Attentional module to focus cameras, mics and other sensors where “the action is” • Avoids continuous streams of data from the environment • Standards • Annotation of activities • Meta-data descriptions • Type of sensors, their relative position in the environment • Environment description (relative to sensors: riverberation, …) • Standard for data storing Fabio Pianesi & Massimo Zancanaro FBK

4. Gregory D. Abowd Georgia Tech

5. Research questions Bus Monitoring Sensors Room #1 Room #2 Room #n Electrical Outlets and Appliances Machine Learning System Air Ducts Inferred Human Activity Plumbing Fixtures • How to address questions of health and (to a lesser degree) sustainability through instrumentation and augmentation of the home. How to enable others to collect data in real homes. • Chronic care management • Early detection and monitoring of interventions for autism • Video data and sensor data • Sensing for the masses • Infrastructure mediated sensing data from real homes • Energy awareness, location-tracking Gregory D. Abowd, Georgia Tech

6. Data sharing requirements • I want annotated home movies of young child behavior, or at least movies that I can annotate and make available as a shared data set for the vision community to work on. • I want to provide (through commercial efforts) the ability to collect low-level sensor data of home activity so that you can collect data in real homes. Gregory D. Abowd, Georgia Tech

7. Aaron Crandall Washington State University D.J. Cook, M. Schmitter-Edgecombe, Chad Sanders, Brian Thomas

8. Collecting and Disseminating Smart Home Sensor Data in the CASAS Project D.J Cook, M. Schmitter-Edgecombe, Aaron Crandall, Chad Sanders and Brian Thomas cook@eecs.wsu.edu

9. CASAS Testbed Comprehensive Agent-oriented Both office and living spaces Scripted and unscripted data Focused on ADL detection

10. The Space & Sensors Describing the physical space: Implications on resident behavior Issues with changes The sensors: Location Relationships Implications Configurations Versions

11. The Data Fields and Format When collected, the CASAS data is very simple:

12. Annotation & ADLs Correct annotation is still a limiting factor Detail of annotation drives cost of effort and accuracy Proper notation of both correct activity completion and activity errors

13. Final Core Issues Ensuring clean data Annotation accuracy & length Generating sufficiently varied data Properly describing test bed configurations

14. WSU Smart Home Dataset Available Now

15. Thank you Shared Datasets: http://www.ailab.wsu.edu/casas/datasets.html Contact info: Aaron S. Crandall acrandal@wsu.edu Diane J. Cook cook@eecs.wsu.edu

16. Lorcan Coyle Lorcan.coyle@lero.ie Lero – The Irish Software Engineering Research Centre University of Limerick Juan Ye, Susan McKeever, Stephen Knox, Matthew Stabeler, Simon Dobson, and Paddy Nixon University College Dublin

17. Research questions • we are interested in activity recognition • Bayesian networks & lattice theory, Dempster Shafer evidence theory, case-based reasoning • more realistic and/or more crisp datasets for evaluations • we are also gathering our own datasets • based on best principles? - CASL • (also we have some “toy datasets”) • McKeever et al., Pervasive LBR 2008 • Stabeler et al., Pervasive LBR 2008 • Knox et al., RIA 2008 • Ye et al., RIA 2008 • Ye et al., ICPS 2008 • Ye et al., Percom 2009 Lorcan Coyle, Lero@UL

18. Data sharing requirements • there should be a web-based repository like the UCI ML repository • we need a common language for datasets • and parsers to allow interoperability • algorithms should be released! • like Weka or in Weka? • results need to be published beyond the paper! • put results up with the datasets • tag datasets with 3rd party opinions and cite the paper where the results are presented • ultimately we need to make it transparent for reviewers/scientists to understand a “good result” Lorcan Coyle, Lero@UL

19. Fernando De la Torre Jessica Hodgins Javier Montano Sergio Valverde Carnegie Mellon University

20. http://kitchen.cs.cmu.edu/

21. Research questions • How to build good computation models to characterize subtle human motion? • Develop machine learning algorithms for activity recognition and temporal segmentation (supervised/unsupervised) of human motion • Judgments about the quality of motion • How to select or fuse multimodal data for activity recognition? • What should be a good protocol for multimodal data capturing? Fernando, Carnegie Mellon University

22. Data sharing requirements • Shared datasets: • 45 people cooking 5 different recipes (brownies, salad, pizza, sandwich, eggs) • Each recipe is about 22 minutes and 5 synchronized modalities are recorded (audio, video, motion capture, inertial measurement units) • Anomalous situations (falling, fire, mistaken putting soap rather than salt, …) • Camera calibration parameters, time stamps for each modality • Shared labels for object recognition, temporal segmentation and activity recognition • Shared code: • Multimodal data visualization toolbox (Matlab). • Baseline experiments for activity recognition and temporal segmentation. • Aligned Cluster Analysis: Clustering of time series. Fernando, Carnegie Mellon University

23. James Fogarty Assistant Professor Computer Science & Engineering

24. Research questions • Attacking human-computer interaction problems using statistical machine learning • Previously with a significant focus on sensing • Sensor-based human interruptibility models • Privacy-sensitive approach to collecting sensed data in location-based applications • Unobtrusive home activity sensing(collaborations pulling me back into this) • More recently focused on domains where it is actually possible to attack the entire problem • End-user interactive concept learning (with application in Web image search) • Mixed-initiative information extraction(with application to semantifying Wikipedia) James Fogarty, University of Washington

25. Data sharing requirements • Convincingly answering compelling HCI questions typically requires some custom data collection (either formative or summative data) • Those datasets are expensive and difficult to collect • We therefore look for the minimal collection to answer our question • Rendering the collected data largely useless for other questions • Data sharing can have important value, but we also need to examine other approaches to achieving the same intended benefits • Work on different problems (like the Web, where there’s lots of data!) • Improved coordination of collection (work with others to reduce costs) • Improved standardization of collection (agree what’s important to collect) • Improved collection tools (lower barrier to getting it in the first place) • Improved annotation tools (lower barrier to coding it later) James Fogarty, University of Washington

26. Stephen Intille Massachusetts Institute of Technology

27. Research questions • How can just-in-time information presented by context-aware technology in the home and worn on the body help people stay healthy as they age? • How do we make activity detection algorithms that work for non-techies in real life in complex situations using practical and affordable sensor infrastructures? End-user concerns/challenges that have not been adequately addressed…* Practical sensor installation* Maintaining sensors* Fixing mistakes * Adding activities Toothbrushing Stephen Intille (MIT)

28. Data sharing requirements • What shared datasets or tools, if any, would best advance your work (on automatic detection of activity for health systems) ? • Datasets of 10-100 families in their homes doing everyday activities for months with accurate labeling of activity, postures, and audio transcription and synchronized with data from 3-axis accelerometers on each limb, object usage data on as many objects as possible, current flow sensing on electrical devices, and indoor position information on each occupant (1m accuracy). • Datasets of 10-100 people doing everyday activities in natural settings for weeks or months with accurate labeling of type and intensity (energy expenditure) of physical activity while wearing 3-axis accelerometers on each limb. Stephen Intille (MIT)

29. Taketoshi MORI Mechano-Informatics, The University of Tokyo Masamichi Shimosaka,Akinori Fujii,Kana Oshima,Ryo Urushibata,Tomomasa Sato,Hajime Kubo,Hiroshi Noguchi Sensing Room and Its Resident Behavior Mining CHI 2009 Workshop Developing Shared Home Behavior Datasets to Advance HCI and Ubiquitous Computing Research

30. Research questions Sensing Room and Its Resident Behavior Mining • We have constructed several room-type human behavior sensing environments. These used many distributed sensors. The key was location sense. The problems were • A long-term recording is difficult, • Time synchronization is difficult, • Annotating is such a bother! • Based on the collected behavior data, we have been constructing services such as action anticipation, beat-one information display and robotic support. • We introduced for these problems, • Multi-layered network system, • Distributed object software scheme, • RDF/OWL knowledge representations. Taketoshi Mori, the University of Tokyo

31. Data sharing requirements Sensing Room and Its Resident Behavior Mining • Developing algorithms to detect unusual behavioral phenomenon or to foresee stereotyped frequently occurring behaviors for supporting human, it is necessary to obtain human’s position in the space with timestamp. Distributed sensors should supply sufficient information to estimated the human position. It may help if the timestamp is marked both at the sensed time by sensors and the recorded time by the home server. • The datasets with many additional data, such as the resident’s profile, 3D room models, the wall and floor textures, the weather and temperature help to construct an appropriate behavior estimation method. • The datasets should be constructed based on some tagged formats as XML or YAML, and preferably the tags are added following RDF. Taketoshi Mori, the University of Tokyo

32. Tim van Kasteren Intelligent Systems Lab Amsterdam University of Amsterdam Co-author: Ben Kröse

33. Research questions • Which probabilistic model is best for modeling human behavior? • How to deal with unsegmented data? • How to capture long term dependencies? • How to deal with the large number of ways in which activities can be performed? • How can we apply these models on a large scale, without the necessity of training data from each house they are applied? • How to deal with different layout of houses? • How to deal with different behavior of people? Tim van Kasteren (University of Amsterdam)

34. Data sharing requirements • To validate the effectiveness of our models, we need: • Datasets consisting of several days (weeks) of data recorded in a real world setting. • We have mainly used wireless sensor networks, but we are interested in validating our models on other sensing modalities as well. • To validate the application of our models on a large scale, we need: • Datasets from multiple houses. • Ideally consisting of a fixed set of sensors and labeled activities. • We offer: • Several real world datasets consisting of at least two weeks of fully labeled data each. Tim van Kasteren (University of Amsterdam)

35. Sumi Helal University of Florida, Andres Mendes-Vazquez, Diane Cook and Shantonu Hussein www.icta.ufl.edu

36. Research questions • How can we synthesize sensory datasets either from scratch or by “stem-celling” existing actual datasets? • Synthesis is necessary to enable researchers with limited resources but with great ideas and algorithms that need to be thoroughly tested. • Synthesis could also be needed by the owner of an actual dataset, to enable/him/her to go back in time and explore additional concerns/goals not thought of during data collection. • What are the synthesis strategies/algorithms? • How good are synthesized datasets? How can we assess our success or failure in this direction. • What does “Sensory Dataset Description Language” standard has to do with data synthesis? Sumi Helal, University of Florida

37. Data sharing requirements • Simply, access to a database of well documented datasets will advance our research and tool development in sensory data synthesis. • What is of great importance to us is documentation of the “protocol” used to collect the data, not just the data itself. • To be able to utilize other people datasets, and to foster a greater level of interoperability and cross use of data sets, we have been working on defining a standard to propose to the community. We call the standard: “Sensory Dataset Description Language” or SDDL. The SDDL specification proposal can be downloaded from: http://www.icta.ufl.edu/persim/sddl/ • We have utilized 4 datasets in defining this standard proposal. We wish to consider many more datasets in refining this proposal. Your comments AND contributions to SDDL are sought. Sumi Helal, University of Florida

38. Allen Yang with Phil Kuryloski and Ruzena Bajcsy UC Berkeley

39. DexterNet: A Wearable Body Sensor System • Primary Goals • Real-time control & sampling of heterogeneous body sensors • Securedsurveillance in indoors and outdoors • Provides geographical and social data • System Architecture • Body Sensor Layer (BSL) • Personal Network Layer (PNL) • Global Network Layer (GNL) • Prototype Systems • Human action recognition • State-of-the-art security features • Real-time communication between Berkeley and Vanderbilt Hospital tested Reference: BSN Workshop, 2009. Allen Yang, UC Berkeley

40. Wearable Action Recognition Database (WARD), version 1 • Free for noncommercial users • 5 motion sensors, each carries an accelerometer and gyroscope sampled at 30 Hz • 20 test subjects (13 male & 7 female) ages 19-75 • 13 action categories collected in an indoor lab • Data processed in Matlab. Visualization tool is included Allen Yang, UC Berkeley

41. Workshop schedule • 9:00   Overview and goals • 9:15   Introductions by attendees • 10:30  Break • 10:45  Targeted questions and answers • 12:00  State-of-the-art in data collection • 12:30  Lunch • 14:00  Discussion: What's possible? • 14:20  Group exercise • 15:20  Group presentations • 16:00  Break • 16:15  Next steps • 17:30  End of workshop

42. Gregory D. Abowd Georgia Tech

43. Question & Answers #1 • Why do you want family home movies? • Sufficient retrospective research in the autism domain has shown that there is evidence of developmental delay in home movies. This has value for early detection and early intervention. • We have shown that you can encourage the collection of relevant developmental milestone behavior from parents, but not of rich evidence like video. • We are working on filtering techniques to pull out the relevant snippets of social interaction. • Ultimately, I envision a way to upload home movies to a secured service that can then extract relevant portions to share with a pediatrician or other professional for screening purposes. Gregory D. Abowd, Georgia Tech

44. Question & Answers #2 • What is the value of infrastructure mediated sensing to other researchers? • This is a way to gather low-level sensing data from real homes. • There is both commercial and research opportunities here and I think the commercial opportunities in demand-side energy management may be able to drive the ability to provide valuable resources for researchers to leverage. Gregory D. Abowd, Georgia Tech

45. Fabio PianesiMassimo Zancanaro FBK-irst Alessandro Cappelletti, Bruno Lepri, Nadia Mana

46. Question & Answers #1 • How would low-bandwidth sensing (e.g. passive infrared motion detection, object movement sensors, RFID) complement the methods used in the NETCARITY project • Attention mechanism to that activates/deactivates camera/mikes when someone enters a room • Fusion of multiple modalities • Recognition of objects (manipulation) • Information about body (and body segments) activity levels, posture changes, etc. Fabio Pianesi & Massimo Zancanaro FBK

47. Question & Answers #2 • How could the data on target behaviors in NETCARITY be used to improve segmentation of activities in recordings of ongoing natural behavior? • Segmentation is a ill-posed problem because it confuses two level: the description of an activity and the intention of the performer • Example: • While I cook spaghetti, I go to the restroom. A friend call and I say “I’m cooking” (still in the restroom!) • I grab an hammer and my wife asks me about what I’m doing: “I’m hanging the painting” but I have not yet started (or not?) • Telic events have a clear end but still lack a clear start • If the apple is finished, you have ate an apple • But it’s hard to agree on the start (or if you leave the apple on the table) Fabio Pianesi & Massimo Zancanaro FBK

48. Question & Answers #3 • How might recordings from the high density microphone arrays used in this project provide value to other researchers? Would this justify the cost? • For what concerns event detection, we had disappointing results from microphone • They can be useful to monitor verbal and para-verbal activities to estimates: • personality traits (Pianesi et al. 2008; Lepri et al. 2009) • mood Fabio Pianesi & Massimo Zancanaro FBK

49. Aaron Crandall Washington State University D.J. Cook, M. Schmitter-Edgecombe, Chad Sanders, Brian Thomas

50. Lorcan Coyle Lorcan.coyle@lero.ie Lero – The Irish Software Engineering Research Centre University of Limerick Juan Ye, Susan McKeever, Stephen Knox, Matthew Stabeler, Simon Dobson, and Paddy Nixon University College Dublin