MIDAS Multi-device Integrated Dynamic Activity Spaces

1. MIDAS Multi-device Integrated Dynamic Activity Spaces Unmil P. Karadkar Ph.D. Candidate Dept. of Computer Science Texas A&M University

2. Research Questions • Theory • Which aspects of information content do humans find valuable? • Techniques • How to adapt content while optimizing for human perception? • How to enable heterogeneous device co-use? • Frameworks • What are the attributes of multi-device systems? • How options are available to system designers for each attribute? • System • Design a flexible architecture to embody multi-device system behaviors • Proof-of-concept implementation and analysis • Practice • Harness device multiplicity to accomplish significant tasks

3. Outline Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work

4. Outline Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work

5. 6” 7” 4.5” Custom 10” 14” 17” 75” 21-24” 40”- 60” 3.5” 13”- 17” • Display characteristics • Network bandwidth • Processing power • Interaction modalities

6. Societal Changes • User demographics • Nature of activity • Access location • Affordability • Free phones with contract • $150 e-readers • Multiplicity • iPhone, iPad, MacBookPro Heterogeneous contexts of information access

7. MIDAS Multi-device Integrated Dynamic Activity Spaces M I D A S

8. Design Goals • Deliver information elements to multiple devices simultaneously • Content • Interaction • Rely on widely available infrastructure • Wired and wireless computer networks • Mobile phones, e-readers, tablets, desktop computers • Sensors embedded in the environment • Support content transformation • Within form (image scaling, text summarization) • Across forms (text to speech, audio extraction from video)

9. Design Goals • Strive to retain content integrity from a human perspective • Select transformations that a user perceives to be closest to the original form • Direct content elements to devices that can present them optimally • Reconfigure presentation dynamically in response to changes in device availability • New devices become available • Current devices become unavailable

10. Outline Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work

11. Background Individual devices Opportunistic device co-use Ubiquitous Computing Infrastructure-enabled device co-use device-based content adaptation • Service separation • Yahoo!, CNN, Weather Channel, UT Mobile • Image distillation • Digestor (Fox 1996) • Text and Web form summarization • Pythia (Bickmore 1997) • Power Browser (Buyukkokten 2002) • Proteus (Anderson 2001) • Web page content extraction • Baseline SVPs + device-based PVPs (Chua 2007) • Location-aware services • Computer for the 21st century (Weiser1991) • Java Ring (Olsen 2001) • IntelliBadgeTM(Cox2003) • Mobile devices in interactive spaces • Personal Server (Want 2002) • InfoStick(Kohtake1999) • Elope (Pering2005)

12. Background—Device co-use • Pebbles(Myers 1997) • Desktop and PDA co-use • Extend Windows applications to PDAs • PDAs as input and control devices • WebSplitter(Han 2000) • Split Web content across a user’s devices • Support for multiple views • ANMoLE (Haneef 2004) • Dynamic content reformulation (splitting video) • Integration of phone and computer networks

13. Background—Device co-use • Ubiquitous Display System (Aizawa 2002) • Large public displays and mobile phones • Control from mobile phone • Multibrowsing (Johanson 2001) • Desktops, notebooks, PDAs, and large displays • Specially encoded links for content routing • Interface Distribution Daemon(Luyten 2005) • Web interface as well as stand-alone applications • Multiple content allocation modes • Usability analysis - Completeness and Continuity

14. Outline Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work

15. 12C Framework • Concurrency • Simultaneous multi-device content presentation • Control • Interaction support • Comity • Permissible diversity in device characteristics • Completeness • Presentation of relevant information elements • Coverage • Distribution of content to optimize device usage • Conversion • Content transformation to suit target device characteristics

16. 12C Framework • Composition • Relationship of components to the interface • Coherence • Consistency in presentation of content elements • Coordination • Reallocation of elements when devices change • Continuity • Support for users to interpret and evaluate system state • Constancy • Reuse of content instances that have been shown before • Confidence • Security and trustworthiness in communication

17. Outline Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work

18. Development Platform Web or something else?

19. WorldWide Web • WWW • Dominant infrastructure • World-wide audience • Stateless server • Client-initiated communication • Content embedded within structure • <p>, <h1> • Changing devices means restarting the activity • Standardized browsers • Little support for custom behavior

20. context-aware Trellis (caT) • Petri net-based hypertext system (Furuta, Stotts - 1989) • Formal, graph-based model • Browsing semantics (user privileges, time of day, location) • Decouple specification from information content • Content not subsumed within structure • <img>, <a> • Separation allows devices to select suitable content • Decouple content from presentation • Browsers present content autonomously • Stateful server • Support for multiple clients • Easy propagation of user actions and content to multiple devices

21. Outline Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work

22. Specification Information Service MIDAS Browser 2 Browser 2 Browser 2 Browser 3 Browser 3 Browser 3 Browser 4 Browser 4 Browser 4 Device 1 Device 2 Device 3 Users Architecture caT Browser

23. Specification Information Service MIDAS Browser 2 Browser 2 Browser 2 Browser 3 Browser 3 Browser 3 Browser 4 Browser 4 Browser 4 Device 1 Device 2 Device 3 Users Architecture Device Manager Browser Coordinator Resource Realizer Application Coordinator Application Coordinator Browser

24. Specification content handles Information Service user actions content handles instance handle status, user actions instance properties instance handle, actions MIDAS content user actions content handle content instance properties Browser 2 Browser 2 Browser 2 Browser 3 Browser 3 Browser 3 Browser 4 Browser 4 Browser 4 Resource Repository Device 1 Device 2 Device 3 Users Architecture Device Manager (instances, devices) Browser Coordinator Resource Realizer Application Coordinator Application Coordinator Browser

25. Device Manager Information Service Fully Replicated Interaction Replicated Instance Optimal MIDAS Optimal Content Handles, properties Device Selection Scheme Content-Device Map Instance properties Instance Scorer Device properties, interaction mode Presentation only Input only Interactive Browser Coordinator Resource Realizer

26. Outline Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work

27. Content Transformation • Images • Scaling (automatic) • Color reduction (automatic) • Text • Text extraction from formatted documents (automatic) • HTML, Word, PDF • Summarization (semi-automatic) • Formatted data such as tables (semi-automatic) • Visual or audio rendering (automatic) • Video • Audio extraction, down-sampling (automatic) • Video summarization (semi-automatic)

28. Resource Manager Authoring support Device dependence for content instances Automatic transformation of images (ImageMagick) Author validation

29. Perception-based Image Transformation • Images • Changes to attributes of photographs • Fastest growing information content on the Web • Goal of user study • Understand human perception of nearness • Scaling of images • Color reduction (and gray scaling) • Obtain actionable metrics to design rules for providing content in its most suitable form

30. Study Design • Two stages • Stage 1: Image classification • Stage 2: Similarity of image pairs • Subject demographics • Five subjects per stage • Graduate students and staff • Different disciplines

31. Stage 1 • Subjects classified 100 photographs • Taken by four photographers • Everyday and travel pictures • Prints of digital photos to make piles • No guidelines regarding classification • Subjects created between 12 and 40 categories • I coalesced categories based on image content • Four final categories

32. People Nature Structures Text Image Types

33. Stage 2 • Subjects viewed image pairs on identical displays • No overlap or switching windows • Each pair differed in size or color • Variables manipulated independently • 20 image pairs for each variable • Answered three questions • Similarity of images (9-point scale) • Suitability of automatic substitution (yes/no) • Acceptance of informed substitution (9-point scale)

34. Scaling

35. Color Reduction

36. 1 3 160x120 2 (1-bit) 320x240 4 (2-bit) 640x480 16 (4-bit) 800x600 256 (8-bit) 1024x768 16 Million (24-bit) 2 4 Data Analysis Colors Size

37. Effect of Information Content Loss Color depth change Scaling Scores on a 9 pt. Scale 1024 X 768 X 24 vs. 160 X 120 X 24 320 X 240 X 24 vs. 320 X 240 X 1 (no loss) (98 %) (no loss) (96 %)

38. Similarity by image type • Images of nature and structures scale well • Human faces • Legibility of textual elements • Algorithms for face and text detection in images exist Scaling

39. Scaling– Image Substitution • No clear trends in terms of distance • Subjects allowed automatic substitution • Corresponded with higher replacement suitability

40. Color Reduction – Image Substitution • Acceptance of automatic substitution is lower than that for scaled images • The suitability ratings are correspondingly higher

41. (9 - Scoredist, size) Wsize x x Wtype 9 (9 - Scoredist, color ) (9 - Scoredist, color ) + Wtogray + Wcolor x x 9 9 Perception-informed Rules • Principles • Prefer scaling over color reduction • Avoid scaling of images that contain people and text • Gray scaling is the best option when used by itself • Normalized score: Original image = 0 (best score) Image Rating = 0.4 Wtype 0.6 Wtype (people, text) = 0.53 Wtype (nature, structures) = 0.47 Wtogray= 1.0 (grayscale) Wtogray= 0.0 (otherwise)

42. Outline Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work

43. Outline Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work

44. Analysis - Design Goals ✔ • Deliver information elements to multiple devices simultaneously • Content • Interaction • Rely on widely available infrastructure • Wired and wireless computer networks • Mobile phones, e-readers, tablets, desktop computers • Sensors embedded in the environment • Support content transformation • Within form (image scaling, text summarization) • Across forms (text to speech, audio extraction from video) ✔ ✔

45. Analysis - Design Goals • Strive to retain content integrity from a human perspective • Select transformations that a user perceives to be closest to the original form • Direct content elements to devices that can present them optimally • Reconfigure presentation dynamically in response to changes in device availability • New devices become available • Current devices become unavailable ✔ ✔ ✔

46. Analysis - 12C Framework Concurrency Control Comity Completeness Coverage Conversion Composition Coherence Coordination Constancy Continuity Confidence Present information elements on multiple devices • Inclusion of devices in MIDAS is optional • WebSplitter, ANMoLE • A user can choose the interaction mode for each device • Inclusion of public devices is not supported • Ubiquitous Display System • Content is directed to devices automatically • Multibrowsing, Ubiquitous Display System

47. 12C Framework Concurrency Control Comity Completeness Coverage Conversion Composition Coherence Coordination Constancy Continuity Confidence Interaction support • Flexible – dependent on device choices • Most architectures support centralized or distributed • It is possible to have no input devices at all • May change during a session

48. 12C Framework Concurrency Control Comity Completeness Coverage Conversion Composition Coherence Coordination Constancy Continuity Confidence Permissible diversity in device characteristics • MIDAS places few restrictions on devices • Pebbles, Ubiquitous display system • Device must run the Browser Coordinator • No requirement of proximity to a user • Remote devices may be included - printers

49. 12C Framework Concurrency Control Comity Completeness Coverage Conversion Composition Coherence Coordination Constancy Continuity Confidence Presentation of relevant information elements • Ensures completeness by presenting all resource ids • Resource Manager aids content authors • Support includes textual description as fallback • IDD assertion is based on fixed content instances • ANMoLE and MIDAS convert content

50. 12C Framework Concurrency Control Comity Completeness Coverage Conversion Composition Coherence Coordination Constancy Continuity Confidence Distribution of content to optimize device usage • Flexible – dependent on Device Manager mode • Fully replicated - redundancy • Interaction replicated • Instance optimal - coverage for individual instances • MIDAS optimal – system-wide coverage • Coverage may not apply to some systems • Ubiquitous Display System, Multibrowsing