1. Studying The Use of Handhelds To Control Smart Appliances Jeffrey Nichols Carnegie Mellon University May 19, 2003

2. The Problem Appliances are too complex

3. The Problem, cont. • Each complex appliance has its own idiosyncratic interface! • Home and Car Stereos • Car Navigation Systems • Answering Machines • … • Increasingly Computerized • Low Usability

4. Specifications Control Feedback Our Solution Separate the interface from the appliance! Key Features • User interface-independent appliance specification • Automatic generation of GUI and speech interfaces

5. Benefits of Our Approach • Handheld has richer interface technology than appliance can afford • Color LCD screen, touch screen, text entry technology • More effort can be put into interface design technology • Appliance manufacturer’s must weigh trade-offs between usability, cost, time-to-market, etc. • Two-way communication channel • Better feedback can be provided to the user regarding the appliance’s state.

6. Automatic Generation of UIs Benefits • All interfaces consistent for the user • With conventions of handheld Other applications and UI guidelines • Even from multiple manufacturers Addresses idiosyncracy problem! • Multiple modalities (GUI + Speech UI) • Can take into account user preferences • Will work on special purpose devices (for disabled)

7. Outline • A First Step • User Studies • Current Work

8. A First Step • Build Reference Interfaces • Remote control interfaces for various appliances that we design manually. • Verify that better interfaces can be created on a handheld • Used for understanding what functional knowledge is necessary to make a good interface.

9. Reference Interfaces • Interfaces were hand-designed for two appliances and two handhelds • Appliances • AIWA CX-NMT70 Shelf Stereo • AT&T 1825 Telephone/Answering Machine • Handhelds • Palm • Microsoft PocketPC

10. Palm Interfaces We initially designed paper-prototype interfaces for Palm telephone stereo

11. PocketPC Interfaces We later implemented interfaces for Microsoft’s PocketPC (simulated remote control). telephone stereo

12. Interface Quality? • We iteratively improved the interfaces using heuristic analysis techniques. • We conducted a think-aloud study with several Carnegie Mellon students to find problems in the interfaces. • Lastly, we conducted a user study that compared our reference interfaces with the manufacturer’s interfaces.

13. Outline • A First Step • User Studies • Current Work

14. User Studies • Two Studies • Study #1: Paper-PrototypePalm vs. Actual Appliance • Study #2: Functional PocketPC vs. Actual Appliance

15. User Studies, cont. • Procedure • We did a between-subjects study. • Each subject worked on two sets of tasks. • In order to minimize subjects, each worked on both the stereo and the phone. • We controlled for order and interface type.

16. Evaluation of Task Performance • Three Metrics: • Time to complete all tasks • Number of times help was requested • How often did the subject need the manual or online help? • Number of missteps • Misstep = the pressing of a button that does not advance the progress on the current task • No missteps were counted after the user requested help.

17. User Study #1: PalmOS • Compared paper prototype interfaces with the interfaces of the actual appliances • Experimenter changed paperas subjects tapped • Control of stereo and phone simulated verbally • When the stereo started playing music, the experimenter said “you now hear music from the stereo”

18. User Study #1, cont. • Participants • 13 Carnegie Mellon Graduate Students • Five female, Eight male • Enrolled in School of Computer Science • Volunteers (unpaid) • Seven owned a Palm device • One had no Palm experience • Four owned Aiwa-brand stereo systems

19. User Study #1 Results Users made five times the errors and needed help twice as often with the actual appliances! All results significant (p < 0.001 for all)

20. User Study #2 • We implemented the interfaces on a handheld and simulated remote control of an actual appliance. • Remote control applications built in Visual Basic on a PocketPC • Control of stereo and phone simulated in software • Feedback appeared to come from the actual appliance

21. User Study #2, cont. • Participants • Twelve students from Carnegie Mellon • Four female, Eight male • Volunteered in response to a newsgroup advertisement • Paid $7 for their participation (30-45 minutes) • All had limited handheld experience • Half (6) owned Aiwa-brand stereos • Two had AT&T digital answering machines

22. User Study #2 Results All differences are significant (p < 0.05) About ½ the time and ½ the errors!

23. Qualitative Results • Why were the reference interfaces better? • Clear feedback and explanation of the current state was possible. • Elements could be disabled on the screen (graying out) • Functions were separated across multiple screens.

24. Outline • A First Step • User Studies • Current Work

25. Current Work Designed a XML-based Specification Lang • Functions of Device State Variables and Commands • Labeling Multiple labels are necessary • Grouping Hierarchical groups • Dependency Information For enabling and structure

26. Current Work, cont. • Built multiple automatic interface generators • PocketPC • SmartPhone • Tablet PC (desktop) • Speech

27. Control of Real Appliances

28. Funding National Science Foundation Microsoft General Motors Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) VividLogic Hewlett-Packard PUC Project Members Brad A. Myers Thomas K. Harris Roni Rosenfeld Michael Higgins Joseph Hughes Kevin Litwack Rajesh Seenichamy Mathilde Pignol Stefanie Shriver Jeffrey Stylos Peter Lucas Acknowledgements

29. Thanks! • For more information see… • http://www.cs.cmu.edu/~jeffreyn/ • http://www.cs.cmu.edu/~pebbles/puc/ • Or e-mail me at… • jeffreyn@cs.cmu.edu

31. Actual Appliance Interfaces • Lots of Problems • Poorly labeled and overloaded buttons • Insufficient feedback • Timer example • Programming the speed-dial • Phone has technical separation between phone and answering machine

32. Qualitative Results • Grouping controls is important • Groups define which elements are placed adjacent to each other and how elements are separated onto pages. • Groupings vary between devices and interface styles.

33. Qualitative Results, cont. • Dual-associated functions are hard to make obvious for users • The record button is associated with both tapes (record onto) and each of the other modes (recorded from). • Some users expected the first mapping to used, whereas the controller used the second mapping.

34. Qualitative Results, cont. • Tree-based structures are not sufficient for fully describing an interface • Some interface concepts, especially dual-associated functions, break the tree because they may interact with the children of several different elements within the tree • The record button breaks the stereo’s tree structure because it is globally accessible but has different local effects.

35. Qualitative Results, cont. • A single function may map to multiple interface widgets (and vice versa) • Example: One state variable could be used to represent all of the playback states of a tape player. The play, stop, fast-forward, and rewind buttons all act on this single variable.

36. Applying These Results • We are actively applying these results to the design of the specification language • A tree-grouping structure is augmented with a dependency graph to help describe dual-mapped functions • Ranking relationships within groups using “priorities” • We will also apply them in the design of the automatic layout engine

37. Future Work • Build the specification language and automatic generation engine

38. A Hard Problem… • Automatically generating a good user interface is hard, but we think we can do it for several reasons: • Remote controls are a special class of user interface that use relatively simple interaction techniques. • Buttons, text fields, and other standard widgets. • Our approach differs from earlier work…

39. The Approach • Study Interfaces • Functional knowledge of the appliance • What must the appliance tell the handheld about itself so that a “good” interface can be constructed. • Design and Layout • How do we turn the knowledge about the appliance into a usable interface? • Design a specification language • Build an automatic interface generator

40. Our Progress… • Study Interfaces • Functional knowledge of the appliance • What must the appliance tell the handheld about itself so that a “good” interface can be constructed. • Design and Layout • How do we turn the knowledge about the appliance into a usable interface? • Design a specification language (in progress) • Build an automatic interface generator

41. Problems with User Study #1 • Paper-prototype study introduced a high possibility of experimenter interference. • Solution • Create an environment that completely simulates what one might experience using a personal universal controller • Interfaces running on an actual handheld • Interfaces should appear to control an actual appliance