
Studying The Use of Handhelds To Control Smart Appliances Jeffrey Nichols Carnegie Mellon University May 19, 2003
The Problem Appliances are too complex
The Problem, cont. • Each complex appliance has its own idiosyncratic interface! • Home and Car Stereos • Car Navigation Systems • Answering Machines • … • Increasingly Computerized • Low Usability
Specifications Control Feedback Our Solution Separate the interface from the appliance! Key Features • User interface-independent appliance specification • Automatic generation of GUI and speech interfaces
Benefits of Our Approach • Handheld has richer interface technology than appliance can afford • Color LCD screen, touch screen, text entry technology • More effort can be put into interface design technology • Appliance manufacturer’s must weigh trade-offs between usability, cost, time-to-market, etc. • Two-way communication channel • Better feedback can be provided to the user regarding the appliance’s state.
Automatic Generation of UIs Benefits • All interfaces consistent for the user • With conventions of handheld Other applications and UI guidelines • Even from multiple manufacturers Addresses idiosyncracy problem! • Multiple modalities (GUI + Speech UI) • Can take into account user preferences • Will work on special purpose devices (for disabled)
Outline • A First Step • User Studies • Current Work
A First Step • Build Reference Interfaces • Remote control interfaces for various appliances that we design manually. • Verify that better interfaces can be created on a handheld • Used for understanding what functional knowledge is necessary to make a good interface.
Reference Interfaces • Interfaces were hand-designed for two appliances and two handhelds • Appliances • AIWA CX-NMT70 Shelf Stereo • AT&T 1825 Telephone/Answering Machine • Handhelds • Palm • Microsoft PocketPC
Palm Interfaces We initially designed paper-prototype interfaces for Palm telephone stereo
PocketPC Interfaces We later implemented interfaces for Microsoft’s PocketPC (simulated remote control). telephone stereo
Interface Quality? • We iteratively improved the interfaces using heuristic analysis techniques. • We conducted a think-aloud study with several Carnegie Mellon students to find problems in the interfaces. • Lastly, we conducted a user study that compared our reference interfaces with the manufacturer’s interfaces.
Outline • A First Step • User Studies • Current Work
User Studies • Two Studies • Study #1: Paper-PrototypePalm vs. Actual Appliance • Study #2: Functional PocketPC vs. Actual Appliance
User Studies, cont. • Procedure • We did a between-subjects study. • Each subject worked on two sets of tasks. • In order to minimize subjects, each worked on both the stereo and the phone. • We controlled for order and interface type.
Evaluation of Task Performance • Three Metrics: • Time to complete all tasks • Number of times help was requested • How often did the subject need the manual or online help? • Number of missteps • Misstep = the pressing of a button that does not advance the progress on the current task • No missteps were counted after the user requested help.
User Study #1: PalmOS • Compared paper prototype interfaces with the interfaces of the actual appliances • Experimenter changed paperas subjects tapped • Control of stereo and phone simulated verbally • When the stereo started playing music, the experimenter said “you now hear music from the stereo”
User Study #1, cont. • Participants • 13 Carnegie Mellon Graduate Students • Five female, Eight male • Enrolled in School of Computer Science • Volunteers (unpaid) • Seven owned a Palm device • One had no Palm experience • Four owned Aiwa-brand stereo systems
User Study #1 Results Users made five times the errors and needed help twice as often with the actual appliances! All results significant (p < 0.001 for all)
User Study #2 • We implemented the interfaces on a handheld and simulated remote control of an actual appliance. • Remote control applications built in Visual Basic on a PocketPC • Control of stereo and phone simulated in software • Feedback appeared to come from the actual appliance
User Study #2, cont. • Participants • Twelve students from Carnegie Mellon • Four female, Eight male • Volunteered in response to a newsgroup advertisement • Paid $7 for their participation (30-45 minutes) • All had limited handheld experience • Half (6) owned Aiwa-brand stereos • Two had AT&T digital answering machines
User Study #2 Results All differences are significant (p < 0.05) About ½ the time and ½ the errors!
Qualitative Results • Why were the reference interfaces better? • Clear feedback and explanation of the current state was possible. • Elements could be disabled on the screen (graying out) • Functions were separated across multiple screens.
Outline • A First Step • User Studies • Current Work
Current Work Designed a XML-based Specification Lang • Functions of Device State Variables and Commands • Labeling Multiple labels are necessary • Grouping Hierarchical groups • Dependency Information For enabling and structure
Current Work, cont. • Built multiple automatic interface generators • PocketPC • SmartPhone • Tablet PC (desktop) • Speech
Funding National Science Foundation Microsoft General Motors Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) VividLogic Hewlett-Packard PUC Project Members Brad A. Myers Thomas K. Harris Roni Rosenfeld Michael Higgins Joseph Hughes Kevin Litwack Rajesh Seenichamy Mathilde Pignol Stefanie Shriver Jeffrey Stylos Peter Lucas Acknowledgements
Thanks! • For more information see… • http://www.cs.cmu.edu/~jeffreyn/ • http://www.cs.cmu.edu/~pebbles/puc/ • Or e-mail me at… • jeffreyn@cs.cmu.edu
Actual Appliance Interfaces • Lots of Problems • Poorly labeled and overloaded buttons • Insufficient feedback • Timer example • Programming the speed-dial • Phone has technical separation between phone and answering machine
Qualitative Results • Grouping controls is important • Groups define which elements are placed adjacent to each other and how elements are separated onto pages. • Groupings vary between devices and interface styles.
Qualitative Results, cont. • Dual-associated functions are hard to make obvious for users • The record button is associated with both tapes (record onto) and each of the other modes (recorded from). • Some users expected the first mapping to used, whereas the controller used the second mapping.
Qualitative Results, cont. • Tree-based structures are not sufficient for fully describing an interface • Some interface concepts, especially dual-associated functions, break the tree because they may interact with the children of several different elements within the tree • The record button breaks the stereo’s tree structure because it is globally accessible but has different local effects.
Qualitative Results, cont. • A single function may map to multiple interface widgets (and vice versa) • Example: One state variable could be used to represent all of the playback states of a tape player. The play, stop, fast-forward, and rewind buttons all act on this single variable.
Applying These Results • We are actively applying these results to the design of the specification language • A tree-grouping structure is augmented with a dependency graph to help describe dual-mapped functions • Ranking relationships within groups using “priorities” • We will also apply them in the design of the automatic layout engine
Future Work • Build the specification language and automatic generation engine
A Hard Problem… • Automatically generating a good user interface is hard, but we think we can do it for several reasons: • Remote controls are a special class of user interface that use relatively simple interaction techniques. • Buttons, text fields, and other standard widgets. • Our approach differs from earlier work…
The Approach • Study Interfaces • Functional knowledge of the appliance • What must the appliance tell the handheld about itself so that a “good” interface can be constructed. • Design and Layout • How do we turn the knowledge about the appliance into a usable interface? • Design a specification language • Build an automatic interface generator
Our Progress… • Study Interfaces • Functional knowledge of the appliance • What must the appliance tell the handheld about itself so that a “good” interface can be constructed. • Design and Layout • How do we turn the knowledge about the appliance into a usable interface? • Design a specification language (in progress) • Build an automatic interface generator
Problems with User Study #1 • Paper-prototype study introduced a high possibility of experimenter interference. • Solution • Create an environment that completely simulates what one might experience using a personal universal controller • Interfaces running on an actual handheld • Interfaces should appear to control an actual appliance