Generating Remote Control Interfaces for Complex Appliances

1. Generating Remote Control Interfaces for Complex Appliances Jeffrey Nichols Carnegie Mellon University August 22, 2002

2. The Problem Everyday appliances are becoming more complicated and thus more difficult to use • Cheap to add computation and features • Expensive to design and build high-quality interfaces

3. The Solution Observation:People are increasingly carrying handheld devices, like mobile phones and PDAs • These devices will soon be able to communicate with devices around them (Bluetooth, 802.11b) • Better built-in UI hardware (color touch screens, etc.) Solution:Use handhelds as remote controls!

4. Feedback Specifications Control Personal Universal Controller An intermediary remote control interface Benefits • Graphical and speech interfaces • Adaptable interfaces (to PUC device and to user)

5. Overview Architecture Specification Language Interface Generators • Algorithms using dependency information Demonstration

6. Architecture

7. Appliance Adaptors

8. Communication Protocol

9. Interface Generators

10. Specification Language XML-based!!!

11. Language Design Approach • Create reference interfaces • AIWA Shelf Stereo • AT&T Telephone/Answering Machine • Test interfaces with subjects • Users twice as fast and made half errors as compared with manufacturer’s interfaces • Analyze interfaces for functional information

12. Language Elements State Variables and Commands • Represent functions of stereo • Variables sufficient for most functions but not all • “seek” button on a Radio Type Information • State variables have types • Boolean, Enumeration, Integer, String, etc. • Interface generators can use type to infer control

13. Language Elements, cont. Label Information • Good labels are an important of any interface • One label does not fit all • The optimal label size changes with screen size • Speech interfaces made need pronunciation and text-to-speech information • Solution: Label Dictionary • Collection of similar labels • As many as necessary • Different modality information

14. Language Elements, cont. Group Tree • Organization is very important to user interfaces • Tree is a good structure as most interfaces can be described as a tree • We use n-ary tree with variables/commands at leaves

15. Language Elements, cont. Dependency Information • Formulas that specify when a variable or command is active in terms of other state variables • Equals, Greater Than, Less Than • Linked with logical operators (AND, OR) • For example, <and> <equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals></and>

16. Overview Architecture Specification Language Interface Generators • Algorithms using dependency information Demonstration

17. Graphical Interface Generator Focuses on panel structure of user interface • Small groups of controls have simple layouts • Complexity comes from relative layouts of groups Structure can be inferred from dependency info • Groups are often defined by sets of controls that are all active at the same time (mutual exclusion) • May overlap other controls that not active

18. Determining Panel Structure • Find mutual exclusion within group tree • Find group that containsdepended-upon state • Look for mutual exclusionsamong group children • Assign branches of group tree to sets of panels • Choose a panel based upontype information and properties of group tree.

19. Choosing Panel Types Rules for Choosing Panels • If state has enumerated type and there are mutually exclusive groups of controls for each value, create a tabbed panel. • If state has boolean type and all other controls are active for only one value, create two full-screen overlapping panels. • Create overlapping panels for each mutually exclusive group and ensure there is a control elsewhere for setting the state.

20. Making the Interface Concrete Choose a control for each object Use a decision tree with questions such as: • Is this a command or state variable? • What is the type of the state variable? • Is the state variable read-only? • Was a panel structure rule applied from this variable?

21. Making Concrete, cont. Rules for choosing row layouts within panels • A labeled group with only two unlabeled controls is given a two-column layout with group label • Controls that prefer a lot of horizontal room are given a full width layout • If a group contains just two labeled controls are given a two-column layout without group label

22. Making Concrete, cont. Conceptual layout has been chosen • Panels, Controls, Rows Allocate space • Examine panel contents and choose sizes Instantiate and place controls

23. Overview Architecture Specification Language Interface Generators • Algorithms using dependency information Demonstration

24. Future Work • Improve quality of generated user interfaces • Back-tracking, etc. • Specification and generation of list interfaces • Necessary for many appliances • Adaptive interface generation • New interfaces modeled on past interfaces • General purpose appliance adaptor for HAVi and UPnP protocols

25. PUC Project Members Brad A. Myers Michael Higgins Joseph Hughes Thomas K. Harris Roni Rosenfeld Stefanie Shriver Peter Lucas Mathilde Pignol Kevin Litwack Funding National Science Foundation Pittsburgh Digital Greenhouse Microsoft Equipment Grants Mitsubishi (MERL) VividLogic Symbol Technologies Hewlett-Packard Lucent Acknowledgements

26. Thanks! Generating Remote Control Interfaces for Complex Appliances August 22, 2002 http://www.cs.cmu.edu/~pebbles/puc/ http://www.cs.cmu.edu/~jeffreyn/

28. Framework for Adaptors PUCProxy

29. Control of IEEE 1394 DV Camera Sony Camcorder with AV/C protocol

30. Control of Audiophase Stereo Custom Hardware and Software Adaptor Continue to Future Work

31. Interface Generation Process Three Phases • Determine panel structure from dependency info • Assign controls to variables/commands • Allocate space and lay out controls