CogTool A tool for interface design and ACT-R research

1. CogToolA tool for interface design and ACT-R research Bonnie E. John HCI Institute Carnegie Mellon University

2. CogTool is an open source tool where you can describe an environment in a storyboard…

3. CogTool is an open source tool where you can describe an environment in a storyboard…

4. …and demonstrate a task

5. CogTool automatically creates an ACT-R model of a skilled person doing this task and produces predictions of task execution time.* * Based on Card, Moran and Newell’s Keystroke-Level Model (1980)

6. And you can look under thehood to see what ACT-R is doing.

7. What is prediction of skilled task execution time good for? Some examples of similar analyses • PERCs “time travel” evaluation at IBM • DARPA set requirement to show 10x productivity improvement over 2002, which we can credibly demonstrate with CogTool • Saved NYNEX from making a $160 million workstation purchase that would have COST them $2 million/year in operating costs • IRS procurement of new IT system turned on “value” calculated, in part, using this type of analysis (IBM lost $700 million contract to AT&T’s $1.4 billion) • NextGen airspace will have economic consequences of time to execute cockpit tasks • Carlsbad Police reduced injuries and loss of life in their in-vehicle information systems. • SAE Recommended Practice J2365, Calculation of the Time to Complete In-Vehicle Navigation and Route Guidance Task.

8. Why should researchers at the ACT-R workshop care? • If you are not in academia • Your organization may builds systems and this could be directly useful for evaluating them • If you are in academia • Consulting • this may be a quicker way to evaluate new systems than directly coding ACT-R • Teaching • Human Factors or User Interface Design classes and this is one technique you could teach your students • Psychology class, part of which may be applied • Want a gentle introduction to cognitive modeling to get students excited • Research • Rapid environment construction • “Rapid Theory Prototyping”

9. First a word about teaching with CogToolKLMs “By-Hand” v. KLMs with CogTool (BRIMS 2010) • The variability between novice modelers has been reduced by 70% • This makes modeling the least variable of all usability techniques!!! By-Hand KLM:Average CV=22% CogTool KLM:Average CV=7%

10. Rapid Environment Construction • Use CogTool’s storyboarding to construct an environment for your ACT-R model • Export the ACT-R code CogToolcreates • Put in your own ACT-R model

11. Rapid Environment Construction • Use CogTool’s storyboarding to construct an environment for your ACT-R model • Export the ACT-R code CogToolcreates • Put in your own ACT-R model • The interface for doing this isn’tas easy as I would like • The environment isn’t yet a trueACT-R device model • Anybody who would like to helpand contribute to our opensource code, please contact me

12. Rapid Theory Prototyping • Or how I explored 7 theories before breakfast

13. UI Prototype Prototypers = UI designers Far easier, quicker to build than fully-functional UI Limited to a few specific tasks Need only be “good enough” to testuser behavior with a proposed UI “cheating”(e.g., wizard-of-oz) Sufficient to suggest what’s important enough to begin implementation and what should be given more thought Tool for thought and communication Rapid UI Prototyping

14. UI Prototype Prototypers = UI designers Far easier, quicker to build than fully-functional UI Limited to a few specific tasks Need only be “good enough” to testuser behavior with a proposed UI “cheating”(e.g., wizard-of-oz) Sufficient to suggest what’s important enough to begin implementation and what should be given more thought Tool for thought and communication Theory Prototype Prototypers = Theory developers Far easier, quicker to build than fully-functional theory Limited to a few specific tasks Need only be “good enough” to testtheory’s behavior against human data “cheating” Sufficient to suggest what’s important enough to begin implementation andwhat should be given more thought Tool for thought and communication Rapid UI Prototyping Rapid Theory Prototyping

15. General approach to rapid theory prototyping • Start with the simplest, most generic model of a theory and ask it to do a task • When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledge • Repair the the model through rapid theory prototyping Repeat steps 2-3 until the model is failing only by chance • Analyze all the failures and decide: • Whether prototyped theory or mechanism matches human performance • If so, prioritize effort to improve the original simple theory to validate and then move into the tool for design

16. Example of an aviation task • Start with the simplest, most generic model of a theory and ask it to do a task Simplest, most generic model of a theory for exploring an aviation device: Information Foraging Theory (Pirolli & Card, 1999) Augmented with the Minimal Model of Visual Search (Halverson & Hornof, 2007) Using a general knowledge corpus Embodied in CogTool-Explorer (Teo & John, 2008) A task: A three step procedure to set the approach reference speed and flap angle, using the CDU in a 777.

17. Example Run of CogTool-Explorer 17

18. 7 theory prototypes in 4 hrsThe final one approaches human behavior (HFES09) P’type-1 P’type-2 P’type-3 P’type-4 P’type-5 P’type-6 P’type-7 Change knowledge (aviation vocabulary) Baseline(simplest,most generic) Elaboration + How it works knowledge

19. Different prototypes target improvement in different steps P’type-1 P’type-2 P’type-3 P’type-4 P’type-5 P’type-6 P’type-7

20. Example: Step 1 P’type-1 P’type-2 P’type-3 P’type-4 P’type-5 P’type-6 P’type-7

21. Step 1 Given the goal: “select landing flap and reference air speed for a approach” Hit the INIT REFbutton

22. Example: Step 1 P’type-1 P’type-2 P’type-3 P’type-4 P’type-5 P’type-6 P’type-7 Change knowledge (aviation vocabulary) Elaboration + How it works knowledge

23. Example: Step 1, Theory Prototype 4 P’type-1 P’type-2 P’type-3 P’type-4 P’type-5 P’type-6 P’type-7 Hierarchical visual regions & recovery

24. Theory Prototype 4:Hierarchical visual regions + recovery • We “cheated” by prototyping this theory in the CogTool storyboard, not the ACT-R code • CogTool runs ACT-R models on a UI description called a storyboard. • States, which contain widgets (e.g., buttons), and transitions between states represent human actions on widgets (e.g., pressing a button) • Flat representation of widgets, so visual search and information foraging considers them equally • Cheat: Prototype visual regions = large “buttons” on the regions • When the correct region is chosen, it transitions to a state with only the buttons in that region • When an incorrect region is chosen, it transitions to a state with only the regions that have not been chosen • 1 hour change

25. Original storyboard for this task:Each state has the CDU’s 69 buttons • How we “cheated” by prototyping this theory in the CogTool storyboard: • Prototyping visual regions = large “buttons” on the regions • When the correct region is chosen, it transitions to a frame with only the buttons in that region • When an incorrect region is chosen, it transitions to a frame with only the regions that have not been chosen • 1 hour change

26. Theory Prototype 4:Hierarchical visual regions

27. Theory Prototype 4:Prototyped as buttons in storyboard

28. VNAV ALTN DEPARR RTE INITREF PROG FMCCOMM HOLD LEGS FIX NAVRAD MENU Theory Prototype 4:Correct region buttons in region

29. VNAV ALTN DEPARR RTE INITREF PROG FMCCOMM HOLD LEGS FIX NAVRAD MENU Theory Prototype 4:Incorrect region other regions No green = no cycling

30. Theory Prototype 4:Tremendous improvement on 1st step P’type-1 P’type-2 P’type-3 P’type-4 P’type-5 P’type-6 P’type-7 Tremendousimprovement

31. Theory Prototype 4:But still insufficient to complete task P’type-1 P’type-2 P’type-3 P’type-4 P’type-5 P’type-6 P’type-7 Tremendousimprovement Not sufficient to do a multi-step task

32. General approach to rapid theory prototyping • Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search • When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery • Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper • Analyze all the failures and decide: • Whether prototyped theory matches human performance • If so, prioritize effort to improve the underlying theoryHigh priority: Hierarchical visual regions + recovery

33. General approach to rapid theory prototyping • Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search • When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery • Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper • Analyze all the failures and decide: • Whether prototyped theory matches human performance • If so, prioritize effort to improve the underlying theory

34. General approach to rapid theory prototyping • Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search • When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery • Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper • Analyze all the failures and decide: • Whether prototyped theory matches human performance • If so, prioritize effort to improve the underlying theory

35. General approach to rapid theory prototyping • Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search • When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery • Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper • Analyze all the failures and decide: • Whether prototyped theory matches human performance • If so, prioritize effort to improve the underlying theory

36. General approach to rapid theory prototyping • Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search • When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery • Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the HFES 2009 paper • Analyze all the failures and decide: • Whether prototyped theory matches human performance • If so, prioritize effort to improve the underlying theory

37. General approach to rapid theory prototyping • Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search • When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery • Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper • Analyze all the failures and decide: • Whether prototyped theory matches human performance • If so, prioritize effort to improve the underlying theoryHigh priority: Hierarchical visual regions + recovery See Leonghwee Teo’s talk on Sunday

38. General approach to rapid theory prototyping • Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search • When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery • Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper • Analyze all the failures and decide: • Whether prototyped theory matches human performance • If so, prioritize effort to improve the underlying theoryHigh priority: Hierarchical visual regions + recovery See Leonghwee Teo’s talk on Sunday

39. General approach to rapid theory prototyping • Start with the simplest, most generic model of a theory and ask it to do a taskFlat visual search • When it fails to do the right action on the task, identify a missing (extra, or wrong) cognitive mechanism or knowledgeHierarchical visual regions + recovery • Give the model this mechanism or knowledge through “rapid theory prototyping”Quick change to CogTool’s storyboard, not ACT-R code Repeat steps 2-3 until the model is failing only by chance7 iterations, described in the paper • Analyze all the failures and decide: • Whether prototyped theory matches human performance • If so, prioritize effort to improve the underlying theoryHigh priority: Hierarchical visual regions + recovery See Leonghwee Teo’s talk on Sunday Other examples of Rapid Theory Prototyping • My poster with Tiffany Jastrzembsk for modeling aging aduts (sort of) • Paper at the ASSETS conference modeling blind users of screen readers