User Studies Methods

1. User Studies Methods Feb 01, 2007

2. Case Studies • Chameleon

3. Case Study: Chameleon • Design proposal introducing new user interface metaphor

4. Case Study: Chameleon • Iterative Design • Paper prototype -> Visual Basic -> Implement • Increasingly refined prototypes • Evaluation of each prototype

5. Chameleon Study #1 • Understand feasibility of basic idea • How people used security features • Explicit vs implicit role switching • Used paper prototype • Recruited 10 people from campus • Unclear, but presumably typical users w/o extensive computer experience

6. Chameleon Study #1 • “We recruited 10 people from around our campus to use the paper prototype while we observed them ad listened to their comments about what they found confusing, easy, difficult, helpful, etc.” • “Participants also filled out a web-based questionnaire about their experiences using the prototype”

7. Chameleon Study #1 • Fairly typical of an early formative study • Formative means early stages of design • Summative means later stages (timing data) • Lots of qualitative feedback • Useful for early stages • Should be able to notice major issues w/o having to do extensive analysis • Little unclear what the tasks were • Specific tasks to understand usability • Freeform tasks to understand utility

8. Chameleon Study #1 • Web survey useful too • Lots of positive and negative comments • Always a good idea to do a survey • Helped flesh out major issues • Switching roles needed to be improved • User motivation issues • Names of roles

9. Chameleon Study #1 • Comments: • Good to show alternative designs after such a study • People not as good evaluating a single design, better to show alternatives and have them compare differences

10. Chameleon Study #2 • Drilling down on the UI • How people should perform key operations • Ex. Moving a file from one role to another • Roughly three designs per operation • Within-subjects design (each person tries all) • How to address learning effects?

11. Chameleon Study #3 • Visual Basic prototype • More refined prototype let them study issuesmore in-depth than possible with paper • Injected an “attack”, window that appeared to be in certain role but was in another • One issue with security studies is timing, may want people to become comfortable and then see if they notice and how they react • Few participants noticed 

12. Chameleon: General Comments • Start simple and with big issues first • Progressively refine the prototypes • Don’t drill down to small issues until needed • UI design studies should inject an attack • See whether people notice • Can try various UIs to compare effectiveness

13. Kazaa File Sharing Study • Good and Krekelberg, CHI 2003 • Could people understand what files were downloadable by others? • Found lots of people sharing inbox.dbx • Found that some people were downloading a fake inbox.dbx file

14. Kazaa Cognitive Walkthrough • Cognitive Walkthrough • Put yourself in shoes of users and try to use the interface from their perspective • Somewhat effective approach, depends on ability of person to see other perspectives • Problem #1: Multiple names for similar things • My Shared Folder - a folder + all shared files • My Media - all shared files by media type • My Kazaa - all shared files by media type • Folder for downloaded files - root folder of all shared files

15. Kazaa Cognitive Walkthrough Problem 2: Downloaded files are also shared files Problem 3: Kazaa recursively shares folders

16. Kazaa Cognitive Walkthrough Problem 4: Can select a folder, but what files are inside? Error-prone approach. Also risk with recursive folders.

17. Kazaa Cognitive Walkthrough Note: Gives one-time warning if you select an entire hard drive

18. Kazaa Cognitive Walkthrough • Problem 5: Inconsistent views • Two UIs for doing similar tasks, but show different information about state of system

19. Cognitive Walkthru Discussion • Fairly effective technique • May be useful to apply multiple times from multiple perspectives • Parent who has things to protect • Teen who wants to download music • May have false positives • Probably best to do cog walkthru with multiple people, combine issues, and triage • Importance (not a problem -> catastrophe) • Cost (trivial -> major rework)

20. Kazaa File Sharing Study • 12 users, 10 had used file sharing before • Figure out what files being shared by Kazaa • Download files set to C:\ (ie all files) • Results • 5 people thought it was “My Shared Folder” • which one UI did suggest

21. Kazaa File Sharing Study • 12 users, 10 had used file sharing before • Figure out what files being shared by Kazaa • Download files set to C:\ (ie all files) • Results • 5 people thought it was “My Shared Folder” • which one UI did suggest • 2 people used Find Files to find all shared files • This UI had no files checked, thus no files shared?

22. Kazaa File Sharing Study • Results • 5 people thought it was “My Shared Folder” • which one UI did suggest • 2 people used Find Files to find all shared files • This UI had no files checked, thus no files shared? • 2 people used help, said “My Shared Folder” • 1 person couldn’t figure it out at all • Only 2 people got it right

23. Kazaa File Sharing Study • 12 participants a little low, though results strong enough to indicate big problems • Could have tried to verify cognitive walkthrough issues • Could have tried to test people’s ability to configure system (defaults important!) • Interesting point: • Had to set up system to prevent any actual sharing of files • We’ve had similar issues wrt phishing

24. Are people still accidentally sharing files? • A rough & ready experiment by your friendly instructor (2006) • eMule (open source) • Combines eDonkey and Kad file sharing Different from FastTrack (Kazaa file sharing) • eMule stats • Downloaded by over 85 million people • 5.3 mil people / 633 mil files on eDonkey • 1.7 mil people / 300 mil files on Kad

35. eMule File Sharing UI

36. Design Model User Model System Image Putting Them Together • Lessons from Chameleon + Kazaa • Examples of how to run user studies • Not the most rigorous studies, but good enough to demonstrate main point • Examples of mental models

37. Other General Comments • Inform people that it’s a security study? • Can’t get useful results if informed • Ethics of not informing people • Involves some element of deception • Phishing studies framed as email studies • Golden rule useful here: treat people as you would like to be treated

38. Heuristic Evaluation • Mentioned in “Why Johnny Can’t Encrypt” • Similar to cognitive walkthrough • Helps find usability problems in a UI design • Can perform on working UI or on sketches • Small set (3-5) of evaluators examine UI • independently check for compliance with usability principles (“heuristics”) • different evaluators will find different problems • evaluators combine findings afterwards

39. Why Multiple Evaluators? • Every evaluator doesn’t find every problem • Good evaluators find both easy & hard ones

40. Heuristic Evaluation Process • Evaluators go through UI several times • inspect various dialogs and screens • compare with heuristics and other usability principles • “Standard” set of heuristics • Can also create domain-specific heuristics • competitive analysis & user testing of existing products • Use violations to redesign/fix problems

41. searching database for matches Heuristic H2-1 • H2-1: Visibility of system status • keep users informed about what is going on • example: pay attention to response time • 0.1 sec: no special indicators needed, why? • 1.0 sec: user tends to lose track of data • 10 sec: max. duration if user to stay focused on action • for longer delays, use percent-done progress bars

42. Heuristic H2-2 • H2-2: Match between system & real world • speak the users’ language • follow real world conventions • Example: Mac desktop • Dragging disk to trash • should delete it, not eject it • finally fixed in Mac OS X

43. Heuristic H2-3 • H2-3: User control & freedom • “exits” for mistaken choices, undo, redo • don’t force down fixed paths • like that BART machine…

44. Heuristic H2-4 • H2-4: Consistency & standards

45. Heuristic H2-5 • H2-5: Error prevention

46. Heuristic H2-6 • H2-6: Recognition rather than recall • make objects, actions, options, & directions visible or easily retrievable

47. Heuristic H2-7 • H2-7: Flexibility and efficiency of use • accelerators for experts (e.g., gestures, kb shortcuts) • allow users to tailor frequent actions (e.g., macros)

48. Heuristic H2-8 • H2-8: Aesthetic and minimalist design • no irrelevant information in dialogues

49. Heuristic H2-9 • H2-9: Help users recognize, diagnose, and recover from errors • error messages in plain language • precisely indicate the problem • constructively suggest a solution