CS305, HW1, Spring 2008 Evaluation Assignment

1. CS305, HW1, Spring 2008 Evaluation Assignment • Part 1: Proposal (Email to horton@cs.virginia.edu by 5pm, Friday, Feb. 8) • A short description of what you plan to evaluate (2-3 lines is enough) • Names of group members (3-4 people) • Possible participants for your evaluation • Choose any software / system to evaluate as long as: • You are at least somewhat familiar with it • You can find two or more participant users who are not experts • You can run the software / system in an experimental manner • It must involve some input/interaction as well as output. (Don’t want it to be passive for participant. GUI not required) • Web site? Only if it has some interesting complexity • Device? Probably OK. Game? Probably OK. Ask me.

2. Evaluation Assignment • Focus on some aspect of the software • Just part of a system if it’s complex • Focus on several usability goals • The user interactions that implement one to three tasks that all together can be done by an expert in, say, 20 minutes • Ideally a highly used or complex part of the software • You choose what goals matter to you. E.g. maybe you do or don’t want to include learnability or efficiency

3. Evaluation Assignment • Perform a co-operative evaluation and analysis of usability defects • Do a dry run with someone in your team • Remember that co-operative evaluation requires both you and the participant to verbalize • Your participants should not be experts but should be able to use or learn the software fairly quickly • If asked, you must volunteer to be someone else’s participant! • Also, create a table like that presented in class to evaluate success in tasks • Keeps good notes and records!

4. Evaluation Assignment: Your Report Turn in a report as follows on Mon., Feb, 18, 11 am • A description of the specific software or system and its aspects that are being evaluated. • Be sure to what the specific goals are for your evaluation! Just a subset of the system? Specific goals for this evaluation? • A summary of your procedures • When, where, how did you do your evaluation? (Only enough detail so I can see that you followed good procedures.) • Also, what tasks did you ask the participant to do? • Should include copies of the task descriptions you gave the user • (see next page)

5. Your report (cont’d) • List of all usability defects or problems by the user that you observe • What happened? What usability principle is violated? • OK if this has only a little detail, but feel free to add more detail if you wish • A more detailed analysis of the most serious defects (at least four defects) • Choose defects in the system’s design • How it supports tasks, or its conceptual model, or its interaction style, or problems with specific UI elements • State when error occurs in the goal/decide/execute cycle • Excerpt from your notes (quotes if possible) on what happened with the user(s) in the evaluation • Cause of the problem • More details than given in step 3 above. Make appropriate use of concepts of usability principles etc. • Any other appropriate detail from slides on usability defects • Recommendations for changes in the system

6. Your report (cont’d) • The table like Figure 12.9 (p. 610) for each task evaluated • But, list high-level tasks based on a user-goal here! • If something obvious, or easy, why bother? • Use this to evaluate how well the system serves user needs • Examples of good choices: • Add new entry to phonebook • Find and play specific MP3 file by name • Compose and send an email • Examples of bad choices: • Type in phone number • Pause the player (assuming there’s a pause-button) • Enter the body of the email message • A summary of any subjective comments made by your participants in the “wrap-up” discussion you have at the end of your evaluation.