Introduction to Team-based Usability Testing

1. Introduction to Team-based Usability Testing “As companies design more for usability and understanding, they will discover a competitive edge, for these principles save customers time and money while increasing morale.”- Donald Norman

2. Objectives • Usability overview • Setting objectives • Preparing the test plan • Conducting the test

3. Usability • Usability “means that the people who use the product can do so quickly and easily to accomplish their own tasks.” - Redish and Dumas, A Practical Guide to Usability Testing

4. Usability • Is not functionality • Functionality is how the system works • Is not beta testing • Beta testing is unfocused and at end of process • Is not market research • Market research is about customer demands

5. When to test • Design phase • surveys and focus groups • low fidelity prototype testing • high fidelity prototype testing • heuristic evaluation • Development phase • usability evaluation

6. Advantages of a usability lab • Lab is helpful for tightly controlling the variables • Lab helps capture low-level details • keyboard monitoring • videotaping for later analysis • eye movement tracking • Labs allow less intrusive observations

7. Disadvantages of a usability lab • Labs are intimidating to users • Labs are rarely portable so they cannot go into the users’ environment • Sometimes low-level information is not as important as making high-level design decisions • Labs are expensive

8. How long will the test take? • How much usability testing has already occurred? • Have users been profiled? • Has a task analysis occurred? • Have usability objectives been set? • How complex is the product? • How much of the product are you going to test?

9. How long will the test take? • How many test participants do you need to get the information you want? • How much training or prior experience does the test team have with the product? • How many different usability objectives do you need to test?

10. Who should run the test? • The most successful approach to usability testing is a cross-functional, team-based approach • People bring varying expertise to the team • All groups benefit from experience with users • Interpretation of challenges and design decisions are more creative with teams

11. Who should run the test? • Team-based testing provides a great way to get support involved early • Developers learn to “hear” user questions when they design • Writers and trainers can help design communication into the product • Human factors experts and user interface designers can provide helpful principles about users and design

12. Who should run the test? • More people can be involved in planning the test than in conducting the test • Target 3-5 people for conducting the test • Sometimes best if the developers are not actively involved in running the test, although they should observe if possible

13. How do you ensure success? • You need the buy-in of upper management if the results are to be applied • You need the cooperation of development to ensure they don’t feel attacked • You need the support of marketing, sales, and product management for funding • You need the support of your immediate manager to spend time testing

14. How do you ensure success? • Pick a manageable portion of the product to test • Select a high-profile project • Select an area that will produce significant results • Advertise your successes (become part of the sales story)

15. Stages of a usability test • Setting objectives • Creating the test plan • Creating questionnaires • Selecting test participants • Training the team • Conducting the pilot test • Conducting the tests

16. Usability objectives • Created during user/task analysis and product scoping • Must be measurable • Should indicate: • Type of user • Task to be performed • Specific performance criteria

17. Defining usability objectives • Five attributes of usability (Nielsen, 1994): • Learnability: system is easy to learn so users can get started quickly • Efficiency: system should be easy to use, resulting in high productivity • Memorability: system should be easy to remember • Errors: system should have low error rate and allow error recovery • Satisfaction: system should be pleasant to use

18. Examples • Novice PC users can change the ink cartridge in their color inkjet printer in less than 5 minutes using the user manual • Users will be able to install and configure the default application in less than 15 minutes • Users should rate the product as either “easy to use” or “very easy to use” (4 or higher on a 5 point scale)

19. Considerations • A single usability objective may result in multiple design decisions • Emphasizes importance of having cross-functional team • Focus on objective that are most important to the success of the product

20. Test plan • Explains what you are testing and how you will conduct the test • Do not skip this step • Important to have a plan that everyone can agree to and understand • Helps communicate your intentions to others in the organization • Do not overdo this step

21. Test plan • Purpose • Why are you conducting the test? • Problem Statement/Test Objectives • What usability objectives are you testing? • User Profile • Who are your users and their defining characteristics?

22. Test plan • Method • Details how you will run the test • Task List • List of the tasks to be completed, projected times for completion, required state of system to test tasks • Test Environment/Equipment • List what you need to conduct the test

23. Test plan • Test Team Roles • Define what each person will do • Evaluation Measures • What data you will collect through the test • Conclusion • What will happen after the test - reports, design meetings, presentations

24. Pre-test questionnaire • Ensure that participant meets user profile • Gather additional detail about the participants’ knowledge and experience

25. Post-task questionnaire • Asks questions about a task as soon as it is completed • Captures feedback when it is fresh • Can help measure change in attitude toward the task • Keep it short

26. Post-test questionnaire • Measure satisfaction and reaction to the product and the overall experience • Ask participants to elaborate on a few critical areas/issues

27. Training the team • Make sure everyone understands the usability objectives and the testing procedure • Define everyone’s roles and discuss how they should behave during the test

28. Conducting a pilot test • Conduct a practice test using the complete team, all prepared materials, and the actual facilities • Make sure at least one person observes the test to make recommendations • Refine any of the materials or facilities as necessary

29. Selecting test participants • Number of participants depends on the number of user groups (differentiating characteristics) • Target 3-5 users from each user group • Best if users come from the real user population rather than internally

30. Observers • Not actively involved in conducting the test, but can benefit from observing • Must not comment or react to the participant’s actions or comments • Should be separated from the actual room where the testing is going on • Should be removed if they cannot avoid feeling defensive or being intrusive

31. Data logger • Be familiar with any data logging forms • Establish standard codes ahead of time for recording events and comments • Remain open to unforeseen directions • Record user comments that summarize or emphasize good and bad points • Keep track of time

32. Facilitator • Main contact with the participants • Conducts the briefing and debriefing • Only person to talk to the participants during the test

33. Briefing • Thank them for participating in the product evaluation • Assure users that they are not being tested; instead, they are helping you evaluate the product • Explain to them what will happen during the test; they will be especially interested in any recording devices and observers

34. Briefing • Ask them to sign a consent form and a non-disclosure agreement • Encourage them to think out loud so you can record their actions and the reasons behind their actions • Make sure you explain what they should do if they have questions or experience problems

35. During the test • Be sensitive to excessive frustration • Be ready to handle unplanned situations • Remind participants to talk out loud • Look and listen for the unexpected • Avoid intervening unless necessary; try using questions to redirect rather than take over as the expert

36. Debriefing • Get users to elaborate on significant events or comments • Ask for reasons to specific behaviors from the test • Watch the video tape with them and ask them to explain what was going on in their mind

37. Summary • Build a cross-functional team to: • take advantage of different skills • get “buy in” from different areas • build awareness of usability testing benefits • When you test: • Set measurable objectives • Test with the right people • Be prepared for the test • Share your results

38. Questions? For more information about this presentation and other training seminars, visit www.userfirst.net. This presentation is at www.userfirst.net/teamusability.ppt. Scott DeLoach Founding Partner, User First Services, Inc. Certified RoboHELP Instructor and Consultant 404.520.0003 scott@userfirst.net