1 / 55

User Testing and Modeling

User Testing and Modeling. ICS 205 Chris Wesson Christina Wuerth November 14, 2003. User Testing. What is User Testing Measuring performance of typical users doing typical tasks Goal:

verity
Télécharger la présentation

User Testing and Modeling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. User Testing and Modeling ICS 205 Chris Wesson Christina Wuerth November 14, 2003

  2. User Testing • What is User Testing • Measuring performance of typical users doing typical tasks • Goal: • Obtain objective performance data showing how usable a system is in terms of usability goals (ease of use, learnability)

  3. User Testing • Applied form of experimentation • Tests whether product is usable by intended user group(s) • Measures/Records: • Time to complete typical tasks • Number and type of errors • Routes users take through tasks (Web) • Typically 6-12 users

  4. User Testing • Is part of Usability Testing: • Obsevations • Questionaires • Interviews • User Testing

  5. User Testing and Modeling • Doing User Testing • Experiments • Predictive Models

  6. Keys to Doing User Testing • Control testing conditions • Careful planning • Same conditions for each participant • Measurements correspond to test • Explicit assumptions • DECIDE framework

  7. Q: How Long Should User Testing Process Take? A: As little time as possible while still yielding useful information. On average most companies take more than 8 weeks for user testing Testing of small piece of well developed procedures can be done in less than 1 week if needed

  8. Things to Consider in Planning a User Test • What aspects of the product need to be improved? • Do the participants in the study represent actual users of the product? • What tasks should the participants perform? • What information will you observe during the tests? • How will you analyze the data you collect? • What will you do with this information once it is analyzed?

  9. User Test Planning Steps • Define goals and concerns • Decide who will participate • Recruit participants • Select and organize tasks to be tested • Create task scenarios • Decide how to measure usability • Prepare other materials for the tests (questionnaires, etc.) • Prepare the testing environment • Prepare the test team • Conduct pilot tests making changes where needed

  10. The Testing Team The Roles: Usability Specialists The usability specialists know what can be accomplished in a user tests and how a test should be planned. They should also understand the interface and potential problems users may have with it. Designers/Developers The designers and developers understand the product and what the user should be able to do with it. Technical Communicators Technical communicators know what aspects of the documentation should be tested and also see potential problems in how the product communicates to the users.

  11. More User Test Team Roles Trainers The trainers understand the problems that users will have when learning to use the product. Marketing Specialists Marketing specialists know who the users will be. Helpdesk/Customer Service Reps Helpdesk and customer service reps know what problems users have had with earlier versions of the software.

  12. Preparing for the Test 1. Schedule activities 2. Assign roles and responsibilities 3. Train team 4. Write out a test plan 5. Practice by running pilot tests Throughout the entire process remember to: - track progress - document all decisions

  13. DECIDE:User Testing Guideline • D etermine goals • E xplore questions • C hoose paradigm and techniques • I dentify practical issues • D eal with ethical issues • E valuate, analyze, present data

  14. Determine Goals, Explore Questions • User testing is best used for testing prototypes and working systems • Goals can be broad (how usable?) • Specific questions required to focus study (can task X be done in time Y?)

  15. Choose Paradigm and Techniques • User testing is part of Usability testing paradigm • Data can be recorded using: • Video • Interaction log • User satisfaction questionnaires • Interviews

  16. Identify Practical Issues:Design Typical Tasks • Create set of “completion” tasks (finding a website) • Choosing which tasks to test is critical • Tasks generally last 5-20 minutes • Tasks are often straightforward • Complex tasks (create design, solve problem) are okay • Start simple to build user confidence

  17. Identify Practical Issues:Select Typical Users • Must know users’ characteristics • Most important: previous experience • Use short questionnaire to identify users • Equal number of males and females

  18. Identify Practical Issues:Prepare Testing Conditions • Control testing environment • Minimize outside influences/noises that could distort results

  19. Identify Practical Issues:Plan How to Run Tests • Create schedule and scripts for running tests • Start with familiarization task • Contingency plan for spending too much time on a given task • Avoid long tasks and long testing procedure • Session < 1 hour

  20. Deal with Ethical Issues • Informed consent form • Point out any: • One-way mirrors • Video cameras • Interaction logging

  21. Evaluate, Analyze,and Present Data • Performance measures recorded from video, interaction logs • Only use simple statistics: • Maximum Minimum • Mean Standard Deviation • Allows evaluators to compare performance between systems and across tasks

  22. Establishing Goals and Concerns Step 1 of the planning phase is to define goals and concerns. Q: What is a goal? A: A quantitative usability aim that can be stated as a declarative sentence example: A user should be able to locate the print menu item in less than 5 seconds with less than 3 errors during first attempt. Q: What is a concern? A: A concerns are questions raised about the usability of the product – these are often raised while planning for the tests. example: While a user find the new error messages confusing?

  23. Establish Which Goals and Concerns will be tested. Not all goals and concerns can be tested simultaneously. Example of 2 conflicting concerns: 1. Will the user use feature X? 2. Will the user find X easy to use? These two questions can’t be answered at the same time instead you must prioritize.

  24. Using Concerns to Plan the User Test Use General concerns to decide who your participants should be Example: Will new users find this product easy to learn? Make sure you test with some beginning users. Use specific concerns to decide which tasks should be performed. Example: Will users be able to find the print command quickly? In less than 5 seconds? Have the users perform a task of printing.

  25. Sources of Goals and Concerns 1. Task Analysis and Quantitative usability goals Example: General: Menus should be easy to navigate Quantitative Goal: Users should find menu choice in less than 2 minutes with no more than 2 wrong choices when first need it. Task should be done with no errors in less than 1 minute after first attempt.

  26. Sources of Goals and Concerns 2. Timely Issues – Certain concerns arrive at different stages of development and should be resolved at that time. Example: 1) Two different design philosophies that you can’t decide between…plan a user test at start of the project to test prototypes of each 2) There has been previous testing earlier in the project, now you want to test a new feature before continuing development.

  27. Sources of Goals and Concerns 3. Heuristic Analysis and Expert Reviews a) Problems that have been predicted by heuristic analysis and expert reviews should be candidates for user tests. b) Problems predicted by designers, planners, developers, human factor specialists, technical communicators should be tested. c) Concerns found by reflecting on the product itself should be tested.

  28. Sources of Goals and Concerns 4. Previous Tests Example: iterative testing with rapid prototypes run several user tests with the same concerns Example: large scale user tests that raise future concerns retest with same concerns to make sure changes improved the problem

  29. Who Should be Participants? Participants must be like the people who will actually use the product. Develop a user profile then choose participants that fit the profile.

  30. Developing a User Profile Base profile on: General market research Analysis of customers of competitors’ products Focus group sessions Observing and interviewing prospective users

  31. Developing User Profile 1. Think about relevant characteristics a) those that all users share b) those that might make a difference among users example: shared characteristic: users will be undergraduate university students differences: users will have lots of computer experience users will have very little computer experience users will have registered for classes before users will have never registered for a class before

  32. Developing User Profile 2. Decide which characteristics matter most in establishing usability experience and motivation contribute more towards and individual’s understanding than education, income, age, etc. Good Factors to Consider 1. Work experience 2. General computer experience 3. Specific computer experience 4. Experience with this product 5. Experience with similar products

  33. Developing User Profile 3. Think broadly about who the users will be Sample questions to reflect on: 1. Not just those currently working, but what about new hires? What is the rate of turnover that the software will need to handle? 2. Don’t limit yourself to the current market…what about growth, who might want to use this product in the future? 3. What other areas of the corporation may what to adopt this software? 4. Reflect on differences within a category of users. Older users versus younger users? Users who have done the job longer versus users that more adaptable to computer environments?

  34. Developing a User Profile 1. Product Name: Class Registration Software 2. General Characteristics of User Population University Students 3. Characteristics of Users Relevant to the test Year in School Computer Experience Previous Experience Registering for Classes

  35. Developing a User Profile 4. Which Characteristics listed in 3 should all users in the test have in common & how will you define them? All users should be students currently enrolled in a university There will undergraduate and graduate students

  36. Developing a User Profile 5. Which characteristics listed in 3 will vary in the test and how will you define them? Computer Experience will vary. Novice = < 1 year experience Medium = > 1 year experience used < 5 days a week regularly Expert = > 1 year experience used > 5 days a week regularly

  37. Select Subgroups for a User Test Subgroups are people who share specific characteristics important to the user profile Divide groups by one characteristic at a time. Example: University Students Subgroup 1 Subgroup 2 Undergraduates Graduates Subgroup 1a Subgroup 2a Undergraduates Graduates Little computer experience Little computer experience Subgroup 1b Subgroup 2b Undergraduates Graduates Lots of computer experience Lots of computer experience

  38. Define Characteristics for Each Subgroup Define what you mean by characteristics Example novice = less than 1 year experience with this product medium = 1 year to 3 years experience with this product expert = more than 3 years experience with this product

  39. Qualify Characteristics of the Subgroups 1. The middle group is often omitted from the subgroups in user studies since most problems found here will be found in novice and expert studies 2. Select a range of participants in each subgroup. Example: Subgroup of novices with < 1 year experience Find users with 1 month, 6 months, and 11 months experience 3. Establish range of each subgroup set min and max for subgroup participants Example: Subgroup of experts with > 3 year experience min would be 3 years but perhaps you want max experience to be 5 years…you don’t want someone whose a hacker even though they may have lots of experience

  40. How Many Participants? Depends on: 1. How many subgroups you need 2. Time and money constraints 3. How important it is to compute statistically significant results * Most of the time all you will care about is inferential statistics

  41. How Many Participants? 6 – 12 participants is the typical choice With a minimum of 3 individuals in each subgroup. * Problems will show up across subgroups

  42. Making the Most of a limited Number of Participants 1. Decide which characteristics are the most important so that your subgroups will be most useful 2. Collect relevant information from participants via interviews or questionnaires to help account for differences that show up in results. Example: one novice user performs faster than all the others, but you find out that he has used a similar product before 3. Select people in subgroups representing the full range of the group

  43. Experiments • User testing based on scientific experimentation • Aim: test hypothesis that predicts a relationship between variables

  44. Variables • Independent: manipulated by researcher • Often multiple independent variables: • System version, User experience • Dependent: affected by independent variable • Common dependent variable: • Time to complete task, Number of errors

  45. Assigning Participants to Conditions • Three Experimental Designs: • Different participants for all conditions • Same participants for all conditions • Matched pairs of participants

  46. Matched Participants • Participants matched in pairs • Based on specific characteristics (expertise, gender, etc.) • Used when participants cannot perform in all conditions

  47. Design Comparison

  48. Data Collection • Data should measure users’ performance • Typical measures: • Response times • Number of errors • Time to complete task

  49. Data Analysis • Questions to ask: • Do data sets from two conditions look similar or different? • Any extreme atypical values? • If so, what does that mean? • Graph data to display differences

  50. Predictive Models • Provide measures of user performance without testing users • Useful when it is difficult to do user testing • Estimate efficiency of different systems on different tasks • Most common models: GOMS family: • GOMS Model, Keystroke Level Model

More Related