Asking Users and Experts

1. Asking Users and Experts by Xianghua Ding Hoang Minh Ho Dac

2. Outline • Asking Users: • Interviews • Questionnaires • Asking Experts: • Heuristic evaluation • Walkthroughs • Cognitive • Pluralistic

3. Asking Users • Interviews • Developing questions • Planning interviews • 4 Types of interviews • Data analysis and interpretation • Questionnaires

4. Developing Questions • Avoid long questions • Avoid compound sentences • “How do you like this cell phone compared with previous ones that you have owned?” • “How do you like this cell phone? Have you owned other cell phones? If so, how do you like it?” • Avoid using jargon • Avoid leading questions • “Why do you like this style of cell phone?” • Be alert to unconscious bias

5. Planning an Interview - Steps • Introduction • To introduce himself, explain the purpose and get consent • Warmup session • Using easy, non-threatening questions • Main session • To ask prepared questions from easy to difficult. • Cool-off period • a few easy questions • Closing session • To thank the interviewee and to clear up the scene • Make the interview as pleasantas possible • The golden rule is to be professional

6. Planning an Interview - Advice • Dressing • In a similar way to the interviewee if possible • Prepare a consent form and ask the interviewee to sign in • Make the equipment work • Make sure your recorder works and know how to use it in advance. • Record answers exactly

7. 4 Types of Interviews • Unstructured interviews • Structured interviews • Semi-structured interviews • Group interviews

8. Unstructured Interviews • Both interviewer and interviewee have control and can steer the topic • More like a conversation • Focus on particular topic but go into depth • Questions are open • No predetermined content and format • Interviewee is free to answer questions as fully or as briefly as she wished • Need a plan to make sure the main things to be covered

9. Unstructured Interviews- Advantage & Disadvantage Disadvantages • Time consuming • Ethical issues • Impossible to replicate the process • Difficult to analyze all data Advantages • Generate rich data

10. Structured Interviews • The interviewer has the most control • Pose predetermined , closed questions • The study is standardized • The same questions are used with each participants • Useful when study goal’s are clear and specific questions can be identified

11. Semi-structured Interviews • Combines features of structured and unstructured interview • Use both closed and open questions • Has a basic script for guidance • so the same topics are covered • Probe are device to get more information • “Do you want to tell me anything else?” • Be aware not to preempt an answer • “You seem to like this color” • Accommodate silence • Prompt the person to help her along

12. Semi-structured Interviews -An example • Which websites do you visit most frequently? • <Answer several but stresses that se prefers uci.com> • And why do you like it? • <Answer> • Tell me more about X? • <Answer> • Anything else? • <Answer> • Thanks. Are there any reasons that you haven’t mentioned?

13. Group Interviews • Involve a small group guided by a interviewer to facilitate discussion • Focus group • Normally 3-10 people are involved • Participants are representative of a certain type of users; they normally share certain kind of characteristics • Allows diverse and sensitive issues to be raised

14. Group Interviews –Advantage & Disadvantage Advantages • Method is readily understood • Findings appear believable • Low-cost • Quick results • Easily be scaled Disadvantages • Facilitators need to be skillful • Difficult to get people together in a suitable location and time

15. Data Analysis & Interpretation • Quantitatively • For structured interviews • Qualitatively • For unstructured Interviews • A coding form may be developed • Comments may be clustered along themes and anonymous quotes used to illustrate points of interest • Tools such as NUDIST

16. Asking Users: Questionnaires • An alternative technique for getting users’ opinions • Can have closed and open questions Strengths: • Distributed to a large number of people • Provide evidence of a wide general opinion

17. Guidelines for Designing Questionnaires • Make questions and instructions clear • If possible, ask closed questions and offer a range of answers • Include a “no-opinion” for questions that seek opinions • General questions should precede specific ones • Group related items • Specify age as a range • Different versions for different population • Balance between white space and compactness • If scales are used, the range should not overlap • The ordering of scales should be intuitive and consistent

18. Example of Poorly Designed Questionnaires 1. State your age in years 2. How long have you worked here? (check one only) 3. How long have you use the Internet? (Check one only) 4. Do you use the Web to: 5. How useful is the Internet to you? Purchase goods Send email Visit chatrooms Find information 1 yr 2 yrs 3 yrs > 3 yrs < 1 yr 1-3 yrs 3-5 yrs > 5 yrs

19. Question and Response Format • Checkbox • Used for demographic or background data • Respondents check an appropriate box or circle a response • Ranges • used for getting opinions • 2 types 1. Likert Scales 2. Semantic Differential Scales

20. Example of Likert Scales Instruction: In the following questions, 1 represents strongly agree and 5 represents strongly disagree. Please check only one. • The company website is helpful: 1 2 3 4 5 • The website color is annoying: 1 2 3 4 5  Should we mix positive questions with negative questions?  What is the best rating scale ? (e.g. odd like 1-3, 1-5,… or even)

21. Example of Semantic Differential Scales Instruction: for each pair of adjectives, place a cross at the point between them that reflects the extent to which you believe the adjectives describe the home page. You should place only one cross between the marks on each line. • Attractive |_|_|_|_|_|_|_| Ugly • Clear |_|_|_|_|_|_|_| Confusing • Helpful |_|_|_|_|_|_|_| Unhelpful  How to calculate the total score?

22. Administering Questionnaires Two important issues 1. How to reach a representative sample of people 2. How to ensure a reasonable response rate • With small number fewer than 20, 100% is often achieved • With larger populations, 40% return is generally acceptable • Solutions • Tell people it is OK to complete just a part • Include stamped, self-addressed envelope • Explained why you need the questionnaires • Assure anonymity • Contact users • Offer incentives

23. Online questionnaires • Email • Can be targeted to specific users • Quick response • Limited to text • Web-based • Flexible graphical design • Errors could be corrected easily • Immediate data validation • Less time for data analysis • Low cost for copying and postage • Have random samples of respondents • Response rate may be lower than paper form

24. Steps to develop web-based questionnaires • Devise the questionnaire on paper, following the guidelines • Identify a random sample of population. Avoid biased or convenience sampling • Turning the paper questionnaire into a web-based version • error-free interactive • Accessible and readable from all online users • Identification information handled confidentially • User-test before distributing

25. Examples of Web-based Questionnaires • What do you think about the questionnaires at this website? Are they good or bad? Why? • http://www.perseusdevelopment.com/surveytips/samplesurveys.html

26. Analyzing Questionnaire Data • Display data graphically (e.g. bar charts) • Often simple statistics are needed (number of participants, percentage of responses…) • Identify any trends, patterns or relationship between responses

27. Example – Statistics Table

28. Example – Pie Chart

29. Example (cont.)

30. Asking Expert • Heuristic evaluation • Introduction • Core heuristics • Doing heuristic evaluation • Heuristic evaluation for web sites • An example • Guidelines of website • Walk through • Cognitive Walkthrough • Pluralistic walkthrough

31. Heuristic Evaluation –Introduction • An inspection technique in which experts evaluate whether user-interface elements conform to a set of heuristics • Closely related to design guidelines • Different sets of heuristics for different products

32. Core Heuristics • Nielsen’s • Visibility of system status • Match between the system and the real world • User control and freedon • Consistency and standards • Help user recognized, diagnose, and recover from errors • Error prevention • Recognition rather than recall • Flexibility and efficiency to use • Aesthetic and minimalist design • Help and documentation

33. Doing Heuristic Evaluation • Briefing session • Experts are told what to do • A prepared script is useful as a guide • Ensure each person receives the same briefing • Evaluation Period • Experts independently inspecting the product, using heuristics for guidance. • At lease 2 passes • Give a feel of the flow of the interaction and the product scope • Focus on specific interface elements • Debriefing session • Discuss their findings, prioritize the problems, and suggest the solutions

34. Heuristic Evaluation for Websites – an Example • http://www.lib.uci.edu/

35. Heuristic Evaluation for Websites – an Example • Heuristics tailored from Nielsen’s original set • Internal consistency • Is the logo, format, text , font or usage of terms consistent? • Minimizing the user’s memory load • Layout • Is it compact? Is the page layout meaningful? Is there too much text on the page? • ……..

36. Heuristic Evaluation for Websites – an Example • Findings about the websites • The formatting of pages and presentation of logos are consistent of website • Some forms require users to recall instead of recognition • http://www.lib.uci.edu/services/workshops/isform.html • The layout is kind of complicated, kind of too much text • http://www.lib.uci.edu/online/ebooks.html

37. Heuristic Evaluation for Websites - Guidelines • Navigation • Avoid orphan pages that are not connected to the home page, which lead users into dead ends • Avoid long pages with excessive white space that force scrolling • Provide navigation support • Avoid non-standard link colors • Provide consistent look and feel • Access • Avoid complex URLs • Avoid long download time that annoys users • Information design

38. Walkthroughs • An alternative to heuristic evaluation • To predict users’ problems without doing user testing • How ? - Walk through a task with the system and recognize usability problems • 2 types: 1. Cognitive walkthroughs • No user participation 2. Pluralistic walkthroughs • Users, experts, specialists, designers, developers… are involved

39. Steps in Cognitive Walkthroughs • Users’characteristics are identified • The walkthrough team come together for task analysis • The walkthrough team go through each task, answering the three questions: • Will users know what to do? • Will users see how to do it? • Will users understand from the feedback whether their action are correct or not? 4.Record what cause problems, why and how serious they are to users 5. Revise the design to fix the problems

40. An Example of Cognitive Walkthroughs What to Evaluate: Java Home Page Task: to find the tutorial for Java 3D Typical users: students, developers Go through each step to complete the task • At Java Home Page • Will users know what to do ? – yes, to search the Tutorial Section • Will users see how to do it ? - yes, click on the left submenu Tutorial • Will users understand the feedback ? – yes, it leads to the List of Tutorials • At The List of Tutorials …

41. Steps in Pluralistic Walkthroughs • Choose a task to evaluate. Take a series of screenshots to complete that task. 2. Each member of the team looks at the screen pictures and writes down the sequence of actions they would take to move from one screen to another 3. First, users present their suggest of actions Next, experts present their findings Last, developers comment 4. Go back to step 1 with another task

42. An Example of Pluralistic Walkthroughs Suppose you are a member in the walkthrough team. Purpose: to evaluate the web site of Sea World Step 1 • Chosen task: to find the location and open hours • Scenarios: There are 2 screens in the path to find Sea World’s location and open hours

43. 1 2 • Step 2: • You are shown the above 2 screens in the path to complete the task • Can you say what action will lead you from the first screen to the second screen? (don’t consult with other members in the team) • Step 3: • Do other members in the team agree with you? • If yes, what do you think about the interface? If no?

44. Cognitive vs. Pluralistic Pluralistic Cognitive • Focus on users’ problems in detail • No user involved in evaluation. • Do not need a working prototype • Time-consuming • Has narrow focus, only useful for certain types of systems • Strong focus on users’ tasks • Multidisciplinary evaluators with user participation in evaluation • Difficult to arrange time and location for the walkthrough team • Time-consuming • Explore only a limited number of tasks

45. Thank you!