Interviews and Questionnaires

1. Interviews and Questionnaires a.k.a. How to talk to your users

2. Agenda • Questions • Interviewing techniques • Questionnaire design • Evaluation Plan discussions

3. First: a little exercise • Interview each other about instant messaging use • 4 or 5 questions, 3 minutes • Person 1: the silent type • Person 2: the going off track type

4. Today’s focus is asking people about stuff… • Interviews • Questionnaires • Assessment of that data

5. Interviews & Questionnaires • Subjective view of participants • Quantitative – very structured • Questionnaires • often quantitative, but not entirely • Structured Interviews • Strict set of questions, deviation would compromise study • Qualitative – less or no structure • Semi-structured interviews • Some deviation encouraged • Unstructured interviews • i.e. the ethnographic interview • Little guide, very explorative

6. Interviews • Potentially lots of detail • can vary questions as needed • Inexpensive • Time consuming to perform and analyze • Some interpretation required • Subject to interviewer biases

7. Questionnaires • Expensive to create • …but cheap to administer • Easier to get quantifiable results • Can gather info from many more people • Protects participant identity • Only as good as the questions asked

8. Structured Interviews • More similar to questionnaires • Require a lot of training for any hope at inter-interviewer reliability • But that means that they tend to give much more repeatable results

9. Unstructured Interviews • Have a plan, but keep interview open to different directions • Get participant to open up and express themselves in their terms and at own pace • Create interpretations with users • Be sure to use their terminology • Take lots of time, but learn a lot as well

10. Semi-Structured Interviews • Predetermine data of interest - know why you are asking questions - don’t waste time • Plan for effective question types • How do you perform task x? • Why do you perform task x? • Under what conditions do you perform task x? • What do you do before you perform…? • What information do you need to…? • Whom do you need to communicate with to …? • What do you use to…? • What happens after you…? • See Gordon & Gill, 1992; Graesser, Lang, & Elofson, 1987

11. Asking Questions • Understand your goals • Consider the ordering of the questions • Avoid complex/long/multiple questions • Avoid jargon; talk in participant’s language • Be careful of stereotypes, biases

12. Clarity is important Questions must be clear, succinct, and unambiguous How much time have you spent reading news on the Web recently? • Some • A lot • Every day • Rarely • Etc. • None • 0 to 5 hours • 6 to 10 hours • 11 to 20 hours • More than 20 hours

13. Avoid question bias Leading questions unnecessarily force certain answers. Do you think parking on campus can be made easier? What is your overall impression of… 1.Superb 2.Excellent 3.Great 4.Not so Great

14. Be aware of connotations Do you agree with the NFL owner’s decision to oppose the referee’s pay request? Do you agree with the NFL owner’s decision in regards to the referee’s pay demand? Do you agree with the NFL owner’s decision in regards to the referee’s suggested pay?

15. Leading questions • People want to do well, give you what you are looking for • Be aware of your own expectations before creating questions and while interviewing • Use value neutral terms What do you like about this system? Vs. Tell me what you thought about this system.

16. Avoid hypotheticals • Avoid gathering information on uninformed opinions • Subjects should not be asked to consider something they’ve never thought about (or know or understand) Would a device aimed to make cooking easier help you?

17. Handle personal info carefully • Ask questions subjects would not mind answering honestly. • What is your age? • What is your waist size? • If subjects are uncomfortable, you will lose their trust • Ask only what you really need to know

18. What’s wrong with this picture? • How much easier is it to use this email client than Outlook? • I see you choose to use your keyboard shortcuts more than the mouse. Is that faster for you? • Your choice of red is different than any other user we saw. Why did you do that?

19. Planning your interview: • Introduction • Warmup • Main session • Cool-off • Closing Record everything exactly in your participants’ languages (don’t forget to test your recording equipment)

20. The warmup or “grand tour” question • The first question helps set the tone for the interview • Familiarize the participant to talking • Encourage the participant that their true opinion does matter • Question should be • Easy to answer • But not answered easily • More than just a “yes” or “no” response • Examples: • Tell me about the work you do? • What made you buy the computer?

21. Prompts • “Nudge” a participant in a direction, or to get additional response • Silent: remain silent until they say more • Echo: repeat back and then ask “then what happens” etc. • Make agreeing sounds: you say “uh huh” and the other person continues • Tell Me More: could you tell me more about that? • Clarifying: summarize and ask for confirmation or clarification, often leads to new discussion

22. Contents of a survey • General/Background info • Demographic data • Also functions as a “warm up” • Correlate responses between groups • Objective questions • Open-ended/subjective

23. Background examples • Demographic data: • Age, gender • Task expertise • i.e. Have you ever worked in a restaurant? • Motivation • Frequency of use • How often do you… • Education/literacy • What training have you had in …?

24. Advantages Clarify alternatives Easily quantifiable Eliminate useless answer Disadvantages Must cover whole range All should be equally likely Don’t get interesting, “different” reactions Closed Format Restricting set of choices Quantifiable

25. Many forms of response • Dichotomous • Multiple Choice • Multiple Response • Rank/Match • Likert • Rating

26. Questionnaire Styles Rank from 1 - Very helpful 2 - Ambivalent 3 - Not helpful 0 - Unused Which word processingsystems do you use? LaTeX Word ___ Tutorial ___ On-line help ___ Documentation FrameMaker WordPerfect

27. Likert-type scale • Typical scale uses 5, 7 or 9 choices • Above that is hard to discern • Doing an odd number gives the neutral choice in the middle • You may not want to give a neutral option Characters on screen were: hard to read easy to read 1 2 3 4 5 6 7

28. What’s wrong with this picture? 1. What is your age? _______________ 2. How long have you used the internet? <1 year 1-3 years 3-5 years >5 years 3. How do you get information about courses? Email Web site Flyers Registration booklet Advisor Other students 4. How useful is the Internet in getting information about courses? ___________________________________________________________ ___________________________________________________________

29. On line questionnaires • Email or internet • Change checkboxes into dropdowns, etc • Take advantage of the technology – check input • Ensure its as accessible as paper (browser and email client compatibility) • Ensure confidentiality – how is this different from paper?

30. Free Web Survey Tools • Survey Monkey • http://www.surveymonkey.com • Survey Share • http://www.surveyshare.com/ • phpESP • http://phpesp.sourceforge.net • Open Source surveys using PHP.

31. Analyzing your quantitative data • “Code” open ended responses or interview questions to make quantitative • Categorize all responses • Look for trends in the data • Count, average, tabulate • Make charts, etc • Run statistical analysis • Use lo-fi methods (post-its, affinity diagrams, etc)

32. Analyzing qualitative data • Find interesting cases, responses • Look for patterns of responses • Use post-its, affinity diagrams, etc. • Look for any useful suggestions, improvements, explanations that help you improve your design • Gather illustrative quotes from users that demonstrate your conclusions

33. Evaluation discussion • Someone else should be able to pick up your plan and execute it. • Be as SPECIFIC as possible • What criteria are important? • What tasks EXACTLY? • What data? How will you record? • What questions will you ask?