Designing Questionnaires

1. Designing Questionnaires Dr. Roger Harris Visiting Professor Research and Innovation Management Centre (RIMC) and Institute of Social Informatics and Technological Innovations (ISITI) roger.harris@rogharris.org http://www.rogharris.org/ @HarrisrwhRoger September 2013

2. 1. Survey Research Dr. Roger Harris Visiting Professor Institute of Social Informatics and Technological Innovation roger.harris@rogharris.org http://www.rogharris.org/ @HarrisrwhRoger September 2013

3. Survey questionnaires • If the researcher is looking to test and quantify hypotheses and the data is to be analysed statistically, a formal standardised questionnaire is required. • If the data to be collected is qualitative or is not to be statistically evaluated, it may be that no formal questionnaire is needed • The heart of a survey is its questionnaire; the main means of collective quantitative primary data • Survey results depend crucially on the questionnaire • Best practices stem from experience and methodological research

4. Questionnaires • A questionnaire is a formatted set of questions for obtaining information from respondents. • The objective of questionnaire design is to translate the researcher’s information needs into a set of specific questions that respondents are able and willing to answer. • Questionnaires collect quantitative data in a standardised way so that the data is internally consistent and coherent for analysis. • There is no scientific principle to guarantee an optimal or ideal questionnaire.

5. Survey implementation choices • Direct interview • costly; slow; high quality data • Telephone interview • Costly; needs expertise. • World Wide Web • Efficient; cheap; limited range of question types; hard to apply full range of implementation procedures; respondents need access. • Self-administered by Mail • Well studied; moderately expensive; can do well • Self-administered by eMail • Cheaper; efficient; respondents need access • Multi-method • Complex.

6. Quiz 1: True or False? A questionnaire can be used to test and quantify a hypothesis True Survey results depend on questionnaires True There is a scientific method to design questionnaires False, best practices stem from experience and methodological research The objective of questionnaire design is to translate the researcher’s information needs into a set of specific questions True Direct interviews are cheap to conduct False Telephone interviews are costly True

7. The 5 most important elements for achieving high response rates • Respondent-friendly questionnaire • The 4 contacts; • Pre-notice invitation • The questionnaire • Thank you • (Replacement questionnaire reminder) • Final outcome notification • For mail – return stamped envelopes • Personalised correspondence • Prepaid token incentive

8. Three types of survey information • Dependent variables • Information we are primarily interested in • Independent variables • Information which might explain the dependent variables • Confounding variables • Other factors related to both dependent and independent factors which may distort the results and have to be adjusted for

9. Characteristics of survey questionnaires • Prescribed wording and order of questions, • to ensure that each respondent receives the same stimuli • Prescribed definitions or explanations for each question, • to ensure interviewers handle questions consistently and can answer respondents' requests for clarification if they occur • Prescribed response format, • to enable rapid completion of the questionnaire during the interviewing process.

10. Quiz 2: True or False? • Questionnaires should be respondent-friendly True • There are 5 contacts with respondents during a questionnaire survey False; there are 4 • The three types of variables in a survey are; dependent, independent and confusing False, they are dependent, independent and confounding • Dependent variables explain the independent variables False, independent variables contain information that might explain the dependent variables • Prescribed wording and order of questions enables rapid completion of the questionnaire during the interviewing process. False. Prescribed wording and order of questions, to ensure that each respondent receives the same stimuli

11. However • Given the same task and the same hypotheses, six different people will come up with six different questionnaires that differ widely in their choice of questions, line of questioning, use of open-ended questions and length. • There are no hard-and-fast rules about how to design a questionnaire, but there are a number of points that can be borne in mind

12. A well-designed questionnaire should : • Meet the research objectives • Obtain the most complete and accurate information possible • Ensure that respondents fully understand the questions • Be organised and worded to encourage respondents to provide accurate, unbiased and complete information • Be easy for the interviewer to record the answer • Keep the interview brief and to the point

13. How people answer questions; ideally • First, they interpret the question and deduce its intent, next , • They must search their memories for relevant information, and, • Then integrate whatever information comes to mind into a single judgment. • Finally, they must translate the judgment into a response, by selecting one of the alternatives offered by the question. This process is known as optimizing

14. How people may answer questions; • By automatic compliance; with no intrinsic motivation to make the answers of high quality • They may become fatigued, disinterested, or distracted and may be less thorough in comprehension, retrieval, judgment, and response selection • Instead of attempting the most accurate answers, respondents settle for merely satisfactory answers • Respondents may interpret each question superficially and select what they believe will appear to be a reasonable answer This process is known as satisficing

15. Minimising satisificing • The likelihood of satisficing is thought to be determined by three major factors: • task difficulty, • question-specific attributes, e.g., the difficulty of interpreting a question and of retrieving and manipulating the requested information • attributes of the questionnaire’s administration, e.g., the pace at which an interviewer reads the questions and the presence of distracting events • respondent ability, • the extent to which respondents are adept at performing complex mental operations, practiced at thinking about the topic of a particular question, and equipped with pre-formulated judgments on the issue in question • respondent motivation • Motivation is influenced by need for cognition, the degree to which the topic of a question is personally important, beliefs about whether the survey will have useful consequences. • Efforts to minimize task difficulty and maximize respondent motivation are likely to pay off by minimizing satisficing and maximizing optimising.

16. Remedies for satisficing • Minimise task difficulty • Minimise the number of words in questions • Avoid double-barreled questions • Decompose questions when needed • Use ratings not rankings • Label response options • Minimise ‘response effects’ • Avoid blocks of ratings on the same scale (prevents ‘straight lining’) • Avoid ‘no response’ options • Avoid ‘agree/disagree’, ‘yes/no’, ‘true/false’ questions • Maximise motivation • Describe purpose of the survey • Provide instructions to think carefully • Include random probes (“give your reason”) • Keep surveys short • Put important questions early.

17. Quiz 3: True or False? • Given the same task and the same hypotheses, six different people will come up with the same questionnaire False • There are no hard-and-fast rules about how to design a questionnaire True • The 4 steps in respondent optimising are; interpret, search, integrate and translate True • Satisficing includes when respondents interpret each question superficially True • The likelihood of satisficing is thought to be determined by 3 factors; task difficulty, respondent ability and the weather. False. The likelihood of satisficing is thought to be determined by 3 factors; task difficulty, respondent ability and respondent motivation

18. Example of a survey • Adoption rates of personal computers varies among people • Knowledge workers • Computer anxiety • “Respondents with higher levels of computer anxiety will be less likely to use personal computers” • The Computer Anxiety Rating Scale with 19 items; able to learn, confidence, apprehension, etc. • “I feel insecure about my ability to interpret a computer printout “ “I am confident that I can learn computer skills “ • Review the literature to define the research question • Define the population to be surveyed • Isolate the variables of interest • Organise the variables into hypotheses that reflect the research question • Operationalise the variables into tangible indicators • Develop questions that will enable measurement of the indicators • Formulate a questionnaire • Carry out the survey • Analyse the data

19. 2. Purpose and Types of Questionnaires Dr. Roger Harris Visiting Professor Institute of Social Informatics and Technological Innovation roger.harris@rogharris.org http://www.rogharris.org/ @HarrisrwhRoger September 2013

20. Approaches • Structured • All respondents provide answers to a set of questions on a standardised questionnaire. Results can be compared to each other and easily summarised • Unstructured • Allows the respondent to talk about whatever aspects of the survey topic they wish. Best suited for qualitative data • Semi-structured • Mixes both. Uses qualitative data to supplement quantitative.

21. Administering questionnaires • Self administered questionnaires may be sent by post, email, or electronically online. • Advantages • Cheap and easy to administer. • Preserve confidentiality. • Can be completed at respondent’s convenience. • Can be administered in a standard manner. • Read out by interviewers. Questionnaires may be by telephone or face to face. • Advantages • Allow participation by illiterate people. • Allow clarification of ambiguity. • More reliable response rate • The method depends on who the respondents are. For example, university lecturers may be more appropriately surveyed by email; older people by telephone interviews; train passengers by face to face interviews.

22. Comparisons of data collection modes

23. 3. Asking Questions Dr. Roger Harris Visiting Professor Institute of Social Informatics and Technological Innovation roger.harris@rogharris.org http://www.rogharris.org/ @HarrisrwhRoger September 2013

24. Common Wisdom • Use simple, familiar words • Avoid technical terms, jargon, and slang • Use simple syntax • Avoid words with ambiguous meanings • Use wording that all respondents will interpret in the same way • Strive for wording that is specific and concrete as opposed to general and abstract • Make response options exhaustive and mutually exclusive • Avoid leading or loaded questions that push respondents toward an answer • Ask about one thing at a time • Avoid double-barreled questions (if there’s an ‘and’ in the question, there are probably two questions • Avoid questions with single or double negations.

25. Optimize question order • Early questions should be easy and pleasant to answer, and should build rapport between the respondent and the researcher. • Questions at the very beginning of a questionnaire should explicitly address the topic of the survey, as it was described to the respondent prior to the interview. • Questions on the same topic should be grouped together. • Questions on the same topic should proceed from general to specific. • Questions on sensitive topics that might make respondents uncomfortable should be placed at the end of the questionnaire. • Filter (branching) questions should be included, to avoid asking respondents questions that do not apply to them.

26. Open versus closed questions • Open questions permit respondents to answer in their own words • Closed questions require respondents to select an answer from a set of choices • The vast majority of survey questions are closed • Closed questions must be grouped into a relatively small number of categories • This requires the development of a coding scheme

27. Open versus closed questions • Open questions are usually preferable for measuring quantities, e.g. age, number of doctor visits, hours devoted to housework, • Closed quantity categories for this type of open question (e.g., less than 1 h, 1–3 h, more than 3 h) can produce errors

28. Comparison of open and closed Open questions Closed questions Advantages Easy and quick to fill in, less reliance on memory Can specify answer categories most suitable for the purposes Minimize discrimination against the less literate (self administered questionnaire) or the less articulate (interview questionnaire) Easy to code, record, and analyze results quantitatively Easy to report results Disadvantages  Do not allow the respondent to give a different response to those suggested.  Suggest answers that respondents may not have considered before. Advantages • Allows exploration of the range of possible themes arising from an issue • Respondent is asked to give a reply to a question in his/her own words • Can be used even if a comprehensive range of alternative choices cannot be compiled • Reveal the issues which are most important to the respondent • Respondents can 'qualify' their answers or emphasise the strength of their opinions Disadvantages • Respondents may find it difficult to 'articulate' their responses •  Respondents may not give a full answer if they may forget to mention important points • Verbatim comments have to be interpreted, coded and reduced to manageable categories

29. Categorical judgments • Such as the ‘‘what is most important problem’’ • Closed categorical question can be used only if its answer choices are comprehensive. • Usual to include other category. • Respondents tend to restrict their answers to the substantive choices that are explicitly offered.

30. Open response-option questions • Both open-ended and also include specific response-options as well. For example, • What features of this implement do you like? •  Performance •  Quality •  Price •  Weight •  Others mentioned: • Eliminates the disadvantages

31. Types of closed questions format • Choice of categories, for example, What is your marital status? • [ ] Single • [ ] Married • [ ] Divorced • [ ] Widowed • Likert style scale, for example, Statistics is an interesting subject • [ ] Strongly disagree • [ ] Disagree • [ ] Cannot decide • [ ] Agree Strongly • [ ] Agree • Differential scales, for example, How would you rate the presentation? Extremely interesting [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] Extremely dull • It may also be diagrammatic; • Extremely interesting Extremely dull

32. Types of closed questions format • Checklists, for example, Circle the clinical specialties you are particularly interested in: General medicine Obstetrics and gynecology General surgery Orthopaedics Ophthalmology Accident and emergency Paediatrics General practice • Ranking, for example, Please rank your interests in the following specialties (1= most interesting, 8=least interesting) • [ ] General medicine • [ ] Obstetrics and gynecology • [ ] General surgery • [ ] Orthopaedics • [ ] Ophthalmology • [ ] Accident and emergency • [ ] Paediatrics • [ ] General practice

33. Question checklist • Is this question sufficient to generate the required information? • Can the respondent answer the question correctly? • Are there any external events that might bias response to the question? • Do the words have the same meaning to all respondents? • Are any of the words or phrases loaded or leading in any way? • Are there any implied alternatives within the question? • Will the question be understood by the type of individual to be interviewed? • Is there any ambiguity in my questions? • Are any words or phrases vague? • Are any questions too personal or of a potentially embarrassing nature? • Do questions rely on feats of memory?

34. Put questions into a meaningful order and format • Opening questions • Opening question easy and pleasant to answer • Not threatening • Questions of special importance should be earlier • Question flow • Flow in some kind of logical order • One leads easily and naturally to the next. • Questions on one subject should be grouped together • Question variety • Vary the respondent's task from time to time • An open-ended question here and there (even if it is not analysed) • Questions involving showing cards/pictures to respondents can help vary the pace and increase interest.

35. The effect of one word

36. Balanced questions • Do you support India taking military action against Pakistan? • Do you support or oppose India taking military action against Pakistan? • If you heard that candidate X supported a higher minimum wage, would that make you more likely to vote for her? • If you heard that candidate X supported a higher minimum wage, would that make you more likely to vote for her, less likely to vote for her, or wouldn’t it make much difference? • Did you happen vote in the presidential election in 2010? • Did you happen vote in the presidential election in 2010, or not ?

37. “Don’t know” – offered or volunteered

38. Using scales to get to “the most”

39. How questions can affect one another • Funnelling. Ask general questions before specific priority questions in order to obtain unbiased responses. • Pay attention to how one question might influence answers to the next: • In considering whether or not to vote for a political candidate, how important is the candidate’s position on HIV/AIDS in making your choice? • What do you think are the two most important health care issues for the government to address?

40. 4. How to use Scales Dr. Roger Harris Visiting Professor Institute of Social Informatics and Technological Innovation roger.harris@rogharris.org http://www.rogharris.org/ @HarrisrwhRoger September 2013

41. Types of scales for closed questions • The four levels of measurement: • Nominal • Ordinal • Interval, and • Ratio

42. Nominal scales • Simplest form of measurement scale • Classifies individuals, companies, products, brands or other entities into categories where no order is implied. • Often referred to as a categorical scale. • It is a system of classification and does not place the entity along a continuum. • Involves a simply count of the frequency of the cases assigned to the various categories • Example of a nominal scale • Which of the following food items do you tend to buy at least once per month? (Please tick) • Okra • Palm Oil • Milled Rice • Peppers • Prawns • Pasteurized milk • The numbers have no arithmetic properties and act only as labels.

43. Ordinal scales • Involve the ranking of individuals, attitudes or items along the continuum of the characteristic being scaled. • For example, if a researcher asked farmers to rank 5 brands of pesticide in order of preference. • Example of an ordinal scale used to determine farmers' preferences among 5 brands of pesticide. • Brand Order of preference • Rambo • R.I.P. • Killalot • D.O.A. • Bugdeath • Provides the order of preference but nothing about how much more one brand is preferred to another - no information about the interval between any two brands.

44. Interval scales • The interval scale has equal units of measurement, thus making it possible to interpret not only the order of scale scores but also the distance between them. • Allows the use of the arithmetic mean as the measure of average. • Two respondents with scale positions 1 and 2 are as far apart as two respondents with scale positions 4 and 5 • Cannot be said that a respondent with score 10 feels twice as strongly as one with score 5. • Temperature is interval scaled, being measured either in Centigrade or Fahrenheit. We cannot speak of 50°F being twice as hot as 25°F. No fixed origin. • Interval scales may be either numeric or semantic. • Most of the common statistical methods of analysis require only interval scales in order that they might be used.

45. Ratio scales • The highest level of measurement • Has the properties of an interval scale together with a fixed origin or zero point. • Examples; weights, lengths and times. • Permit the researcher to compare both differences in scores and the relative magnitude of scores. • For instance the difference between 5 and 10 minutes is the same as that between 10 and 15 minutes, and 10 minutes is twice as long as 5 minutes. • Virtually all statistical operations can be performed on ratio scales.

46. Likert-type scales • Most often uses 5 points; • “I would use this tool in my work” • [Strongly Disagree ] [Disagree] [Neither Agree Nor Disagree] [Agree] [Strongly Agree] • Semantic scale • “Was the product easy or difficult to use?” Very Difficult [ ] [ ] [ ] [ ] [ ] Very Easy • Scales might have 7or 11 points. • The points offered should cover the entire measurement continuum, leaving out no regions • Points must appear to be ordinal, progressing from one end of a continuum to the other • The meanings of adjacent points should not overlap. • Each respondent must have a relatively precise and stable understanding of the meaning of each point on the scale. • Most or all respondents must agree in their interpretations of the meanings of each scale point • Offering a neutral midpoint may encourage satisficing, or • Eliminating the midpoint will force people with no opinion to pick a point measurement. • Cross-sectional validity and test–retest reliability increases from 2- to 3- to 5-point scales but were equivalent thereafter for 7-, 9-, and 14-point scales.

47. Some labels that work Bipolar intervals Unipolar intervals Not at all Slightly Moderately Very Extremely Definitely won’t Probably won’t Might or might 7not Probably will Definitely will Never Sometimes About half the time Most of the time Always • Extremely bad • Moderately bad • Slightly bad • Neither good nor bad • Slightly good • Moderately good • Extremely good • Dislike a great deal • Moderately dislike • Dislike a little • Neither like nor dislike • Like a little • Moderately like • Like a great deal

48. Quiz 5: True or False? • There are three levels of measurement; Nominal, Ordinal, and Interval False. There are four levels of measurement; Nominal, Ordinal, Interval, and Ratio • Ordinal scales involve the ranking items along the continuum of the characteristic being scaled True. • In an interval scale, two respondents with scale positions 1 and 2 are as far apart as two respondents with scale positions 4 and 5 True. • Interval scales can only be numeric False. Interval scales may be either numeric or semantic • A ratio scale has the properties of an interval scale together with a fixed origin or zero point True

49. 5. Questionnaire Appearance Dr. Roger Harris Visiting Professor Institute of Social Informatics and Technological Innovation roger.harris@rogharris.org http://www.rogharris.org/ @HarrisrwhRoger September 2013

50. Physical appearance • The physical appearance of a questionnaire can have a significant effect upon both the quantity and quality of data obtained • Ill-designed questionnaires can give an impression of complexity, and too big a time commitment • Unnecessarily confusing layouts making it more difficult for interviewers, or respondents in the case of self-completion questionnaires, to complete • Keep questionnaires as short as possible • In a rural situation an interview should not last longer then 30-45 minutes.