Developing Questionnaires and Structured Interviews

1. Developing Questionnaires and Structured Interviews Alan C. Acock

2. Why learn how to do these? • Questionnaires and structured interviews are the primary way many researchers measure variables. • Give you a critical eye to evaluate the work of others. • You may have a great theory, review of literature, and research hypotheses. You may have the best research design possible. You still need to measure your variables and this is where questionnaires and structured interview enters the research process.

3. How to Develop a Questionnaire • Guideline 1 • Create a realistic list of all research questions and hypotheses you wish to address. • List all the variables you need for the study. • This provides a checklist that can be used to make sure you have items that address each research question or hypothesis.

4. Guideline 2Include demographic, socioeconomic, and geographic information. • You will need to control or adjust for variables such as • race • gender • age • income • education • marital status • family size • geographical information

5. List assumptions about participants • Language skills • What is their motivation to participate? • What are red flags for them? • Where might they misrepresent true beliefs? • What do they know? • What can’t they answer?

6. Questionnaires in other languages • Know your participants • Formal Spanish may not be ideal—mixed language • Translate to Spanish with appropriate adjustments. • Back translate to English using different translator. • Do practice interviews to find what is not clear—cognitive mapping.

7. Cognitive Mapping • Ask person like those you will interview. • Person reads the question. • What is the question asking? • Person reads the response options and picks one. • Why did you pick that option?

8. Order counts • Order should be fixed so all participants answer questions in same order. • Easy questions at the start. • Critical questions next in case somebody quits before the very end of the survey. • Hard questions that are not absolutely critical at the end. • Sequencing can change answers.

9. Develop skip map • Computer assisted interviews where a person’s answer to one item automatically routes them to the next appropriate item. • Written questionnaire limit skips. • Interviewer training is critical for skips. • Need to pretest with different types of people who will answer different sets of questions.

10. Develop data entry plan • Computer assisted interviews automatic—produce dataset ready to analyze. • Scanning in answers restricts format, but is accurate and quick. • Entering by hand is very time consuming and error prone. • Step A—put numbers for each question in right margin. • Step B—enter the data that is in the margins.

11. Types of questions—closed ended • A. strongly agree, B. agree, C disagree, D strongly disagree. • Check income category. Check years of education. Check age category. • Easy to enter data. • Easy to compare answers.

12. Types of questions—open ended • Participants own voice • Doesn’t impose researchers categorization • Confounded with language/verbal skills • Often skipped or trite answers—How is your marriage? Answer—OK, I guess. • Structured questionnaires don’t have follow-ups to draw out meaningful responses • Difficult to analyze

13. Vague/Double Meaning • Good questions involve one point or dimension. • Real bad: Has your spouse yelled at you or hit you in the last month? • Problematic: How happy is your marriage? • Happy regarding parenting • Miserable regarding sexual relationship • Great regarding equitable division of chores • Terrible in terms of career goals • Fantastic in terms of companionship • Be as specific and unidimensional as possible

14. Avoid value laden wording • Do you support the Tax Reform solution that will benefit so many people including those on a fixed income, the elderly, children, and the middle class? • Do you hate gays? Hate is likely to be avoided even if the person really does not support gays.

15. Mitigate threatening questions • Have you had unsafe sexual intercourse in the last month? • Some people have sex without using a condom. Did you happen to have sexual intercourse in the last 12 months without using a condom?

16. Formats: Grid Response set—Reverse order--Format is efficient.

17. Skip directions for self administered Teenagers 1. Are you male or female?  Male (go to question 4)  Female (go to question 2) 2. Have you menstruated?  Yes (go to question 3)  No (go to question 4) 3. How much pain, if any, do you have when you menstruate the first time?  None  A little  Quite a bit  Extreme 4. How many days of school did you miss in the last month? ____ Enter the number of days

18. Sample Questionnaire • The file: sample questionnaire.doc contains a sample of small part of a structure interview.

19. Evaluation of this structured interview—First questions on chores • Is participant able to give number of hours each household member spends on each task (first section)? • What about multitasking? • Would men be as good as women at estimating? • Diaries might be more useful

20. Evaluation of this structured interview—Depression scale • Standard measure; widely used. • Response set problem—not bother to even read last couple items—just check 1 or 7 for all of them. • Response set gives researchers false sense of reliability. • Between item b and c we could say “have an especially enjoyable experience”

21. Evaluation of this structured interview • Question 8. Is this threatening? • Does opening statement mitigate the treat? • Are respondents able to answer this question? • Correlations between parent reports about their children and the children’s reports are weak.

22. Evaluation of this structured interview • Question 11. Is yes or know what we need? • We lose possible variation. If we can get meaningful responses we need to maximize variation. • With some groups, maximizing variation may be counter productive. • Can distinguish between yes and no • Can make meaningful/reliable distinctions between other response options.