Developing Survey Items

1. Developing Survey Items Cari Brito Psychology Department October, 2003

2. Information Needs • Every item should be required for making decisions, choices, selections, or judgments. • Obtain all essential information • Obtain only what is directly applicable • Avoid items that only satisfy curiosity • Classify information items into 3 priority categories • High: essential or main reason for survey • Medium: substantial value for decisions or actions • Low: provide supporting info or background • List in order of priority or importance  can help reduce length

3. Basic Question Attributes • Focus – every Q should focus on a single, clear, specific issue or topic. • Poor: When do you usually go to class? • Better: What time do you ordinarily leave for your first class? • Brevity – the longer the Qs, the more difficult the response task. Short Qs are less subject to error • Poor: Can you tell me how many siblings you have, whether they are girls or boys, and how old they are? • Better: What is the age and sex of each of your siblings? • Clarity – meaning of Qs must be clear to all respondents • Poor: About how much of the storage space in your dorm room do you and your roommate use? • Better: What proportion of the storage space in your dorm room is used for your things and what is used for your roommate’s?

4. Expressing the Question • Use core vocabulary; words and phrases used in casual speech. • 1255 surveyed by NY’s American Museum of Natural History: • 77% interested in plants and trees, but only 39% interested in botany • 48% interested in fossils, but only 39% interested in paleontology • 42% interested in rocks and minerals, but 53% interested in geology • Grammar: Use simple sentences where possible; use complex sentences only when required; change long, dependent clauses to words or short phrases where possible • Poor: How do you work it out when you want one thing and your roommate wants another and you both feel very strongly about it? • Better: How do you settle disagreements with your roommate when you both have strong feelings about it?

5. Avoid Negatively Phrased Questions • 1992 survey: 20% of Americans doubted that the Holocaust had occurred, another 12% were not sure. • “As you know, the term Holocaust usually refers to the killing of millions of Jews in Nazi death camps during World War II. Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” (Roper Starch Worldwide) • “Do you doubt that the Holocaust happened or not?” (Gallup) • 9% doubted the truth of the Holocaust, another 4% were unsure. • Poor: Do you believe the university should not have the right to search your dorm room? • Better: Should the university have the right to search your dorm room?

6. Use Balanced Items • Don’t write items favoring one position or another • Poor: Do you support the use of animals in undergraduate laboratory courses? • Better: Do you support or oppose the use of animals in undergraduate laboratory courses? • Poor: Do you think Textbook Rental provides an excellent, low-cost service to students? • Better: What do you think about Textbook Rental?

7. Avoid Questions that are Too Demanding • They may confuse respondents • It’s probably too much to ask someone to rank-order 50 universities. • There are memory limits • Qs relying on memory should have clear time referent • Poor: How many times last year did you study at the library? • Better: How many times during the SP03 semester did you study at the library? • Don’t ask the impossible of someone’s memory • When asking about past behaviors, don’t make the interval too long  chances of memory errors increase. • Poor: How many drinks did you have in the past month? • Better: How many drinks did you have in the past week? • Can aid memory problems by providing a list • Drank with friends, drank at a bar, bought beer, etc.

8. Question Format: Open or Closed • Unstructured (“open-ended”) – only the Q is expressed; Structured (“closed”) – alternative answers are listed • Structured formats are preferred when feasible • Dimension of answers is clear • Respondents know more precisely what is being asked and what info is sought – they can perform more reliably • Data comparability • Open-ended responses need to be categorized when editing & postcoding data • Judgments have to be made about meaning or intent of respondents. • Recording Accuracy • it can be difficult to write and later read open-ended answers • Inappropriate Reasons • It takes time and effort to write effective structured Qs. • Structured items limit the “richness and variety” of people’s answers. That is true.

9. Using Response Scales • Implicit scale • Exists in the minds of respondents • Common understanding between the questioner and the respondent, concerning the scale to be used. • What is your age? Generally, respond in number of years • Weight, height, eye color, distance between cities, etc. • Explicit scale • The scale can be specified in the question • Months, pounds, dollars, degrees of temp, number of certain things • Scale not clear: How much do you study each week? • A lot; every day, whenever I need to, etc. • Explicit scale: How many hours do you study each week? • Depicted scale • Can’t be included in question, depict as response options: verbally, numerically, or graphically.

10. Common Response Categories/Scales • For any kind of rating: Excellent / Good / Fair / Poor • Satisfaction with amounts: Too many / Not enough / About right • For measuring amounts: Very much / Quite a bit / Some / Very little • Frequency of activities: Always / Often / Sometimes / Seldom / Never • How important, interested, or satisfied: Very / Fairly / Not at all • For opinions: Favor / Oppose • LikertScales: State issue or opinion, obtain respondents level of agreement or disagreement with those statements.  Strongly agree  Agree  Neutral  Disagree  Strongly disagree • Basic survey topic categories are: attitudes, images, decisions, needs, behavior, lifestyle, affiliations, and demographics

11. Scale Points to Include • No clear advantage to a 5- or 7- (or more) point scale. • 5-point scale provides sufficient discrimination • Might become de facto 3-point scale  tendency to avoid selecting the ends • 7-point scale may yield 5 points, but extra level of discrimination may increase completion time • Avoid mixing formats: use all 5- or all 7-point scales • A 5-point scale can be made into a 7-point scale • For Likert scales, add: Very strongly agree & Very strongly disagree

12. Likert Scales – On being “Neutral” or “Undecided” and Not Knowing • Reasons for not including Neutral or Undecided • May encourage some to take lazy way out  not have to ponder an item • May appeal to those who don’t wish to reveal their position on a controversial topic. • Reason for including • May have respondents who are truly neutral  frustrating • Omit • If you believe all potential respondents probably have developed positive or negative attitude toward topic; have an opinion • Instruct to select choice in each item that is CLOSEST to their position • Use “Don’t know” carefully when measuring opinions or attitudes • Only if respondents could have very little or no knowledge about topic • Can have them respond: “Using your best guess …”

13. Likert Scales – Balance Favorable & Unfavorable Statements • If all worded same way, items may not be read carefully • The instructor held my attention during class lectures. • The instructor wrote exams that fairly tested the material covered. • The instructor seemed to be well prepared for class. • The instructor was available for extra help outside of class. • Some students have a response bias called “response acquiescence”  tendency to agree with virtually all statements. • Have to read each item carefully & make item-by-item decision. • Generally try for ½ and ½ • The instructor was seldom able to hold my attention during class lectures. • The instructor wrote exams that fairly tested the material covered. • The instructor often appeared to be unprepared for class. • The instructor was available for extra help outside of class. • Make sure to note beforehand which are fav & which are unfav so that they can be re-coded later on.

14. Be Aware of Response Biases • Social desirability • Give socially acceptable responses • Yea- and nay-saying • Global tendency toward pos or neg answers • Prestige • Respondent wants to enhance his or her image • Threat • The Q instills anxiety or fear • Hostility • The response task brings out anger and resentment • Auspices • If know who is sponsoring the survey; their feelings toward the sponsor biases their answers. • Mental Set • Develop set of perceptions and assumptions that act as a frame of reference for responding • Order • Initiation (learn how to respond), routine (develop response strategy), fatigue • Extremity • Scale is dichotomized: pick extreme options; Fence-riding is the opposite – pick only the middle

15. Collecting Demographic Information • Only include demographic categories important to the objectives of your survey. • The more demographic items you include, the longer the questionnaire, and the greater the risk respondents will become bored. • Can also irritate respondents • Some requests (e.g., income) can be seen as invasions of privacy. • Some guidelines • When asking about age, ask for DOB • When asking about annual income (or other sensitive info), provide a range • Don’t let the alternatives overlap • If comparing to other surveys, ask for identical demographics • Use standard categories: Borrow items from the Census Bureau (www.census.gov) • Novice: “married, single, divorced, and widowed” • Pro: “single” is ambiguous  “married, separated, widowed, divorced, never married”

16. Miscellaneous Tips • Organize questionnaire carefully • Cluster items on same topic together – if possible, no more than 10 items in a list • Also, cluster by response scale • Write brief separate instructions for each section. • Provide clear directions – don’t assume they know what to do  tell them • Be careful about the order of items • Start with easy and interesting • Demographics last • Work hard on the Introduction • Work hard on the survey’s visual appeal • Provide a reward for responding

17. Field Pretest: “Piloting” • Rewrite Qs several times • Seek informal reviews of the items for clarity • Do a test run with a small group from the population • Perform a crude item analysis on categorical items • In the past month, how many times have you smoked marijuana?______ 0-10 _____ 11-20 ______ 21-30 • If every person chooses 0-10, you know that the categories are too inclusive – include more alternatives…

18. Don’t Lose Track of Objectives • They should guide the construction of the questionnaire • Will also direct the analyses and comparisons performed after the data are collected.

19. References • Alreck, P. L., & Settle, R. B. (1985). The survey research handbook. Homewood, IL: Richard Irwin, Inc. • Fowler, F. J., Jr. (2001). Survey research methods (3rd ed.). Thousand Oaks, CA: Sage. • Patten, M. L. (1998). Questionnaire research: A practical guide. Los Angeles: Pyrczak. • Rossi, P. H., Wright, J. D., & Anderson, A. B. (1983). Handbook of survey research. San Diego: Academic Press.