The (mis)measure of people

1. The (mis)measure of people

2. Agenda • Basics of survey structure & questionnaire design • Types and formatting of items

3. Questions • What kind of surveys do you typically receive during the year? • What is your usual response to them? • What makes them more intriguing so that you take them? • Think of an example for your survey– customer satisfaction

4. Targeting your survey • who is your intended market? • how diverse are they? • what do you most need to know? • what level of information is needed? • how are they best accessed? • what are their characteristics that may affect response? • what are their expected objections?

5. Levels of measurement

6. Designing the Questionnaire Step 1: Specify what information will be sought Step 2: Determine type of questionnaire and administration method Step 3: Determine content of individual questions Step 4: Determine form of response to each question Step 5: Determine wording of each question Step 6: Determine sequence of questions Step 7: Determine physical characteristics of questionnaire Step 8: Reexamine steps 1 - 7 and revise if necessary Step 9: Pretest questionnaire and revise if necessary

7. Wording of Questions (a) Use simple words & short sentences (b) Avoid ambiguous words and questions (occasionally, sometimes, often, etc. Try to anchor each point with a specific description) (c) Avoid leading questions (d) Avoid implicit alternatives (e) Avoid implicit assumptions (f) Avoid generalizations and estimates (g) Avoid double-barreled questions (h) Avoid response order (i.e., position) bias

8. The attribute "customer service," for example, can be phrased as any one of the following: • Direct question • How satisfied are you with customer service? [Very satisfied, satisfied, Neither • satisfied nor disatisfied, Disatisfied, Very Dissatisfied • Direct request • Please rate customer service. [Excellent, Very Good, Good, Fair, Poor] • Statement with which to agree or disagree (Likert scale) • ABC Company provides good customer service. [Strongly agree, Agree, • Neither agree nor disagree, Disagree, Strongly Disagree] • Question of frequency • How often does customer service exceed expectations? [Very Frequently, • Frequently, Neither frequently nor infrequently, Infrequently, Very infrequently] • Question of extent • To what extent does customer service exceed expectations? [To very great • extent, To great extent, To some extent, To little extent, To very little extent]

9. Flame-broiled or Fried? Original Question: “Do you prefer your hamburgers flame-broiled or fried?” Result: flame-broiling (Burger King) beat frying (McDonald’s) by a 3-1 margin Revised Question #1: “Do you prefer a hamburger that is grilled on a hot stainless-steel grill or cooked by passing the raw meat through an open gas flame?” Result: 53% preferred the grill (McDonald’s) Revised Question #2: “The chain that grills on a hot stainless-steel griddle serves its cooked hamburgers at the proper temperature without having to use a microwave oven. And the chain that uses the gas flame puts the hamburgers after they are cooked into a microwave oven before serving them. Just knowing this, from which of these two chains would you prefer to buy a hamburger?” Result: the grill (McDonald’s) beat the flame (Burger King) by a 5.5 to 1 margin

10. Handling Sensitive Questions • Don’t ask unless absolutely necessary! • Give broad response categories (e.g., income, age) • Place near end of questionnaire • “Hide” them in less sensitive questions • Use counterbiasing statement • Use randomized response technique • Transition sections & inform of rights

11. Use of Counterbiasing Statement • “Recent studies have shown that a high percentage of males use their wives’ cosmetics to hide blemishes. Have you used your wife’s cosmetics in the past week?” Providing an acceptable context for answering what might have been sensitized before

12. Rank-Order Scales Rank the following soft-drinks from 1 (best) to 5 (worst) according to your taste preference: Coca-Cola _____ 7-Up _____ Dr. Pepper _____ Pepsi-Cola _____ Tab _____

13. Paired Comparison Scale For each of the following pairs, which soft-drink do you think is better (please check one soft-drink for each pair). ____Coca-Cola or ______7-Up ____7-Up or ______Pepsi-cola ____Tab or ______Coca-Cola ____Dr. Pepper or ______Pepsi-cola ____ Pepsi-cola or ______Tab ____ Coca-Cola or ______ DR. Pepper ____ Pepsi-cola or ______Coca-Cola ____Tab or ______ Dr. Pepper ____7-Up or ______Tab ____ Dr. Pepper or ______ 7-Up

14. Constant Sum Scales • Allocate a total of 100 points among the following soft-drinks depending on how favorable you feel toward each; the more highly you think of each soft-drink, the more points you should allocate to it. (Please check that the allocated points add to 100.) • Coca-Cola _____ points • 7-Up _____ points • Dr. Pepper _____ points • Tab _____ points • Pepsi-Cola _____ points • 100 points

15. Customized Rating Scales Odd vs. Even Scale Points Odd Strongly Agree _____ Agree _____ Neutral _____ Disagree _____ Strongly disagree _____ Even Strongly Agree_____ Agree _____ Disagree _____ Strongly disagree___

16. Balanced Very good ______ Good ______ Fair ______ Poor ______ Very Poor ______ Unbalanced Excellent ______ Very Good ______ Good ______ Fair ______ Poor ______ Balanced vs. Unbalanced Scales

17. Forced Extremely Reliable ___ Very Reliable ___ Somewhat Reliable ___ Somewhat Unreliable ___ Very Unreliable ___ Extremely Unreliable ___ Forced vs. Unforced Scales Unforced Extremely Reliable ___ Very Reliable ___ Somewhat Reliable ___ Somewhat Unreliable ___ Very Unreliable ___ Extremely Unreliable ___ Don’t know ___

18. Labeled vs. End Anchored Scales End Anchored Excellent _____ _____ _____ _____ Poor _____ Labeled Excellent _____ Very Good _____ Fair _____ Poor _____ Very Poor _____

19. 5 Point Excellent _____ _____ _____ _____ Poor _____ 10 Point Excellent _____ _____________ _____________ _____________ _____________ _____________ _____________ _____________ _____________ _____________ Number of Scale Points Poor

20. Amount of Information 1 2 3 4 5 10 .................100 Number of Scale Points Recommended Number of Scale Points Begins to flatten out in return of information More than 5 options will provide little additional discrimination

21. Agreement: • Strongly Agree • Agree • Undecided • Disagree • Strongly Disagree • Importance: • Very Important • Important • Moderately Important • Of Little Importance • Unimportant Formats for Likert Items 1…2…3…4…5 • Frequency: • Very Frequently • Frequently • Occasionally • Rarely • Very Rarely • Never • Quality: • Extremely Poor • Below Average • Average • Above Average • Excellent • Likelihood: • Almost Always True • Usually True • Often True • Occasionally True • Sometimes But Infrequently True • Usually Not True • Almost Never True

22. Semantic Differential Scale X= ABC Bank 0= NBC Bank X O old-fashioned ____ ____ ____ ____ ____ ____ ____ modern not friendly ____ ____ ____ ____ ____ ____ ____ friendly expensive ____ ____ ____ ____ ____ ____ ____ inexpensive not trustworthy ____ ____ ____ ____ ____ ____ ____ trustworthy O X O X O X

23. Student’s fruit preferences by eating occasion (importance of time in consumption decisions) strawberry banana orange peach grape plum apple kiwi Breakfast Preferences strawberry banana grape orange kiwi plum apple peach Snack during day strawberry grape orange apple plum banana peach kiwi Supper dessert

24. Physical Characteristics of Questionnaire: Recommendations • questionnaire should appear simple to complete (white space is your friend!) • minimize number of pages (smaller fonts are OK, provided form appears simple) & items • mix-up response formats occasionally (avoids response set bias and breaks monotony) • use directions as necessary for each group of items, but keep them short and simple • number items within each section • have respondents check boxes rather than lines • Use consistent response formats (don’t change too often) • use shading, boxes, lines, etc., to keep it interesting

25. Question considerations • start with easy, non-threatening questions (warm up) • place more personal, threatening questions at end • never start mail survey with open-ended questions • for historical demographics, follow chronological order • ask about one topic at a time • when switching topics use a transition • reduce response set • for filter or contingency questions, make a flow chart • thank the respondent at beginning and end • keep the survey short • keep language simple • include “don’t know” and “NA” for appropriate options • try not to make questions dependent on each other

26. Foot in the door… First make a small request, then when granted, make a larger (more desirable) request • works best with prosocial/altruistic requests • works best if no extra incentives are offered • 10% improvement over simple request, 20% if no incentive • works by cognitive dissonance & self esteem

27. Door in the face… Make a large request, then when it is refused, make a smaller (more desired) request • more effective if prosocial/altruistic (do it for everybody) than selfish • should be no delay between requests • slight delay produces 10% improvement, no delay 20% • works by reciprocal concessions

28. Supplementary Slides (not assigned for study)

29. Scale Development Items Items • define target population • define specific domains to be measured • generate items reflect each domain • construct item format • construct survey • test items for feedback • revise • administer to representative group • frequency distribution (representative?) • Cronbach’s alpha (above .70?) • validation & reliability checks • complete final version Items

30. Cronbach’s alpha as a measure of scale internal reliability

31. Analyzing Data • Frequency distribution • Cross tab & chi-square • Correlation & regression • t-test & ANOVA • Cluster analysis • CHAID • MDS

32. Frequency distribution

33. Crosstabs & chi-square If significant, then rows & columns are dependent

34. Correlation & regression r = .50 r2 = .25

35. T-test: Comparison of differences between means

36. ANOVA & Post Hoc Comparisons

37. Cluster Analysis: hierarchical grouping on the basis of similarity Unity Each cluster represents a similarity grouping

38. Multidimensional Scaling

39. CHAID Sole user more likely to personalize by upgrading People with older computers more likely to upgrade Influence of system characteristics Online realize need for faster modem Focus on less recent PC buyers, target for upgrade packages with faster modems

40. Comparison of Three Primary Methods of Administration mail telephone personal (A) Sampling Control ability to secure sampling frame + + ability to secure correct respondent - + + response rate - + + (B) Information Control ability to probe for detailed answers - + ability to handle complex information - + amount of information obtained - + flexibility of question sequencing - + + protection from interviewer bias + - - ability to obtain personal information + - - ability to show visual displays - + ability to establish rapport with respondent - + ability to offer anonymity + - - (C) Administrative Control time requirements - + cost requirements + - quality control / supervisory requirements + -

41. Return Rate • sample size calculator • notify beforehand • free postage (SASE) • personalize (name & address) • relevant to needs & interests • incentives • closing deadline in 2-2.5 weeks • reminder • security, confidentiality, anonymity • if <50% check for bias (what have in common) • mail-ins have 3-5% response rate (10% is good)

42. Considerations for Web Based Surveys • Draft the questions • Provide a password (if needed) • E-mail reminders (know return cutoffs) • Personalize them (mailing list) • Design for people with minimal skills • Use a larger font • Response icons • Textured background • Open questions use a text box • E-mail to researcher for comments • Thank you page

43. Personal and Telephone Interviews: “foot-in-the-door” strategy (Reingen & Kernan 1977) Mail Surveys (1) pre-notification, especially by telephone (Schelegelmilch & Diamantopoulos 1991) (2) cover letter / type of appeal (Houston and Nevin 1977) (3) use SASE, not business reply envelope (4) personalized address (5) no bulk mail (6) sponsorship (if corporate, refer to “research department” and never include sales pitch (7) monetary incentives (Brennan, Hoek & Astridge 1991) (8) offer survey results (9) follow-up postcard, duplicate questionnaire, or telephone call