Survey Sampling - 2

1. Survey Sampling - 2 Survey Design

2. Specification of the research goals • The most critical stage in designing any survey but often one of the most difficult • Need to turn a fairly simple question e.g. ‘What is the impact of social capital on people’s life chances?’ into a series of hypotheses which you want to test • E.g., People who have an extended structure of formal social networks better able to find jobs than those who do not have such networks. • Need to first ask yourself which is the most appropriate survey design for the hypotheses being tested

3. Decision processes • Survey design depends on the research questions being asked • Need to choose an appropriate survey design • What is the population of interest? • Is there a way of selecting a sample that will be unbiased and represent the population you are interested in? • What is your unit of observation – the household, families, individuals, institutions, countries, something else? • What is your unit of analysis – e.g. you may observe individuals but your unit of analysis may be the number of spells they have had in paid employment • Do you need cross-sectional data or longitudinal data to answer your research questions?

4. Remember… Who responds? • Individuals • Households • Establishments/ organizations • Land • Animals or plants About… • Individuals • Events • Households • Other people in the household • The local area • An industry Unit of ANALYSIS Unit of OBSERVATION

5. Decision processes • Survey design depends on the research questions being asked • Need to choose an appropriate survey design • What is the population of interest? • Is there a way of selecting a sample that will be unbiased and represent the population you are interested in? • What is your unit of observation – the household, families, individuals, institutions, countries, something else? • What is your unit of analysis – e.g. you may observe individuals but your unit of analysis may be the number of spells they have had in paid employment • Do you need cross-sectional data or longitudinal data to answer your research questions?

6. Cross sectional designs • Cross sectional surveys provide a snap-shot of a population at one point in time • Often used for large government surveys, for one-off projects, political polling • If the research questions are about describing or explaining current status a cross-sectional design is what you need • Are many design and data collection variants within a cross-sectional design determined by the research questions • Sampling sub-populations, age group or by geographical location • Questionnaire administration only • Diaries – time-use, nutrition, or travel • Responses to visual stimulus of some kind e.g. market research on branding • An experimental design using randomized control and treatment groups (can apply to longitudinal surveys as well)

7. Cross-sectional designs used for trend analysis • Repeated cross sectional surveys (e.g. Family Resources Survey, General Household Survey, Labour Force Survey) • Common design for large government surveys • New sample drawn for each survey • Carry similar questions each year • Used for trend or time series analysis at aggregate level • A cross-sectional design often includes a longitudinal element • Retrospective life histories of employment, marriage and fertility, housing, migration • Questions about a reference period in the past e.g. the last month, year, or since leaving full-time education etc.

8. When do we need longitudinal data? (1) • Longitudinal designs collect data about different time points for the same units of observation or cases (usually individuals but could be dwellings or employers etc.) • Information on transitions and change over time • To measure inherently longitudinal phenomena • Change • movements into or out of poverty • labor market transitions and durations • longer term social mobility (sosyalhareketlilik) • household formation and dissolution • residential mobility • Stability • persistence of poverty over time • persistence of people’s attitudes over time • Inter-generational transmission

9. When do we need longitudinal data? (2) • Temporal element central to inferences about causality • If X and Y: Prove an association between variables • If X and Y and not Z: Isolate the relationship • If X then Y: Temporal ordering • To control for the effects of unmeasured fixed differences (heterogeneity) between subjects • To study social change by separating out age, period and cohort effects • To establish the effect of a treatment e.g. a policy intervention by comparing periods before and after the introduction of the policy

10. Designs for collecting longitudinal data • Retrospective studies • Cross sectional collection of historical information • Past events and transitions are recalled by respondents e.g. life time employment history • Record linkage panels • From data collected for administrative purposes e.g. taxation or social welfare system or from Censuses • Need to reliably identify individuals and allow them to be linked over time • Provide powerful data but limited in scope for analysis of social change • Prospective studies • same individuals are re-interviewed in successive years • suffer less from recall error than retrospective studies

11. From concepts to indicators to questions • The type of data you need to collect depends on what you need to know to answer your research questions • Steps in the process: • Defining the key concepts you are need to measure • Operationalize those concepts, i.e., what indicators will measure the concept of interest? • Develop indicators into questions in your questionnaire • You will examine measurement error in greater detail over the coming weeks

12. Working with Abstract Concepts

13. Standardization • The aim is to produce as standardized questionnaire that will produce data for quantitative analysis • “Standardized” – all participants are asked the same questions in the same format in a consistent manner • Why standardize? • Every participant receives the same treatment or stimulus, any variation in the resulting data will represent real differences between cases • This ideal is very difficult to achieve!

14. Reliability and validity • Aim: To have questions which are… • Reliable: They are understood the same way at each time they are asked and therefore yield consistent answers on separate occasions • Valid: They are measuring what we think they are measuring and are understood in the same way by each respondent

15. Basic rules of question design • What do you think some basic rules of question design might be? • What be might some characteristics of unreliable questions that we can spot simply by reading them?

16. The Basics of WRITING Good Questions

17. Good Design Helps Minimize Nonresponse and Measurement Error • Instrument should be easy to understand, to navigate, and to complete • Encourage response • Questions must be: • Clear • Unambiguous • Minimize possible bias • E.g., questions with socially desirable answers • The harder it is for the respondents, the higher the survey nonresponse rate

18. How Hard Can It Be? • That’s everyone’s first reaction • Turns out (and you will likely experience it in this class), it’s harder than it seems • Here are some reasons (and there are more): • The devil’s in the details: Differences in words and question wording matter – sometimes a lot • It’s not just asking questions, but asking the right questions (with the proper response scales) that get at what you’re trying to learn • The questions need to be robust to lots of factors such as how different people (mis)interpret the questions (including response scales, question ordering, etc.) – consistency is key The reality: It’s just downright hard to write questions that are perfectly clear and unambiguous, yet reasonably simply worded, and that get at what you want to know

19. Anecdote • Two Jesuit novice priests both wanted a cigarette while they prayed. They decided to ask their superior for permission. The first asked but was told no. A little while later he spotted his friend smoking and praying. "Why did the superior allow you to smoke and not me?" he asked. His friend replied, "Because you asked if you could smoke while you prayed, and I asked if I could pray while I smoked!"

20. An Example – What’s “Truth?” Dilmann 2007.

21. ? • Why did these results for the alternative sets of categories and the two questions differ so dramatically? • Which answers should be trusted? • Howcan you ever be sure that whatever questions are asked in surveys obtain answers that provide the best possible estimates of the distribution of the characteristic of interest in the population?

22. Eliciting Correct Information Not As Simple As One Might Expect Groves et al.

23. CRITERIA FOR ASSESSING EACH SURVEY QUESTION • Does the question require an answer? • To what extend do survey recipients already have an accurate, ready made answer for the question they are being asked to report? • Can people accurate recall and report past behavior? • Is the responded (katılımcı)willing to reveal the required information? • Will the responded feel motivated to answer each question? • Is the respondents understanding of response categories likely to be influenced by more than words? • Is survey information being collected by more than one mode? • Is changing a question acceptable to the survey sponsor?

24. A survey question is more than a general inquiry. Survey questions make it possible to determine the distribution of a characteristic (an attitude, belief, behavior, or attribute of each respondent) in the survey population. Neither of the following questions meet that criterion. • If you fixed dinner at home last night, did you eat meat as part of that meal? • Yes • No • When you go out to eat, which type of food do you most prefer? • Turkish • Italian • Chinese • Other

25. CRITERIA FOR ASSESSING EACH SURVEY QUESTION • Does the question require an answer? • To what extend do survey recipients already have an accurate, ready made answer for the question they are being asked to report? • Can people accurate recall and report past behavior? • Is the responded (katılımcı)willing to reveal the required information? • Will the responded feel motivated to answer each question? • Is the respondents understanding of response categories likely to be influenced by more than words? • Is survey information being collected by more than one mode? • Is changing a question acceptable to the survey sponsor?

26. All are acceptable depending on the precision of the information you require.

27. Respondent’s opinion on issue for which wording changes produce substantial inconsistencies in response choice: “Tall people are more likely to be elected Prime Minister of Turkey.” Do you strongly agree, somewhat agree, somewhat disagree, or strongly disagree with this statement? (or) Do you very strongly agree, strongly agree, agree, disagree, strongly disagree, or very strongly disagree? (or) On a scale of 1 to 7, where 1 means entirely agree and 7 means entirely disagree, use a number to indicate how strongly you agree or disagree.

28. People do not have ready-made responses to opinion and belief questions to nearly the same degree that they possess answers to questions about their educational level, the kind of car they drive, or whether they own their home. • The vaguer the question, the vaguer the categories (e.g., using only numbers, as in a 1–7 scale), and the more remote these items are from people’s experiences, the more likely a question is to produce inconsistent responses if we ask the same person to answer this question at different times.

29. CRITERIA FOR ASSESSING EACH SURVEY QUESTION • Does the question require an answer? • To what extend do survey recipients already have an accurate, ready made answer for the question they are being asked to report? • Can people accurate recall and report past behavior? • Is the responded (katılımcı)willing to reveal the required information? • Will the responded feel motivated to answer each question? • Is the respondents understanding of response categories likely to be influenced by more than words? • Is survey information being collected by more than one mode? • Is changing a question acceptable to the survey sponsor?

30. Asking how many hours a person watched television on the first Sunday of the previous month is an example of a behavior that is hard to remember. Respondentsare unlikely to be able to recall something so precise from that long ago. To solve this problem, surveyors often ask how many hours per day a person “usually” watches television. To answer, the respondent must recall what she “usually” does and estimate. Recalling the frequency of these routine or ordinary behaviors is subject to considerable potential error, as evidenced by the effect of the category choices offered as part of the survey question.

31. This tendency for recall and reporting to decline as a function of length of recall has yield an important measurement error model. If there were no recall problems, the ith person would report in responsetothequestion. The model specifies that instead of , the respondent reports

32. : is the proportion of events that are reported despite concerns about sensitivity and social desirability. b : is the rate of decline in reporting as a function of time : is the deviation from the model for the ith individual. The model specifies that the proportion of events correctly reported exponentially declines. The model implies that • for events that are distinctive, near another easily recalled event and important in the life of the respondent, a is close to 1.0 and b is close to 0.0. • for nonsensitive events that are easily forgotten, a may be close to 1.0 but b is large.

34. CRITERIA FOR ASSESSING EACH SURVEY QUESTION • Does the question require an answer? • To what extend do survey recipients already have an accurate, ready made answer for the question they are being asked to report? • Can people accurate recall and report past behavior? • Is the responded (katılımcı)willing to reveal the required information? • Will the responded feel motivated to answer each question? • Is the respondents understanding of response categories likely to be influenced by more than words? • Is survey information being collected by more than one mode? • Is changing a question acceptable to the survey sponsor?

35. The fact that respondents know an answer does not mean they are willing to provide it. Many respondents are reluctant to reveal certain information about themselves, for example, their income. Others may be unwilling to answer questions about previous drug use, or having shoplifted when they were teenagers. • Considerable evidence suggests that people are more likely to give honest answers to self-administered than to interview questionnaires For example, when asked the question, “How often have you driven a car after drinking alcoholic beverages?” only 52% responded “never” to the self-administered questionnaire versus 63% for the comparable telephone survey (Dillmanand Tarnai, 1991).

36. CRITERIA FOR ASSESSING EACH SURVEY QUESTION • Does the question require an answer? • To what extend do survey recipients already have an accurate, ready made answer for the question they are being asked to report? • Can people accurate recall and report past behavior? • Is the responded (katılımcı)willing to reveal the required information? • Will the responded feel motivated to answer each question? • Is the respondents understanding of response categories likely to be influenced by more than words? • Is survey information being collected by more than one mode? • Is changing a question acceptable to the survey sponsor?

37. Designers of interview surveys often write questions independent of any motivational considerations, leaving it to the interviewer to encourage or persuade respondents to carefully select and report complete answers. • Questionnaires are sometimes constructed in a way that requires respondents to consult a separate instruction booklet to understand unclear questions. • Motivation can be encouraged in many ways, ranging from incentives and follow-up reminders to respondent-friendly questionnaire design.

38. CRITERIA FOR ASSESSING EACH SURVEY QUESTION • Does the question require an answer? • To what extend do survey recipients already have an accurate, ready made answer for the question they are being asked to report? • Can people accurate recall and report past behavior? • Is the responded (katılımcı)willing to reveal the required information? • Will the responded feel motivated to answer each question? • Is the respondents understanding of response categories likely to be influenced by more than words? • Is survey information being collected by more than one mode? • Is changing a question acceptable to the survey sponsor?

39. Responses to the questions about studying and presentedat the beginning of this section were clearly influenced by more than words alone. • Attitudinal and belief questions typically rely on vague quantifiers, such as strongly favor to strongly oppose, high priority to low priority, agree to disagree, or even vaguer ones that rely on numbers such as –3 to +3, 1–7, or 1–10. • Such numerical scales require respondents to give a certain amount of definition to any category they choose to use. The vaguer the question and answer categories, the greater the potential for measurement error.

40. CRITERIA FOR ASSESSING EACH SURVEY QUESTION • Does the question require an answer? • To what extend do survey recipients already have an accurate, ready made answer for the question they are being asked to report? • Can people accurate recall and report past behavior? • Is the responded (katılımcı)willing to reveal the required information? • Will the responded feel motivated to answer each question? • Is the respondents understanding of response categories likely to be influenced by more than words? • Is survey information being collected by more than one mode? • Is changing a question acceptable to the survey sponsor?

41. This means that data collected by each mode needs to be comparable with that collected by another. An illustration is the fact that 42% of the telephone respondents versus only 23% of the mail respondents chose more than 2 1⁄2 hours in response to the hours of study question posed at the beginning of this section. • These challenges of mixed-mode surveys will be discussed later.

42. CRITERIA FOR ASSESSING EACH SURVEY QUESTION • Does the question require an answer? • To what extend do survey recipients already have an accurate, ready made answer for the question they are being asked to report? • Can people accurate recall and report past behavior? • Is the responded (katılımcı)willing to reveal the required information? • Will the responded feel motivated to answer each question? • Is the respondents understanding of response categories likely to be influenced by more than words? • Is survey information being collected by more than one mode? • Is changing a question acceptable to the survey sponsor?

43. Often sponsors want information about which respondents have virtually no knowledge and for which formulating a meaningful answer is difficult. • Political considerations may also dictate the selection of question and answer categories. Thus, it is important to ask sponsors whether questions that appear troublesome are subject to change and if so, how much.

44. Writing questions is a difficult challenge precisely because so many factors simultaneously influence whether a proposed question obtains accurate answers. • The eight questions listed here constitute the mental checklist when, surveyors asking for help thrust a draft and ask, “What do you think?” The search for tools and principles for writing questions is a means of overcoming each of these potential problems. • This is the topic to which we now turn.

45. WHICH QUESTION STRUCTURE ISMOST APPROPRIATE?

46. Types of Question Content • Dillman(1978) distinguishes five types of question content • Behavior: what people do • Beliefs: what people think to be true or false • Knowledge: what people know to be true or false • Attitudes: what people think to be desirable • Attributes: information about a person’s characteristics • Clarity about what you are trying to measure with your question is essential

47. Data for Quantitative Analysis • When designing a question you need to think about how that variable will be used in analysis • Different levels of measurement needed for different types of descriptive or inferential statistical analysis • The way we ask a question will produce data at nominal, ordinal, or interval levels • What are these levels of measurement? • How are they used in analysis?

48. Nominal Level of Measurement • A variable (question) where the different response options have no set rank-order • We cannot order the categories from low to high, lesser to greater in any meaningful way • Religious affiliation or ethnicity have no natural rank-order • Another type of categorical variable

49. Ordinal Level of Measurement • Rank-order categories from low to high, lesser to greater • Cannot specify in numeric terms how much difference there is between categories • Response options of “child”, “teenager”, “young adult”, “middle aged”, “elderly” • Inherent ordering, but we do not know the numerical gap between categories • A type of categorical variable

50. Interval Level of Measurement • Variable (question) in which categories can be ranked from low to high in some meaningful way • Can specify the amount of difference between values of categories, e.g., age in years from youngest to oldest and we know that the difference between age 20 and 25 is 5 years • Many interval questions in surveys • Hours of work, income, height and weight • Anything that is collected as a number and produces a distribution • Often described as continuous variables