Survey Design and Sampling Methods for Accurate Data Collection

1. Lecture 4 - Survey design • Sampling • Sample size/precision • Data collection issues • Sources of bias

2. Why do surveys? • Information on particular population • prevalence of a disease • behaviour, knowledge, attitude • Planning of services • Collect information on data not routinely available: • e.g., mental health status, health behaviours • Repeat surveys to monitor trends (serial cross-sectional studies)

3. Bias and precision of the survey estimates • Bias: • selection bias relates to sample selection • information bias relates to information collected • Precision • relates to sample size

4. Reasons to sample • Reduce cost • Increase accuracy and quality of data collected

5. Definitions • Sampling unit • person or group (e.g., household) • Sampling frame • list of sampling units in the population • censuses • electoral lists • telephone lists • are institutional populations excluded (e.g., prisons, nursing homes)

6. Target and study population • Target population: • population for generalization of results • Study population: • population for collection of data • may be total target population or a sample

7. Types of sample • Non-representative • convenience • volunteers • Representative • simple random • systematic • cluster • multistage

8. Simple random sample • Each sampling unit in the population has equal probability of being included • Sampling with replacement: • each unit placed back in pool • Sampling without replacement (usual method): • each unit selected is kept out of pool

9. Simple random sample (cont’d) • Methods: • manual • tables of random numbers • computer-generated random numbers

10. Systematic sample • Select every nth individual from a list • can use existing numbers • e.g., patient appointments, medical records • Advantages: • Does not require complete sampling frame • Simple to carry out • Disadvantages: • May be unsuitable for cyclic or ordered data (e.g., every 5th patient when only 5/day)

11. Stratified sampling • Separate sample selected from different strata of population • Requires separate sampling frame for each stratum • Useful if there are small but important subgroups of the population (e.g., very old, very young, institutionalized, sick)

12. Cluster sampling • Sampling frame comprises groups (households, villages, schools) • Step 1: Simple random sample of groups • Step2: All individuals in each group included in survey • Advantages: • enumeration of population not needed • more efficient use of resources

13. Multistage sampling • Larger units sampled in first stage, smaller units later • e.g.: • stage 1 - sample of towns • stage 2 - sample of city blocks or census tracts • stage 3 - sample of households

14. Sampling for “hidden populations” • Homosexual men: • gay bars, newspapers • Injection drug users: • convenience sample (e.g., treatment facilities) • snowball sampling (through networks) • Capture-recapture methods • identify biases of sampling method

15. Planning a survey • Define target population • Select method of sampling • sampling unit, sampling frame, etc • Calculate sample size • Define survey data collection methods • Non-respondents • number of attempts to reach • different days, times

16. Sample size estimations • Requirements: • level of precision (width of confidence interval) • expected variability (estimated from previous studies, pilot study, or literature)

17. Design of questionnaires • List study variables • Collect existing questions and instruments • Adapt and/or develop new questions • Format questionaire • Pre-testing (timing, responses, clarity, etc.) • Revise, determine priorities, shorten

18. Question wording: clarity • Use concrete rather than abstract terms, e.g., • During a typical week, how many hours do you spend doing vigorous exercise? • Not: How much exercise do you get? • Avoid jargon, technical terms, slang • Avoid double-negatives (Do you disagree that doctors should not make house calls?) • Use active vs passive voice (Has a doctor ever told you vs Have you ever been told by a doctor?)

19. Question wording: clarity • Break long sentences into short ones (20 word or fewer) • Use good grammar but use informal style • Avoid hypothetical questions • Evaluate reading level (normally not more than 8th grade)

20. Question wording: neutrality • Do not suggest desirable response, e.g.: • Not: do you ever drink alcohol? • Better: how often do you drink alcohol? • Give permission to give undesirable response e.g.: • Sometimes people forget to take medications their doctor prescribes. Do you ever forget (or how often do you forget) to take your medications?

21. Question wording • Introduce attitude questions, e.g.: • People have different opinions about their medical care. We are interested in your opinion. • Avoid double-barreled questions • How much coffee or tea do you drink each day? • Avoid assumptions • How much help do you get from your family?

22. Response wording • Make them short • Use as few options as possible • Consider different types of non-response: • refuse • don’t know • no opinion • not applicable • omission by subject or interviewer

23. Response wording • Make sure responses are mutually exclusive (or give instructions to “check all that apply”) • Consider use of response card for multiple questions with same set of responses

24. Organization of questionnaire • Group questions by subject matter • Introduce each group with short descriptive statement (e.g., now I am going to ask you some questions about your use of health services) • Begin with more emotionally neutral questions • More sensitive questions (e.g., income, sexual function) near end of questionnaire

25. Organization of questionnaire • interviewer-administered: repeat time frame fairly frequently • self-administered: repeat time frame at top of each page or each set of questions, e.g.: During the past year, how many times have you: • Visited a doctor? • Been a patient in an emergency department? • Been admitted to hospital?

26. Organization of questionnaires • Group questions with similar response scale • Format skip patterns • screener questions • branching questions • Time frame • group questions that ask about same time frame • “usual” behavior vs specified time period • assist respondent with milestones to help define reference time frame

27. Questionnaire mode • Face-to-face • Telephone • Mail • Other: • diaries • Mixed mode

28. Face-to-face interviews:advantages • reduce items with no response • easier for older, less educated, lack of fluency in language • some formats easier to administer: • skip patterns to avoid irrelevant questions • open-ended questions - can probe for more complete response

29. Face-to-face interviews:disadvantages • cost • time • effort (interviewer training, evaluation of inter-rater reliability) • interviewer biases • differences in sociodemographic characteristics of interviewer and subject

30. Telephone interviews:advantages • less expensive than face-to-face • reduce items with non-response • some formats easier to administer: • skip patterns to avoid irrelevant questions • open-ended questions - can probe for more complete response • large, representative samples can be organized from one office • avoids bias associated with appearance of interviewer

31. Telephone interviews:disadvantages • misses households without telephone • misses those with unlisted ‘phone numbers • bias when calls made during day • multiple calls may be needed • perceived as intrusive by some • difficult to administer items with multiple response options

32. Mailed questionnaires:advantages • least expensive • can be coordinated from one office • social desirability minimized • inconsistent results on completeness of reporting (e.g., for # MD visits)

33. Mailed questionnaires:disadvantages • relatively low response rates • multiple mailings, cover letter, letterhead, advance warning, token of appreciation, SSAE • difficult to get information on non-respondents • differences between early and late responders • items may be omitted: 5-10% may be unusable • cannot control order of questions • postal strikes

34. Analysis of surveys • Missing data • exclude • imputation: e.g., based on characteristics of respondents • sensitivity of estimate to method of imputation • Weighting of estimates • for stratified samples

35. Analysis of surveys (cont’d) • Crude estimates, confidence intervals • Continuous data: Mean, median, quartile • Categorical data: proportion • Confidence intervals to describe precision

36. Bias and precision of the survey estimates • Bias: • selection bias relates to sample selection • information bias relates to information collected • Precision • relates to sample size

37. Selection bias in surveys • Does the final analysis sample represent the original target population? • Sources of bias: • sampling method • non-response • missing data

38. Information bias in surveys • Bias in measurement of outcomes • Sources of information bias: • non-validated measurement instrument • unblinded or poorly trained data collectors • response set • etc.