Eric Vance Director of LISA Assistant Research Professor Department of Statistics

Designing Surveys and Reporting the Results Workshop for Human Rights Researchers and Campaigners AAAS Scientific Responsibility, Human Rights and Law Program Eric Vance Director of LISA Assistant Research Professor Department of Statistics June 8, 2012 Washington, DC

This workshop will consist of six 25-minute modules: ?

Module 1: Introductions Data Experiments Knowledge Understanding Decisions

Eric Vance from Virginia Tech is leading this workshop Triple major in Math, Economics, and Statistics at UC Berkeley (1997) PhD in Statistical Science from Duke University (2008) Assistant Research Professor in the Department of Statistics, Virginia Tech (2008-present) Director of LISA, Virginia Tech’s Laboratory for Interdisciplinary Statistical Analysis (2008-present) Teaches statisticians how to communicate and collaborate with non-statisticians

Eric Vance from Virginia Tech is leading this workshop Traveled around the world three times through 67 countries On-the-ground statistician in Mozambique during the summer of 2011 On Sunday, traveling to South Sudan for 12 days to pitch the idea of training statisticians to communicate and collaborate with non-statisticians to help researchers rebuild higher education in agriculture

Eric Vance from Virginia Tech is leading this workshop http://www.lisa.stat.vt.edu/?q=blog/59 http://magazine.amstat.org/blog/2012/05/31/internationalexperiences/

Laboratory for Interdisciplinary Statistical Analysis LISA helps VT researchers benefit from the use ofStatistics Experimental Design • Data Analysis • Interpreting ResultsGrant Proposals • Software (R, SAS, JMP, SPSS...) LISA’s mission is to train statisticians to become interdisciplinary collaborators and promote the value of statistical thinking in all phases of scientific research. In 2011 there were 1046 total clients of LISA’s three services: 355 Collaborative Projects from 62 VT departments 304 visitors to Walk-in Consulting 387 attendees at LISA Short Courses www.lisa.stat.vt.edu

Statistics in All Phases of Research Data Control sources of variation,detect outliers Visualize the data; analyze with statistical models Experiments Knowledge Determine practical and statistical significance of results Design experiments to answer research questions Understanding Make scientifically sound decisions and communicate them Decisions

Statistics can be extremely useful in human rights work While working as an intern at Benetech, KristianLumdeveloped statistical methodology to estimate the number of disappearances in a state in Colombia. A warehouse of police records in Guatemala was sampled, using a formal design, to determine whether there were records related to rights violations, and how many. Statistics Without Borders (SWB) is analyzing surveillance data regarding child disease in Somalia. Over 40 teams of SWB statisticians have worked on projects related to global health that are not political in nature. http://community.amstat.org/statisticswithoutborders/home/

Work on exercises in workshop packet for Module 1: Introductions Data Experiments Knowledge Understanding Decisions

Module 2: Survey Fundamentals

Statistical analyses should start with a research question What are your questions?What do you want to know? Do you need data to answer these questions? If so, what data do you need? How will you collect these data?

There are many ways to collect data besides surveys Direct measurements or observations: instead of asking someone how much water they use, install a water meter. Use existing data sources: collect data from the water utility Explore complex issues with focus groups: interview a group of women about how they use water in their daily lives.

Scientific surveys measure the statistics behind the stories Personal interviews and focus groups are designed to collect anecdotes and to understand “why” something is happening Surveys can tell you “what” is happening to a large number of people under different situations Surveys can also help explain “why” something is happening Surveys can transform stories into numbers and allow us to make statistical comparisons: before vs. after; here vs. there; Group 1 vs. Group 2

Scientific surveys elicit unbiased answers from samples representative of the overall population In Module 3 we will discuss ways to elicit accurate responses The key concept of a survey is that the one can draw conclusions about the overall population based on the results from a much smaller sample An exit poll of 1000 randomly sampled voters will be within 3% of the final vote percentage 95% of the time The more precise you need your final result to be, the more people you must sample: an exit poll of 9600 voters will be within 1% of the final vote percentage 95% of the time

5 steps involved in designing a survey Clearly define your research objectives (What do you want to know?) Define the population to be sampled (Who do you want to know about?) Develop a sampling plan and … design a questionnaire to minimize errors and biases (How does each question relate to your research objectives?) Pilot test and retest your survey (Fix errors and start again at Step 1)

Step 1: clearly define your research objectives State CLEARLY and CONCISELY your Overall research goals Specific scientific questions Refer to these objectives constantly throughout the design of your survey to ensure your survey is answering the desired questions of interest

Step 2: Define the population to be sampled Who will you interview to answer your research questions? The overall group of interest or the target group is the population. Subject: Any material we measure Plant, person, piano, etc. Population:representation of all the possible outcomes or measurements of interest Sample:subset of the population to be measured (i.e., a group of subjects that represent the population)

Step 3: Develop a sampling plan Once the target population has been identified, next the sampling plan must be devised Goal: Randomly select a small percent of the population that will in turn represent the ideas of the population as a whole. The sampling plan involves: The technique used to select the subjects for your study Simple random sampling (draw names out of a out or from a list) Stratified random sampling (sample one person per team/table) Cluster sampling (randomly pick a team/table and sample everyone from that table) Systematic sample (sample the first person from each team/table) The number of people needed for your study Sample size calculations (sample enough for precise results for $)

Simple random sampling Subjects chosen by random mechanism Each subject has an equal chance of being in the study Easiest to summarize BUT most tedious to implement in the field Example:Randomly select 10 students from the Stat 3005 class roster to ask a question.

Stratified random sampling First divide population into strata(groups) based on similarity Then randomly select subjects within each strata Easier to implement May result in more precise summary Example: Randomly select 5 male students and randomly select 5 female students from the STAT 5615 class roster to ask a question.

Cluster sampling Example: Population: opinions of all students (attending class) at Virginia Tech Randomly select a certain number of classes Ask all students in each class their opinion Note: Cluster sampling is often NOT as efficient as stratified sampling for surveying. Population has many clusters First randomly select a number of clusters Then sample all the units within each cluster Require clusters to be representatives of population

Systematic sampling Example: Telemarketers randomly sample every 10th phone number on the Yellow Book to make marketing calls. Select every kth subject from a list of all possible subjects

Sample size calculations How many people do we interview? Answer: It depends Sample size calculations can be computed using statistical methods (Ask a statistician for help!) Sample size calculations also involve characteristics of the study: Time, money, precision required For many Gallup polls, the population of interest is all adult Americans. To represent this population, the sample usually consists of around 1,000 adults. When sample sizes approach 500 or more the gains in accuracy get smaller and smaller for the increase in sample size.

Sample size calculation for a proportion Let n = sample size σ = standard deviation d = confidence interval size α = significance level Then, to get a (1-α/2)*100% confidence interval, we need a sample size of:

Sample size calculationfor a proportion For example, suppose we want an estimate for a 95% confidence interval of width 0.2 (meaning we have a 0.1 margin of error). If we know from a pilot study that the standard deviation of the population is 1, then, σ = 1 d = 0.2 α = 0.05 And plugging these numbers into the previous equation, we get, n = 384.15 Which means we need to sample 385 people.

Work on exercises in packet for Module 2: Survey Fundamentals

Module 3: Questionnaire Design

Step 4: Minimize biases and errors when designing the questionnaire and sampling plan Three major types of biases and errors: Selection bias or coverage error: Your sample is not representative of your population Nonresponse bias: Those who respond to your survey are different in important ways from those who choose not to respond Measurement error: Survey responses are inaccurate

Selection bias or coverage error Definition: Not all members of a population have a known, nonzero chance of being selected for survey Problem: Survey may turn out to be biased Possible Solutions: Identify target population (might require some expertise in the subject of the survey) Construct a sampling frame - a list of all possible respondents Avoid duplicates and respondents that are outside of target population; and excluding a portion of target population Randomize

Nonresponse bias Definition: Survey error that happens when respondents are different from nonrespondents in a significant way Problems: Filters out certain types of respondents The reason for which a person responds (or, conversely, does not respond) to a survey is related to the subject of the survey Possible Solutions: Provide incentives for completing survey Explain why the survey is important Keep the survey short and sweet Give more weight to answers from hard-to-reach respondents (and ask a statistician for advice)

Nonresponse bias In a national sample of board-certified physicians, a short survey was mailed asking physicians to nominate the five best hospitals in their specialty regardless of cost or location. Up to three follow-ups were mailed to nonresponders to gain participation. The final response rate was 47.3%. Males were significantly more likely to respond than females, which would not be an issue if men and women answered in the same way… But, men were significantly more likely to nominate one or two top hospitals in their specialty. In addition, women were significantly more likely to nominate hospitals only in their region.

Measurement error Definition: Inaccurate answers to survey questions (sometimes due to lack of clarity in writing) Problems: Makes it difficult to judge if answers are accurate May lead to incorrect conclusions about target population Possible Solutions: Write clear, concise questions Be aware of leading questions Be aware of social factors that may influence responses Explain why the survey is important

Measurement error In a study about measurement error in earnings data, respondents were asked to report their annual wages. The reported wages were then compared to earnings statements on detailed W-2 records. Not surprisingly, the study found that respondents tended to over-report their wages when compared to their W-2 records. Also, the discrepancy between reported and official wages decreased as official wage increased.

The best way to write good survey questions is to pilot test and re-write An article appearing in the International Journal of Market Research gives great advice about questionnaire design. This YouTube video summarizes 10 things to look out for. http://youtu.be/53mASVzGRF4

Keep the questionnaireasshort as possible The Creative Research Systems has the following useful suggestions: (http://www.surveysystem.com/sdesign.htm) Follow the “KISS” method meaning “Keep it short and simple!” Categorize questions into 3 groups: Must Know Useful to Know Nice to Know If the questionnaire seems too long, start omitting the “nice to know” questions Don’t get caught in the trap where you find that you have a captive audience, so you begin asking questions that are not pertinent

Think about the order of questions Group related questions together Choose first question carefully. The first question should: Apply to everyone Be easy to read Be interesting Place sensitive questions near the end Give respondents a chance to become comfortable with questionnaire Ask about sequential events in the order that they occurred Avoid unintended question order effects

Avoid potential question order effects Priming Early questions refresh respondents’ memory for subsequent questions Carryover Respondents believe questions are similar and answer them with same criteria Consistency Respondents answer questions similarly to try to appear consistent Norm of evenhandedness Respondents answer questions similarly to try to be fair Anchoring Early questions set a standard for comparison to later questions Subtraction Considerations in answers to early questions are left out of subsequent judgments Avoiding extremeness Respondents try to seem neutral by choosing some items while rejecting others

Save demographic questions forthe end of the survey The following demographic questions should be saved for the end of the questionnaire: Age, Education, income, martial status, etc. Ensures that respondents will not feel that they are losing their anonymity when answering the rest of the questions Choose the most important questions for your survey to be asked at the beginning of the survey

Open versus closed questions Open questions allow the respondent to freely answer the question. Imposes fewer restrictions and allows for more depth in the overall answer Closed questions force the respondent to answer the question by choosing from predetermined choices. Advantage: Ease in analysis One suggestion is to test the survey on a small group with an open question. From those responses form a closed question that encompasses the categories expressed in the responses to the open questions. Allow for an “other” option in closed questions, to permit respondents to write their own responses

Avoid double-barreled questions Refrain from having two concepts embedded in one question Example: “Do you have time to read the newspaper every day?” Notice you are asking about “time” and “reading the newspaper every day”. Revision: “Do you read the newspaper every day?” If the answer is no, you can create a question to determine the reasons the person does not read the newspaper.

Convert opinions and words into numbers using the Likert scale A popular technique in survey design is the use of scaling questions. Respondents are able to select a number or category that represents their answer to the survey question. Likert scaling is common technique used in questionnaires. A Likert item is question or statement on a questionnaire where the respondent gives a rating for their response on a topic. The rating is usually the level of agreement the respondent has concerning the statement or question. A Likert item is balanced, meaning there is an equal number of positive and negative positions. http://en.wikipedia.org/wiki/File:Example_Likert_Scale.jpg

Convert opinions and words into numbers using the Likert scale v v The 5-point and 7-point scale responses are the most common Make sure the visual middle option actually corresponds to the middle value: Example: Revision: Likert items can be analyzed separately or the items may be summed and the sum can be analyzed. The sum of Likert items is called the Likert Scale. Disagree Slightly Disagree Neither agree or disagree Slightly Agree Agree Disagree Neither agree or disagree Slightly Agree Agree Strongly Agree

Step 5: Pilot test (and retest)your survey You should pretest the survey on a smaller sample whenever possible! This pilot test can Allow you to revise the questionnaire if needed Allow you to create a closed question from the responses for an open question Help you estimate the variability in the responses to your questions and determine the necessary sample size

Work on exercises in packet for Module 3: Questionnaire Design

Module 4: Survey Implementation

Decide how to collect the survey data After we know what data we want to collect, who we want to survey, and how we ask the questions we must determine the best instrument for collecting the survey data Data collection options: Personal Interviews—either paper or PDA/Smartphone Telephone Interviews Mail Surveys Email Surveys For more discussion of data collection options see http://www.surveysystem.com/sdesign.htm.

Personal interviews A face-to-face encounter between the interviewer and the subject Advantages: People usually respond when confronted face-to-face Can get a better sense of the reaction of the subject Prevent misunderstandings Disadvantages: More costly Interviewers who are not trained properly may introduce bias into the sample

Telephone interviews Most popular instrument for survey in the United States since 96% of homes have telephones Personal interviews and telephone interviews are usually the most successful forms of surveying with response rates around 60 to 75% Advantages: Less expensive than personal interviewing Random phone numbers can be dialed Fast results Disadvantages: People are reluctant to answer phone interviews Phone calls can usually only be made from around 6pm-9pm Phone surveys normally need to be shorter in length than personal interviews

Mail surveys Advantages: Cheap Questionnaire can include pictures People are able to answer on their own time Disadvantages: Timely processes Response rates have a tendency to be low

Email surveys Advantages Cheap Fast You can attach pictures or sound files Disadvantages People may respond multiple times People who have email may not be representative of the population as a whole

In 2011 in Mozambique we interviewed 1600 households using PDAs An on-the-ground statistician ensured the data were of high quality by downloading the data every night, checking them for errors, giving feedback to the surveyors, and correcting any errors found <switch to Prezi presentation about Mozambique survey>

Work on exercises in packet for Module 4: Survey Implementation

Module 5: Reporting the Results

Data must be processed, analyzed, and reported Download or input your data onto a computer Clean your data—start with the most important variables Ensure all data are in the correct format Decide what to do with missing data Detect outliers and coding errors by visual or graphical inspection Process textual data by reading, classifying, and counting

Example of processing data in Excel Pre-workshop survey data downloaded to Excel Clean your data Ensure all data are in the correct format Fix numerical data so they are all numerical Decide what to do with missing data Ignore some missing values Decide which missing values should be 0 Decide which what to do with “I Don’t Know” responses Detect outliers and coding errors by visual or graphical inspection Process text by classifying it and creating a Pareto chart

The best statistical analysis is often just a plot or graph of your data Summarize your data one variable (height) at a time Histograms show the distribution of the data points

The best statistical analysis is often just a plot or graph of your data Summarize your data one variable (height) at a time Box plots summarize the distribution of the data

The best statistical analysis is often just a plot or graph of your data Median Summarize your data one variable (height) at a time Box plots summarize the distribution of the data Outliers? 25th percentile 75th percentile

Summarize two variables based on quant/quant, quant/qual, qual/qual The relationship between two quantitative variables can be visualized in a scatter plot and quantified by correlation or regression The relationship between a quantitative and qualitative variable can be shown in a side-by-side box plot and summarized with a t-test The relationship between two qualitative variables can be shown in a table or a mosiac plot and summarized by Fisher’s Exact Test or a Chi-squared test

Plot two quantitative variableson a scatter plot The relationship between two quantitative variables can be visualized in a scatter plot and quantified by correlation or regression

Use side-by-side box plots to show the difference between two groups The relationship between a quantitative and qualitative variable can be shown in a side-by-side box plot and summarized with a t-test

Summarize qualitative responses by classifying and using a Pareto chart Process textual data by reading, classifying, and counting A Pareto chart orders categories from highest to lowest

Guideline for reportingstatistical results Focus on the statistical estimates of quantities you care about and how they compare to other quantities rather than describing the results of a statistical test Example: The average height of females (n=15) in our sample was 64.9 in. This was 4.5 inches shorter than the average male (69.4 in, n=5). A 95% Confidence Interval for this difference is (-9.4 to 0.7). A t-test showed that this difference was not statistically significant (p=0.08).

Work on exercises in packet for Module 5: Reporting the Results

Work on exercises in packet forModule 6: Questions and Answers What is the question you most want answered today? Can you convince your team that your question should be the one Dr. Vance will answer?

References Dillman, Don A., Jolene D. Smyth, and Leah Melani Christian. Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. 3rd ed. Hoboken, NJ: John Wiley & Sons, Inc, 2009. Lietz, P. (2010) Research into Questionnaire Design. International Journal of Market Research, 52, 2, pp. 249-272. Scheaffer, Richard L., William Mendenhall III, and R. Lyman Ott. Elementary Survey Sampling. 6th ed. Belmont, CA: Duxbury, 2006. http://en.wikipedia.org/wiki/Likert_scale http://www.surveysystem.com/sdesign.htm http://www.csudh.edu/dearhabermas/sampling01.htm http://www.youtube.com/watch?v=53mASVzGRF4 Eric Vance ervance@vt.edu LISA, http://www.lisa.stat.vt.edu

Mozambique Survey Asked people living in villages to rate how “painful” a task it was to fetch water on a 6-point Likert scale (ranging from 1- not painful at all, to 6- extremely painful) Question was given to households in villages both with and without a water pump Some households, especially those without water pumps, must travel hours per day to fetch water How can we best depict the resulting data? Histograms Box-and-whisker plots

Water Fetch Pain

Eric Vance Director of LISA Assistant Research Professor Department of Statistics