A Journalist’s Guide to Survey Research and Election Polls

1. A Journalist’s Guide to Survey Research and Election Polls Cliff Zukin * Rutgers University AAPOR The American Association for Public Opinion Research Washington Press Club Briefing 9/24/12 *This presentation reflects the author’s views. It has not been vetted or endorsed by AAPOR.

2. First, can we trust polls: 2004? http://www.realclearpolitics.com/epolls/2004/president/us/general_election_bush_vs_kerry-939.html

3. Can we trust polls: 2008? http://www.realclearpolitics.com/epolls/2008/president/national.html

4. What AAPOR Thinks You Should Know • Who did the poll? Who paid for it? • When was it done? • Who was sampled? • How were respondents contacted? • What is sampling error? • Why are data weighted? • How is the question worded? • How are the questions ordered?

5. What You Want to Know • How are margins of error determined, and what does the error rate mean? • To what extent are traditional polls being replaced by targeted cell phone or Internet polls? • Discuss Internet polls and their validity. • Any new polling techniques for 2012? • Explain weighting, and how subjective it is, and how to evaluate it. • What are the most common errors made by political journalists in writing about polls?

6. Our Roadmap (Through the Methods Box. See the handout. ) • Sampling—why polls work: Good and Bad Samples • Developments and Challenges facing the SR profession: cell phones, IVR (robo-polls), etc. • Question wording & ordering • Things you want to be wary of…SR for journalists • Election Polling—How it’s different & why polls differ

7. SAMPLING: The Science of Polling http://faculty.elgin.edu/dkernler/statistics/ch01/3-1.html

8. All Scientific Polling is Based on the Notion of Sampling • Bloody Whiners I’VE NEVER BEEN CALLED • Polls are estimates • A sample is drawn to represent an underlying population • The sample must be representative • If you don’t do this well, the rest doesn’t matter

9. Telling Good Polls and Bad Polls Apart Starts with the Sample • Probability Samplesare GOOD • Scientific—known chance of inclusion • Random, or thereabouts, selection • Generalizable from sample to population • Non-ProbabilitySamples are BAD • Not representative: 1,200 people are not a sample of anything • Person in the street • ALL Internet opt-in surveys • Any self-selection

10. Journalistic Standards • In order to represent the population statistically, asurvey should be based on a probability sample. -NY Times • Methodologically, inall or nearly all cases we require a probability sample, with high levels of coverage of a credible sampling frame. Self-selected or so-called “convenience” samples, including internet, e-mail, “blast fax,” call-in, street intercept, and non-probability mail-in samples do not meet our standards for validity and reliability, and we recommend against reporting them -ABC/Washington Post

11. Probabilities, Better Known as The Odds: Why Polls are Accurate…We’re the House 36 Total Outcomes

12. Dice Outcome(Or, the Central Limits Theorem)

13. Probabilities: Why Polls are Accurateor “We’re the House” • Chances of getting a 7 = 17% average or mean (of 100 rolls (100%)/ 6 ways = 17%) • Chances of getting between 6 and 8 = 45% (14% + 17% +14% = 45%) • Chances of getting a 5 through 9 = 67% (11% + 14% + 17% +14% + 11% = 67%) • So, the chances of being within 4 points on either side of the expected (mean) is 95%. Or the margin of error is + 4% points

14. Probability(good) Samples Depend on Mathematical Axioms: The Central Limits Theorem & the Law of Large Numbers • Life (or a sample of life) is distributed normally and generally fits a bell-shaped curve Central Limits • The more observations you have, the more your distribution will approximate a bell-shaped curve Large numbers • After a certain point, sample size matters, but population size doesn’t. Law of M&Ms

15. At a Secret Location in Hacketstown NJ…

16. …I Counted the M&Ms in the Last Slide

17. Relationship between Sample Size and Sampling Error

18. Sampling Error • Sampling Error is a theoretical minimum • It is only one kind of error, but it is quantifiable • It applies not to the GAPbetween candidates, but to each point estimate MOE ± 3

19. SO WHAT? Why is this Important? • INTERNET POLLS ARE NON-PROBABILITY SAMPLES (WITH ONE EXCEPTION) • It is not possible to calculate sampling error on a Non-Probability Sample • And even with lipstick, a pig is still a pig Ipsos Poll for Reuters DAILY ELECTION TRACKING 9/11/12: These are findings from an Ipsos poll conducted for Thomson Reuters from Sept 7-11, 2012. For the survey, a sample of 1,269 American registered voters (age 18 and over) was interviewed online. The precision of the Reuters/Ipsos online polls is measured using a credibility interval. In this case, the poll has a credibility interval of plus or minus 3.1 percentage points for all respondents.

20. It’s Not Just Sampling Error:Total Survey Error Comprises 4 Sources • Sampling Error: Sample, not population surveyed • Coverage Error: Sample may not map to pop • Measurement Error: Question wording/ordering • Non-response Error: Many decline to be surveyed

21. Trends in Survey Research

22. Trends in the Survey Research Industry • Cell phones • IVRs (Robo-Polls) • Fewer high-quality polls • Challenge of interviewing a representative sample • Increased cost of finding respondents and dialing

23. Percentage of Wireless Only Adults 2001-2005 *Data Source: CDC/NCHS National Health Interview Survey

24. Percentage of Wireless Only Adults 2001-2011 *Data Source: CDC/NCHS National Health Interview Survey

25. Percent of Adults Ages 18-29 Source: Current Population Survey and Pew Research Center surveys

26. RDD Samples without Cell Phones – Age Bias

27. Phone Status of Adults, Dec 2011 Among those households with telephones… • There are four times more “cell phone only” (32%) in the population as “landline only” (8%) • Taken together, fully half of the public is “cell phone only” (32%) or “cell phone mostly” (18%) • Did you know: Most IVR Polls do not include cell phones in their samples? This is definitely a question you should ask before reporting

28. Journalistic Standards: NY Times • Interactive Voice Response Polls — Interactive voice response (IVR) polls (also known as "robo-polls") employ an automated, recorded voice to call respondents who are asked to answer questions by punching telephone keys. The Times does not publish IVR polls. • Internet Polls — Non-probability samples are commonly used in Internet polls, call-in polls, blast e-mail polls and a variety of others. The entire population does not have an equal chance of being contacted. Most Internet polls are based on panels of self-selected respondents. The Times does not publish most Internet polls.

29. Manipulating Public OpinionOr can you make a poll find anything you want? NO! BUT question wording and question order are extremely important

30. Good Questions: • Are simple, direct, clear to all, and avoid jargon • Don’t presume information • Are balanced • Ask about only one thing • Don’t tax respondent’s memory or cognitive ability

31. Bad Questions: • Are complex or presume information • Are leading or unbalanced • Are double-barreled or double negative • Are loaded through emotional or red flag words • Give biasing or unequal information in the Q stem

32. Bad Questions: 1. Are complexor presume information: Do you favor or oppose a 1 percent tax by the United Nations on international air travel which by itself would produce half of all the revenue the organization needs and a savings for all members, which for the United States would be about half a billion dollars a year. The tax itself would add about a billion dollars to America’s international travel and air freight bills. Do you favor or oppose that? 32% Strongly favor 30% Somewhat favor 11% Somewhat oppose 25% Strongly oppose 2% Don't know/No answer

33. Bad Questions: 1. Are complexor presume information: President Obama has announced changes to federal student loan programs that would: allow some college graduates to limit federal student loan repayments to 10% of their discretionary income starting in January, two years before the cap was due to take effect under federal law; forgive remaining debt on the federal loans after 20 years, five years earlier than under current law; and allow those with more than one student loan to consolidate their debt. Do you approve or disapprove of these changes?

34. Getting Real:Citizen Knowledge and Attentiveness • 17% follow “news about political figures and events in Washington very closely” • 28% named John Roberts as Chief Justice out of four choices (7/10) • 43% know Republicans control the House (11/11) • 53% Know it is the Republicans who want to reduce the size of government Source: Pew Research Center

35. Bad Questions: 2. Are leading (or unbalanced): Are you in favor of the mandatory drug testing of professional athletes? BIASED Rather, Do you favor or oppose the mandatory…UNBIASED or … professional athletes, or not?UNBIASED

36. Bad Questions: 2. Are leading (or unbalanced): Should the federal government increase its role in providing loans with a goal of making sure everyone who puts in the effort to graduate from college can afford to do so? • It’s okay for you to ask leading questions of experts, but not for us to ask them of random folks Totals may not add up to 100% due to rounding.

37. Bad Questions: 3. Are double-barreledordouble negative: Did you vote in the 2008 and 2010 elections – yes or no?

38. Bad Questions: 3. Are double-barreledordouble negative: Do you want to see less money spent on defense and more on social problems?

39. Bad Questions: 3. Are double-barreledordouble negative: (Agree or disagree) Should the federal government no longer be involved in college loans, and instead leave that entirely to the private sector?

40. Bad Questions: 4. Areloaded through emotional or red flag words Fifty state legislatures have passed resolutions calling on Congress to pass the flag amendment and send it to the states for ratification. Do you think members of Congress who may personally oppose the amendment should vote in favor of it anyway so that “we the people” can decide the issue?

41. Bad Questions: 4. Areloaded through emotional or red flag words Do you believe in killing unborn babies?

42. Bad Questions: 5. Through giving one-sided information If it would result in increased opportunities for educating New Jersey citizens, would you favor or oppose building a new TV transmitter at liberty science center?

43. Bad Questions: 5. Through giving one-sided information Many gambling experts believe that internet gambling will continue no matter what the government does to try to stop it. Do you agree or disagree that the federal government should allocate government resources and spend taxpayer money trying to stop adult Americans from gambling online? Balance is necessary: Some people say…while others say…

44. ALASKA, or (I’ll Ask her) • Zogby: for environmental interests Do you think oil companies should be allowed to drill for oil in America’s Arctic National Wildlife Refuge? Allow 38%Not allow 55% 17 percentage points against drilling

45. ALASKA, or (I’ll Ask her) • Luntz: for energy interests And right now, if you had to choose, do you strongly support, somewhat support, somewhat oppose, or strongly oppose the exploration and production of oil and natural gas from the ANWRas one of the many ways to increase our nation’s energy supply? Support 51%Oppose 34% 17 percentage points pro-drilling

46. ALASKA, or (I’ll Ask her) • Harris poll: How much would you support or oppose the following items that might be on the agenda for the new Congress? Energy reform to allow companies the ability to drill for oil in certain areas such as the Alaska National Wildlife Refuge to decrease our reliance on foreign oil? Support 53%Oppose 47% 6 percentage points pro-drilling

47. Maxims of Question Wording • Question wording matters most when respondents know the least • Projections and hypotheticals are generally unreliable • You can never really underestimate how much people know about politics and government • Giving information before asking the question is a double-edged sword • It must be balanced--as easy to disagree as to agree • No ONE question tells the story

48. CONTEXT EFFECTSEach Question Affects the Following Ones • How important is Social Security to you? • Do you think Social Security will be there when you need it? • Are you worried about private investment options for Social Security? • What is the most important problem facing the country?

49. WHAT’S THE TOPIC HERE…. • “The First Amendment of the United States applies to everyone regardless of gender, race, religion, age profession, or point of view.” (96%) • “The First Amendment protects the right of individuals to create a private organization consisting of a specific group of people based on age, gender, race, ethnicity, or interest.” (77%) • “The First Amendment protects the right of organizations like the Boy Scouts, the Girl Scouts, and the National Association for the Advancement of Colored People to exist.” (91%)

50. IT’S BEEN IN THE NEWS RECENTLY…. • “Individuals have a right to join a private group, club, or organization that consists of people who share the same interests and personal backgrounds as they do if they so desire.” (91%) • “Private organizations that are not funded by the government should be allowed to decide who becomes a member and who does not become a member on their own, without being forced to take input from other outside people or organizations.” (76%)