Trying Out New Survey Techniques – Lessons Learned

1. Trying Out New Survey Techniques – Lessons Learned 2013 Northwest Hazardous Materials Management Conference Liz Tennant, Strategic Advisor, Office of the Program Director June 25, 2013

2. The Local Hazardous Waste Management Program in King County A regional partnership to protect and enhance public health and environmental quality by reducing the threat posed by the production, use, storage and disposal of hazardous materials.

3. Program Partners • 37 Cities and 2 tribes in King County • King County Water & Land Resources • Public Health--Seattle & King County • Seattle Public Utilities • King County Solid Waste

4. King County, Washington 1.92 million residents 799,000 households 62,000 businesses 2,134 square miles 14th most populous county in the U.S.

6. 2012 Survey Goals • Updated snapshot of residents’ attitudes,awareness and behavior about household hazardous materials and waste. • Update on who is using our collection services, and how satisfied they are with the services. • Better understanding of why people are not using the facilities and how to better reach and motivate them to dispose of hazardous wastes properly.

7. Considerations • Last surveys conducted 5 years ago. • Need for statistically valid samples. • Need to align with increasingly diverse demographics. • Particular concern with input from South King County.

8. Approach Three complementary surveys: • General survey of residents throughout King County • Focus groups of special populations in South King County • Survey of collection customers

9. Decision to Try New Methods • General residential survey using address based sampling instead of random-dialed phone calls • Oversampled the non-Caucasian population in South King County • Supplemented with focus groups

10. What is Address Based Sampling? • Obtain a random sample of addresses in the area you are surveying. • Mail a letter or postcard to the selected addresses. • Provide information about how to respond. • Analyze the responses you receive. • Potentially can adjust the sample.

11. Final Survey Design: • Worked with consultant to design an approach that would meet survey goals. Survey much shorter than 2007. • Letter and paper survey mailed to 10,000 randomly selected households. • Multiple ways to respond: • mail back paper survey (postage provided) • On-line survey in English or Spanish • Toll-free phone number in English or Spanish

12. Final Survey Design: • Oversample of 300 non-Caucasian households in South King County. • Telephone calls to targeted phone numbers (cell and land line) • Monitored responses to ensure demographic alignment. • Built in contingency for random dialed phone calls if needed to get a demographically aligned sample.

13. 2012 General Survey Responses

14. High Level Findings (n=1,270) • Generally people are using the collection sites in their region. • Most people say they don’t have hazardous waste. • Slightly under half say they know about sites; about one third say that they have used them. • Strong support for importance of government providing these services from both collection site users and non-users.

15. South County Focus Groups • Selected 4 non‐English speaking communities based on demographic data: • Spanish‐speaking • Vietnamese • Filipino • (Tagalog‐speaking) Korean Focus Group • Korean • Conducted 5 focus groups (two Hispanic) recruited by and facilitated by a multi‐cultural, multi‐lingual team.

16. South County Focus Groups • Focus group leaders worked to recruit a diverse mixture of people in each 15 person focus group. • Conducted 43 individual interviews in the African American community. • General survey questions were modified for focus groups and interviews. • Materials were translated into the relevant language by each focus group facilitator.

17. Focus Group Findings • Results can’t be generalized due to small sample size. • Communities differ in their approach and level of concern. • Common perception that small amounts of hazardous waste improperly disposed don’t make a difference. • Widespread confusion about what is hazardous. • Confusion about solid waste disposal and hazardous waste disposal.

18. Focus Group Findings • The Spanish community members did not know about the services that we provide, despite our two year Hispanic outreach campaign. • The elderly were unaware of the home collection services we provide. • Generally that services are inconvenient and should be expanded. • The results help illuminate some of the responses in the other surveys.

19. Lessons Learned • Project was too complex with too many moving parts. • Project management schedule was too ambitious. • Both consultants sought revised budgets to cover core activities. • There were problems with communications, oversight and work quality with both consultants. • Schedule regular face to face meetings with consultant at key checkpoints. • Make sure the consultant understands what level of statistical analysis you are looking for and other expectations. • Make sure consultant includes adequate time and budget for writing the report.

20. Lessons Learned - Next Time • Don’t schedule the General Survey and the Focus Groups at the same time. • Develop a realistic project schedule. • Make sure that the consultant includes adequate time and budget to cover: • Needed meetings • Analyzing the results • Writing the report • Schedule regular face to face meetings with the consultant at key checkpoints.

21. Lessons Learned - Next Time Make sure that the consultant understands: • The type and level of analysis that you expect. • They need to translate technical presentation of statistics into commonly understood terms. • They are expected to meet standard report requirements. • An outline of the report must be approved prior to drafting it. • The first draft should not have grammar or spelling errors and should be close to final.

22. Lessons Learned – Next time • Develop focus group questions before or after the general survey questions, not at the same time. • Work closely with the consultant to ensure that concepts are accurately translated and are asked consistently. • Interview consultants to be sure that consultant team members have adequate English proficiency and an understanding of the content that we are trying to collect.

23. 2012 Surveys and Key Findings: http://lhwmp.wordpress.com/category/publications/

24. Questions? For more information: LizTennant@kingcounty.gov 206-284-7974