ABCs of Satisfaction & Impact Surveys

1. ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21st , 2008

2. ABCs of Satisfaction & Impact Surveys Objectives • Understand the importance and relevance of science-based surveying • Understand method of developing telephone and Internet surveys to measure inquirer satisfaction and key outcomes important to contact centers • Leave the workshop with the tools necessary to develop surveys tailored to the needs of each contact center • Workshop exercise

3. “Lessons Learned” from surveying for: • Centers for Disease Control & Prevention • Consumer Response Services Center (CDC-INFO) • Monitor calls & e-mail responses • First 5 LA • Countywide Parent Helpline

4. We will discuss: • Survey: • Design • To Do’s • Common Pitfalls • Administration • Options • Data Use

5. Satisfaction & Impact • Satisfaction: • Immediate • Caller’s opinion about • Overall Service • Various Aspects of the Service (Drill Down) • Impact: • What happened as a result of the call?

6. Satisfaction & Impact Surveys: Step 1 • What is important to us when we consider the caller’s perspective and experience? • What questions are our key stakeholders asking that we need to answer?

7. Satisfaction & Impact Surveys: Step 2 • Determine method of administration • Interactive Voice Response • Live Surveyor • Web-Based • Feasibility • Balance between cost & knowledge gain

8. Satisfaction & Impact Surveys: Step 3 Develop questions • Response options • Closed • Yes/No • Likert Scale • e.g. Very Satisfied to Very Dissatisfied, Press 1 for Very Satisfied • Allow a middle option? Neither Satisfied or Dissatisfied • Open • Pro • Information • Con • Labor consideration

9. Satisfaction & Impact Surveys: Step 3 Develop questions • Handout example of closed and open-ended questions

10. Satisfaction & Impact Surveys: Step 4 • Design Survey • Question wording • Neutral • Response order effect • Carefully consider which question is first and which is last

11. Satisfaction & Impact Surveys: Step 4 • Design Survey • IVR Rules • Brief and to the point • No more than six response options • Consider length • Longer = more respondents drop off • Language options

12. Satisfaction & Impact Surveys: Step 5 • Determine Survey Protocol • Who is surveyed-Immediate Satisfaction • Every caller • Works well if no automated transfer option • Training • Bias • Random selection • Works well with a robust system • program algorithm that automatically chooses

13. Satisfaction & Impact Surveys: Step 5 • Who is surveyed—Impact (Call Back) • Again consider: • Purpose of the survey • Caller preference • Part of service follow up effort • How the information will be used • Other considerations: • Labor allocation/resources • To survey • To interpret/make use of information

14. Satisfaction & Impact Surveys: Step 6 • Pilot Testing • Works out the bugs • Improves introductory instructions • May revise questions and responses based on pilot test results

15. What to Expect: Response Rate • Immediate Satisfaction Survey • 22% to 33% • Follow Up Survey • Will reach half of those you try to call • If you try more than once • Vary time/day called

16. Results: Over 90% give the First 5 LA Parent Helpline the highest satisfaction rating for overall service! Over half who received a referral successfully accessed services

17. Results: Over 75% give CDC-INFO the highest satisfaction rating for overall service Over 75% report learning something new from their call Over half report that what they learned makes them want to change a behavior

18. Questions?

19. Exercise: See handouts • Select one question you would like to answer with a satisfaction survey

20. Exercise—Break Out: • Develop a survey question and responses: • Doable/feasible to implement in your setting

21. Exercise—Debriefing: • Share examples from each group

22. Questions? Contact information: Elizabeth Harris, Ph.D. 818-990-8301 eharris@emt.org