Measuring the Impact of Entertainment-Education Programs

1. D. Lawrence Kincaid, Ph.D. Center for Communication Programs Bloomberg School of Public Health Johns Hopkins University Measuring the Impact of Entertainment-Education Programs 4th International Entertainment-Education Conference September 26-30, 2004 at the Lord Charles Hotel in Somerset West, South Africa.

2. Between a “rock and a hard place” . . . without a paddle Association or cause and effect? From experimental control by means of research designs To 8 conditions for causal attribution Overview

3. The biggest limitation to the evaluation of communication • Members of the population who watch the program are always different from those who do not watch. (Self-selection bias) • These differences are usually related to the expected outcomes. (behavior, etc.) • So, a simple comparison of outcomes by exposure is biased by the differences between those exposed and not exposed.

4. The Counter-Factual Dilemma • How much does a communication program change the behavior of audience members who are exposed to the program compared to what they would have experienced if they had not been exposed? • Problem: A perfect solution would require a parallel universe. Used only as an ideal. • Randomized experimental design cannot be used for full-coverage programs, so the problem has to be solved with measurement and theory.

5. Solutions for Full-Coverage Programs 1. Statistical Controls: Use multiple regression to control for all confounding variables that might affect exposure and the outcome. 2. Simple Matching: Limited to 1-3 variables. 3. Propensity Score Matching: Unlimited number of variables that might affect exposure and the expected outcomes. 4. Theory-based evaluation

6. A Practical Alternative • The goal is to estimate how much change can be attributed to communication. • Establish the criteria for measuring how much impact can be causally attributed to your program. • Obtain empirical evidence to support each criterion. • Draw an appropriate conclusion.

7. First, how do you know who was really exposed? (implementation) The South African TV Drama, Tsha Tsha • No. of characters audience can correctly name from photographs (k=4) • No. of correct items of knowledge about the drama (k=8) • No. of specific episodes they watched that they can identify(k=26)

8. What is the name of the character shown in each photo below?

9. Knowledge of the Drama Independent of the Expected Outcomes (unaided) • Who is studying for a UNISA degree? • Who steals money from Viwe’s father? • Who is Joy’s father? • What gift does Viwe give to Andile? • Who gets a sexually transmitted infection? • What item does Unathi sell to get money for food? • Who gives Mimi R1,000 for her business? • Who does Cedric send to Johannesburg to deliver dagga?

10. Number of the 26 episodes watched Mark which episodes you have seen: 1. Prince opens a hair salon and interviews girls in the town 2. Mimi and Andile have sex, and she is scared he will wake her grandmother . . 13. Funeral of Andile’s mother . . 26. A party is organized for Andile to dance in Joburg

11. A Continuous Measure of Recall of Tsha Tsha

12. Breaking recall into levels for the analysis of impact

13. 1. Did the outcome change over time? N = 754; not statistically significant 2% of 10,000,000 = 200,000 youth

14. 5. Was the observed change large and abrupt? * Statistically significant: p<0.01 Note: A one-year time interval

15. 2a. Was the change associated with exposure to communication? * Statistically significant: p<0.01

16. 2b. Was the change associated with exposure to communication? Mean Percent

17. 3a. Did exposure occur before the observed change? Wave 3 No Yes Total No 405 140 545 Yes 103 105 208 Total 508 245 753 Wave 1 Decided to Remain Faithful to My Partner Net Increase = 37 Chi-square = 5.63; p<0.05

18. 3b. Did exposure occur before the observed change? Decided to Remain Faithful to My Partner N=753; Chi-square = 46.8, p<.001

19. 4a. Multiple Regression to Estimate the Independent Effect of Exposure after Controlling for Other Influences *South Africa, 2004 Learned about AIDS on TV .13 Female AIDS Attitude .11 .26 Lagged AIDS attitude (wave 1) .13 .09 .33 Education .12 Recall of the Drama .09 Frequency of TV viewing .31 -.01 Kwazulu province residence * Including lagged attitude means that the impact of other variables is on change in attitude.

20. 4b. Propensity Score Matching to Estimate Effects of Exposure of a Drama on AIDS Attitudes Wave 3 South Africa, 2004 Positive AIDS Attitude Variables Used to Construct the Propensity Score for Matching: 1. Age 2. Education 3. Gender 4. Income Level 6. Lagged Aids Attitude 7. Learned about AIDS on TV 8. Province (KwaZulu Natal)

21. 6. Is there evidence of a dose response? AIDS Attitude by Level of Recall

22. 7. Is the causal inference justified theoretically? What is a theory? • How communication works. • A tool for thinking and action.

23. Theory-Based Evaluation • Identify causal pathways (mediating variables for effect) • Demonstrate: • Impact of communication on mediating variables • Impact of mediating variables on behavioral outcome

24. INTENTION BEHAVIOR IDEATION MODEL OF COMMUNICATION AND BEHAVIOR C O M M UNICATION SKILLS & KNOWLEDGE INSTRUCTION reinforcement IDEATION COGNITIVE Beliefs Values Perceived Risk Subjective Norms Self-Image EMOTIONAL Emotional Response Empathy Self-Efficacy SOCIAL Support & Influence Personal Advocacy DIRECTIVE Dissemination Promotion Prescription Attitudes confirmation NONDIRECTIVE Dialogue Counseling Entertainment Social Networks enabling PUBLIC Advocacy Regulation ENVIRONMENTAL SUPPORTS & CONSTRAINTS Source: Adapted from Kincaid (2000)

25. A predictive model of communication & change Implies simultaneous effect of all influences. Knowledge Personal Advocacy Attitudes Social Support & Influence Self-Image BEHAVIOR Perceived Risk Emotions Self-Efficacy Implies communication can effect all influences. Norms

26. Mediating Effects Behavioral Outcome Communication Program Direct and Indirect Impact of Communication on Behavior

27. A Path Model of the Effects of Recall of the Drama on Identification with Boniswa and AIDS Attitudes Identification with a character in the drama indirect effects Attitude towards HIV/AIDS direct effects Level of exposure to the drama direct effects

28. 7. Path Model of the Effects of Recall of the Drama on Identification with Boniswa and AIDS Attitudes at Wave 3South Africa, 2004 Watches “Days of Our Lives” .06 Identification with Boniswa Abstained for a month or more .09 .34 Learned about AIDS on TV .13 .12 .13 AIDS Attitude .11 Female .26 Lagged AIDS attitude .13 .33 .09 Education .12 Recall of the Drama .09 Frequency of TV viewing .31 -.01 Kwazulu province

29. 7. Is the causal inference justified theoretically? YES

30. 8. Are the results consistent with previous program research? YES

31. Eight Criteria for Causal Attribution 1. Change over time in the expected outcome is observed. 2. An association between that change and program exposure is observed. (correlation) 3. Exposure occurs before the observed change is measured. (time-order)

32. Eight Criteria for Causal Attribution 4. No evidence of confounding variables that may have accounted for the change. 5. The observed change is abrupt and large.(immediacy and magnitude) 6. The impact increases in proportion to level or duration of exposure.(Dose response)

33. Eight Criteria for Causal Attribution 7.A causal connection is justified. (causal pathways and theoretical coherence) 8. Consistency with previous program research.(replication with variation)

34. Unfinished Work We can still not adequately measure many of the important effects of EE, especially drama. 1. Expanding the perspective and ways that people make decisions 2. Emotional Involvement (caring, fear) 3. Measuring effects of narrative

35. Measuring the effects of a story The Eagle and the Cocks • Two cocks in the farm yard fought to decide who • should be master. • The loser withdrew to a dark corner, while the victor • flew to the roof top and crowed lustily. • An eagle spied him from high up in the sky, • swooped down, and carried him off. • The other cock came out and ruled without a rival. • PRIDE COMES BEFORE THE FALL. Aesop’s Fable (620 B.C.)

36. FIN Thank you!

37. Why Programs Fail or Appear to Fail • Inappropriate theory • Appropriate theory, but inadequate • application of the theory to the program • Appropriate theory, adequately designed, • but poorly implemented • Appropriate theory, adequately designed, • well implemented, but ineffective research • Appropriate theory, design, implementation, • and research design,but inadequate • measurement of the degree of impact

38. Sources of Program Success • Appropriate theory • Correct application of the theory • to the design of the program • Successful implementation • Adequate evaluation research • Adequate measurement of impact

39. Implications • Five sources are interrelated. • If not effective, it’s difficult to know why. • Program may be effective, but the • research cannot show it. • You can only improve if you know what • happened and why. • There are realistic limits to what we can • learn from program evaluation.