Evaluation in Health Promotion

1. Evaluation in Health Promotion Presentation by Irving Rootman to SFU Class on Principles and Practices of Health Promotion November 1, 2010

2. Description of Evaluator • “The evaluator counts the ants at the picnic of progress” (Mohan Singh)

3. Outline • What distinguishes evaluation in health promotion from evaluation in other fields? • RE-AIM Framework

4. Definition of Evaluation in Health Promotion (EWG, 2001) • “Evaluation is about the systematic examination and assessment of features of a programme or other intervention in order to produce knowledge that can be used by stakeholders for a variety of purposes”

5. Principles for Evaluation of Health Promotion Initiatives (EWG, 2001) • Participation • Appropriateness • Multiple methods • Capacity-building

6. Those who have a direct interest in a health promotion initiative should have the opportunity to participate in all stages of its planning and evaluation Encourage the adoption of participatory approaches to evaluation that provide meaningful opportunities for involvement of all those with a direct interest Conclusion-Recommendation(EWG, 2001)

7. Participatory Action Research (PAR) Participative Research Collaborative Inquiry Participatory Rural Appraisal (PRA) Appreciative Inquiry Dialectical Research Conscientizing Research Emancipatory Research Participatory Learning Research Empowerment Evaluation Types of Participatory Research (Green et al., 1997)

8. Definition of Participatory Action Research Systematic investigation… Actively involving people in a co-learning process… For the purpose of action conducive to health** --not just involving people more intensively as subjects of research or evaluation (Green et al., 1997) *

9. To generate knowledge about persons without their full participation in deciding how to generate it, is to misrepresent their personhood and to abuse by neglect, their capacity for autonomous intentionality. It is fundamentally unethical. Heron, J. (1996) Co-operative Inquiry, London, Sage

10. The use of randomized control trials to evaluate health promotion initiatives is in most cases inappropriate, misleading and unnecessarily expensive Support the use of multiple methods Support further research into the development of appropriate approaches to evaluating health promotion initiatives Conclusion-Recommendations(EWG, 2001)

11. Additional Conclusions about Evaluation in Health Promotion (EWG, 2001) 1. Evolving 2. Valuable 3. Not enough 4. Many planning models 5. Multi-disciplinary 6. Evaluators play many roles 7. Theory is essential

12. Dimensions of RE-AIM Model (Glasgow, 2004) • Reach: What % of potentially eligible participants will take part and how representative are they? (Individual Level) • Effectiveness: What impact did the intervention have? (Individual Level) • Adoption: What % of settings and intervention agents will participate and how representative are they? (Setting Level) • Implementation: To what extent are the intervention components delivered as intended? (Setting/Staff Level) • Maintenance: What are the long-term effects (Individual Level); To what extent are intervention components continued or institutionalized? (Setting Level) (Glasgow& Linnan, 2008)

13. Example of Use of RE-AIM Model: Diabetes Self-Management Projects • Both used SCT and Social Ecological Model, similar measures, adult P.C. diabetes patients, similar recruitment methods • Differed in intensity, implementation and other features

14. Example of Use of RE-AIM Model (Cont.) Program 1: In-Office Self-Management • Touch-screen computer program prior to office visit • Followed by R.N. review of action plan • Follow-up phone call • Took 30-45 minutes; 5minutes for R.N. • Used regular staff in 30 clinics in Colorado

15. Example of Use of RE-AIM Model (Cont.) Program 2: Linked Health Coach S.M. • 2 two- hour visits to health educator • Patient worked through problem-solving , computer-administered program to produce action plan

16. Example of Use of RE-AIM Model (Cont.) Comparison of Programs: • Program 1 produced better “reach” • Program #2 produced slightly less improvement on P.A. and H.E. but larger QOL Change • Largest difference in “adoption” with 20% of MD’s willing to participate in #2 v.s. 6% in #1 • Both produced excellent “implementation”

17. Example of Use of RE-AIM Model (Cont.) Comparison (Cont.): • #1 cost $222, #2 $547 per participant • Both improved self-efficacy, perceived support in comparison to controls • Few relations between mediators and outcomes Conclusions: • Likely more health plans would adopt #1 because of lower cost and greater cost-effectiveness • Lower adoption rates of #1 by MD’s needs to be addressed

18. Challenge: Sampling Remedies: Population-Based Recruitment Over-recruitment Report on representativeness Limit exclusion criteria “Reach” Challenges and Remedies (Glasgow and Linnan, 2008)

19. Challenges: Understanding Outcomes Knowledge of mediators Conflicting/ambiguous results Inadequate control conditions Remedies: Assess broad set of outcomes Include mediator measures Sub-group analyses Design control condition to fit question “Effectiveness” Challenges and Remedies (Glasgow and Linnan, 2008)

20. Challenges: Program only studied in optimal conditions Program not adopted Remedies: Involve potential adoptees Approach settings early “Adoption” Challenges and Remedies (Glasgow and Linnan, 2004)

21. Challenges: Protocols not delivered as intended Unable to answer questions about cost, time or staff requirements Deciding if program adaptation is good or bad Remedies: Assess nature of treatment Involve practitioners in program design Vary staff characteristics, and evaluate staff impact and costs Specify critical theoretical components and elements that can be adapted “Implementation” Challenges and Remedies (Glasgow and Linnan, 2004)

22. Challenges: Program effects not maintained Attrition of settings, delivery staff and participants Remedies: Include maintenance phase in plan Plan for institutionalization and sustainability Take steps to evaluate and report on attrition “Maintenance” Challenges and Remedies (Glasgow and Linnan, 2004)

23. RE-AIM: Strengths/Weaknesses Strengths: • Comprehensive • Focus on population impact • Multi-level • Relatively easy to use • Useful for program and policy development • Builds on other theories at different levels Limitations: • Not much research using model • Not necessarily “participatory”

24. RE-AIM: Implications for Aging • Has been used in studies of older adults in relation to: • Physical Activity • Chronic disease management • Nutrition • Heart Disease prevention

25. Example: Home-based Exercise Purpose: Evaluate home-based exercise program for older adults Methods: 105 frail homebound older adults recruited from 10 Faith in Action Sites; Volunteer trainers assisted subjects; Surveys at baseline and after 4 months in program; RE-AIM used as conceptual framework; focused on “adoption” and “implementation” Findings: Participants exercised average of 2.2. times per week; improved social functioning Conclusion: Evidence for A and I components of model Etkin, C.D. et al., 2006

26. References • Etkin, E.D., et al. (2006). Feasibility of Implementing the Strong for Life Program in Community Settings, The Gerontologist, 46: 284-292. • Glasgow, R.E. and Linnon, L.A. (2008). Evaluation of Theory-Based Interventions. In Glanz, et al., Health Behavior and Health Education: Theory, Research and Practice, Forth Edition, San Francisco: Jossey-Bass. • Green et al., Participatory Research…Ottawa: Royal Society of Canada, 1997. www.lgreen.net/guidelines.html • Rootman, I., Goodstadt, M., Hyndman, B., McQueen, D., Potvin, L., Springett, J., Ziglio, E. (Eds.),(2001). Evaluation in Health Promotion: Principles and Perspectives, Copenhagen: European Regional Office of the World Health Organization.