Developing evaluation frameworks for large, multi-program projects:

1. Developing evaluation frameworks for large, multi-program projects: Building logic “meta-models” and visual databases Jason Newberry, Andrew Taylor, Robert Case, and Kristen Roderick Centre for Research and Education in Human Services general@crehs.on.ca

2. Overview of Presentation • How large government policy frameworks create new challenges for health and social service evaluators • Ontario’s Early Childhood Development initiatives: A Work in Progress • The development of an evaluation framework: logic “meta-models” and visual databases

3. Please Note: • Although this presentation draws on examples from our ongoing work with programs run by the Ontario government, the specific content of these examples is intended only to illustrate the evaluation process we have designed and does not reflect the official position of the Ontario Government regarding these programs.

4. A New Reality for Evaluators • In the public sector, there is a greater need for evaluation because of: • A rebirth of interest in the social determinants of health and primary prevention. • A new emphasis on results and evidence-based management. • A new acknowledgement of the need to let different regions develop unique approaches to addressing shared priorities. • Evaluators are now confronted with the task of evaluating the impact of many diverse programs that fall under very broad funding strategies. • This requires developing evaluation frameworks and strategies that are capable of evaluating very different programs that have been created to achieve similar goals.

5. A Specific Example: Early Childhood Development Initiative in Ontario • Federal/Provincial funding of programs that broadly target early childhood development (0-6 years) and encourage different jurisdictions to develop their own approaches • In Ontario, 26 different provincial initiatives are funded • Most initiatives have multiple service sites (e.g., one initiative has over 100 program sites) • Within initiatives, each site may offer different services, or have a different focus • Types of interventions are very diverse. For example: • Direct clinical treatment of children • Parent education and skill development • Pre- & post-natal services • Promotional campaigns, awareness-raising • Social and practical support

6. A new emphasis on public sector accountability may lead to questions like…. “Are funded Early Years programs in the province effective in improving early childhood development?”

7. Beginning to answer “meta- questions” New frameworks are needed that can lead to… • ways of speaking to meta-level evaluation questions like “to what degree did diverse, multi-site programs function as components in a larger coordinated plan?” • ways of comparing and consolidating the theories of change underlying diverse interventions with similar goals. • ways of comparing and consolidating the measurement approaches to diverse interventions.

8. What are the barriers? • Information overload and difficulty in organizing that information. • Information “underload” where evaluators are twice and thrice removed from the field. • Neither of the two “traditional responses” works: • Large, centralized, standardized evaluation systems do not transfer well to the field; exceedingly difficult to design with diverse programs. • Decentralized approaches which encourage each program or community to do their own evaluation often may promote poor quality evaluation designs; makes very difficult to be summative across programs.

9. What are the solutions? • Blended approaches, which emphasize shared theory of change and shared definitions of key evaluation terms, but allow for program-to-program and region-to-region diversity in how measurement is carried out. • Using available technology to facilitate the creation, understanding and ongoing coherence of evaluation frameworks.

10. Federal/Provincial/Territorial ECD Agreement Four broad target areas: • promote healthy pregnancy, birth and infancy; • improve parenting and family supports; • strengthen early childhood development, learning and care; and • strengthen community supports.

11. The Main Activities and Outcome Objectives of Early Childhood Development Initiatives • Resources and Tools • Networking • Training • Program Effectiveness • Public Education • Community Advocacy & Leadership • Parent Education • Personal Support • Assessment & Counselling • Interactive Learning • Clinical Care, Nutrition • Screening & Assessment • Referral Child Health Service System Community Health Family Health Activities Outcome Objectives • Knowledge • Skill • Use of Services • Parenting Behaviour • Support • Independence • Knowledge • Awareness • Commitment • Support • Knowledge • Collaboration • Consistency • Capacity • Access • Use of Services • Stress • Early ID • Behaviour • Readiness

12. Training • Public Education • Parent Education • Interactive Learning • Knowledge • Skill • Parenting Behaviour • Support • Independence • Knowledge • Awareness • Commit-ment • Support • Knowledge • Collaboration • Consistency • Capacity • Access • Use of Services • Behaviour • Readiness Examples of Preventive Activities and Associated Outcome Objectives in Childhood Development Initiatives Child Health Community Health Family Health Service System

13. Use of Services • Parenting Behaviour • Support • Use of Services • Early ID • Behaviour • Readiness Example of Assessment and Treatment Activities and Associated Outcome Objectives in Early Childhood Development Initiatives • Personal Support • Assessment & Counselling • Screening & Assessment • Referral Child Health Community Health Family Health Service System

14. Knowledge • Skill • Use of Services • Parenting Behaviour • Stress • Early ID • Behaviour • Readiness Activities often have Outcomes at Multiple Levels • Parent Education Child Health Community Health Family Health Service System

15. Activity or Program Activity or Program Activity or Program Outcome Evaluation Short-Term Outcome Objective Short-Term Outcome Objective Long-Term Outcome Objective Long-Term Outcome Objective Long-Term Outcome Objective Priority Goal Priority Goal Evidence Based Planning Vision or Plan Implications for Practice

16. Beginning with Program Maps

17. Beginning with Program Maps

18. Sample Program Logic Model Program and System Supports Early Childhood Development, Learning & Care Pregnancy, Birth, and Infancy --- Parenting & Family Support Providing personal, practical support to parents Training of, consulting with, service providers Evaluating provincial NP services & community impact Ongoing parental awareness media campaigns Developing & implementing service coordination policies, protocols Creating, disseminating child development resource materials Promotional advocacy campaigns Developing networks of service providers Organizing community events ACTIVITIES Providing referral info & hotline Providing professionally led support groups for parents Providing parent education & training Providing parent screening & assessment Provide small learning groups for children Providing counseling to parents Providing child screening & assessment Improved identification of at-risk parents, infants and children (PPD, P4, developmental delay, etc). Increased capacity to assess program planning & effectiveness Increased service provider knowledge of pregnancy & child development supports & services/gaps Increased SP knowledge & skills of pregnancy & child development best practices & interventions Increased service provider collaboration Increased feelings of family & social support Increased child stimulation & interaction Improved parenting knowledge and practices (pre- & postnatal care, nutrition, child development, parenting skills, health/safety) Improved knowledge of supports & services for healthy pregnancy & infant/child development SHORT-TERM OBJECTIVES Increased service provider, parent, & community awareness of the importance of the early years Increased parent & community awareness of pregnancy & child development services Enhancing & expanding core programming Improved emotional functioning & well being of children Increased entry into appropriate supports & services Increased parenting capacity Increased use of other supports & services Increased accessibility of pregnancy & child development intervention programming LONG-TERM OBJECTIVES Increased coordination & integration of services in community Increased general awareness of the importance of parenting Improved child health & physical well-being Increased child prosocial behaviours Improved school readiness Improved infant health & well-being Improved maternal health Increased community participation in EY & prevention activities Increased commitment, support, programming for EY services GOALS Promote children’s optimal development and readiness to learn, so that they may reach their fullest potential Promote Healthy Pregnancy & the Birth of Healthy Babies For all children to be healthy, safe, and secure within strong families and communities Link to Map

19. Coding Initiative Maps

20. Coding Initiative Maps

21. We can examine which initiatives share common activities and the different service approaches used...

22. Go to Roll Up Go to Pregnancy, Birth, & Infancy Model Go to Parenting & Family Support Model Go to Community, System, & Programming Supports Model Overall Logic Model: Early Childhood Development, Learning & Care (column 3)) Other direct service columns ACTIVITIES Providing screening, assessment, & monitoring for children Providing counseling, treatment, & crisis for children Providing physical activity programs for children Providing nutrition programs to children Creating links to other resources, services, & programs Providing interactive learning,arts, & play for children Increased use of other supports & services SHORT-TERM OBJECTIVES Decreased stress of children Increased child awareness (initiative specific – e.g., injury prevention) Improved parent-child/family communication Improved child behaviour Increased early identification & diagnoses of children Miscellaneous initiative & system specific objectives (see unique logic models) LONG-TERM OBJECTIVES Improved cognitive ability of children Improved social skills, communication & behaviour of children Improved health & physical well-being of children Improved emotional functioning of children GOALS GOALS

23. Overall Logic Model: Early Childhood Development, Learning & Care (column 3)) Other direct service columns ACTIVITIES Providing screening, assessment, & monitoring for children Providing counseling, treatment, & crisis for children Providing physical activity programs for children Providing nutrition programs to children Creating links to other resources, services, & programs Providing interactive learning,arts, & play for children Increased use of other supports & services SHORT-TERM OBJECTIVES Decreased stress of children Increased child awareness (initiative specific – e.g., injury prevention) Improved parent-child/family communication Improved child behaviour Increased early identification & diagnoses of children Improved social skills, communication & behaviour of children Miscellaneous initiative & system specific objectives (see unique logic models) LONG-TERM OBJECTIVES Improved cognitive ability of children Improved health & physical well-being of children Improved emotional functioning of children GOALS Go to Roll Up GOALS Go to Pregnancy, Birth, & Infancy Model Go to Parenting & Family Support Model Go to Community, System, & Programming Supports Model

24. We can examine which initiatives share common short-term and long-term objectives

25. We can move from the roll-up to the unique logic models of particular initiatives. These contain customized versions of shared objectives in the roll-up (in this example, P6)

26. Prenatal and Postnatal (information, education and support) Providing a variety of targeted programs for special populations, including teen parents Providing funding for community based projects that meet identified criteria Providing ongoing support for funded projects, and support for proposal development Supporting local planning and proposal review Engaging local non-traditional partners as champions, funders and committee members (e.g., business) Creating community resources Providing ethnically and culturally appropriate programs Providing school readiness programs Providing programs which support children’s healthy development Creating and running awareness campaigns Providing parenting services (information, education and support) Increased linkages between service partners and alternative resources Increased community collaboration/linkages among service providers and other supporters of parents Increased new and locally-created programs to fill gaps Increased access to services for targeted populations Increased awareness of avail community services Increased community participation in early years activities Increased size and density of families’ social networks Increased use of services and supports Reduced parent stress Increased fit between (new and existing) services and local priorities and identified needs Increased knowledge among service partners about local gaps and priorities Increased awareness in the community at large of the importance of EY Reduced child stress Increased diversity of funding and supports Increased opps. for children to participate in activities that promote healthy development Increased use of services and supports Improved/ smoother transitions to school Increased parenting knowledge and skill Improved knowledge of avail. supports & services Improved program quality Decreased duplication of services Increased parenting capacity Increased commitment/support for EY services More dense network of inter-relationships among EY services and other family supports Increased general awareness of the importance of parenting Activities Improved child health and physical well-being Improved social maturity of children Improved communication skills of children Improved sustainability of services Short term objectives Improved language and cognitive skills of children Improved social skills/competency of children Long term objectives EY goals P6 goals Stronger sense of shared philosophy for young children and their families Increased access to a more cohesive, comprehensive range of EY programs, services and parenting supports Improved quality and capacity of existing children’s services Promote children’s optimal development and readiness to learn, so that they may reach their fullest potential

27. Prenatal and Postnatal (information, education and support) Providing a variety of targeted programs for special populations, including teen parents Providing funding for community based projects that meet identified criteria Providing ongoing support for funded projects, and support for proposal development Supporting local planning and proposal review Engaging local non-traditional partners as champions, funders and committee members (e.g., business) Creating community resources Providing ethnically and culturally appropriate programs Providing school readiness programs Providing programs which support children’s healthy development Creating and running awareness campaigns Providing parenting services (information, education and support) Increased linkages between service partners and alternative resources Increased community collaboration/linkages among service providers and other supporters of parents Increased new and locally-created programs to fill gaps Increased access to services for targeted populations Increased awareness of avail community services Increased community participation in early years activities Increased size and density of families’ social networks Increased use of services and supports Reduced parent stress Increased fit between (new and existing) services and local priorities and identified needs Increased knowledge among service partners about local gaps and priorities Increased awareness in the community at large of the importance of EY Reduced child stress Increased diversity of funding and supports Increased opps. for children to participate in activities that promote healthy development Increased use of services and supports Improved/ smoother transitions to school Increased parenting knowledge and skill Improved knowledge of avail. supports & services Improved program quality Decreased duplication of services Improved social maturity of children Increased parenting capacity Improved communication skills of children Increased commitment/support for EY services More dense network of inter-relationships among EY services and other family supports Increased general awareness of the importance of parenting Activities Improved child health and physical well-being Improved sustainability of services Short term objectives Improved language and cognitive skills of children Improved social skills/competency of children Long term objectives EY goals P6 goals Stronger sense of shared philosophy for young children and their families Increased access to a more cohesive, comprehensive range of EY programs, services and parenting supports Improved quality and capacity of existing children’s services Promote children’s optimal development and readiness to learn, so that they may reach their fullest potential

28. The ECD Visual Data Base • has led to clearer and more accurate framework and terminology for summarizing early years activities and objectives. • has allowed us to describe activities in terms of both content and service delivery modality.

29. The ECD Visual Data Base • allows us to talk about the focus of various types of work within the Early Years plan, not just presence and absence. • helps us to work towards cross-initiative consistency in understanding important outcome objectives and how they relate to one another.

30. The ECD Visual Data Base • allows ministry staff to quickly and easily understand individual initiatives and the connections between them. • contributes to a better understanding of how initiatives fit into the 4 priority areas the Federal/Provincial agreement.

31. Moving to measurement • We have recently entered a phase that moves the framework towards measurement • We are repeating a process that is similar to the generation of the program maps. • Each initiative is completing a table of indicators, data items & sources, and design/analysis recommendations, that are linked to each objective

32. Moving to measurement • These tables will also be electronically linked to the visual database (the roll-up table and logic models)

33. For a given initiative, we can move from a logic model to the roll-up, and from there to the indicator table to examine specific evaluation questions, methods, and measures. These may be transferable to initiatives sharing common objectives...

34. Table of Indicators, Measurement, and Design

35. Table of Indicators, Measurement, and Design

36. The Indicator library • This current phase of developing indicators, examining current evaluation data, and proposing future evaluation activities has led to the development of an “indicator library”. • The indicator library is analogous to our development of the roll up. While the roll-up summarized ECD activities and objectives, the indicator library summarizes measurement and design across all the initiatives.

37. The Indicator library • facilitates cross-site collaboration within initiatives but also collaboration across initiatives. • allows for the possibility of consistency of measurement across programs that share common objectives, programs models, and populations. • fosters best practices in evaluation.

38. The lessons learned • We began by building consistent and concrete theories of change across diverse situations, rather than trying to impose a consistent measurement system. • We did this work with local programs and broad province-wide systems at the same time and used each level to clarify the other. • This strategy has allowed us to use an incremental approach to improving measurement and reporting practices at many levels and over time, which is cost-effective and builds ownership.

39. Local, flexible site level evaluations Initiative sites Initiative sites Initiative sites Provincial level evaluations of initiatives Outcome evaluation frameworks & measurement Overall meta logic models (1 for each priority area) The Roll-up of activities and objectives Initiative logic model Initiative logic model Initiative logic model Initiative map Initiative map Initiative map