Research and Evaluation: Grantee Panel

1. Research and Evaluation: Grantee Panel

2. Objectives • Share examples of how programs are building evidence at various stages of the evidence continuum • Highlight diverse grantees and subgrantees • Small and large • Focus area representation • Multi-site/multi-issue programs • Rural

3. Building Evidence of Effectiveness Evidence Based Attain strong evidence of positive program outcomes Evidence Informed Obtain evidence of positive program outcomes Assess program’s outcomes Ensure effective implementation Identify a strong program design

4. Panelists • Fa’izRab, AmeriCorps Program Manager, Lutheran Family Services • Julie McClure, Director, Napa County Office of Education Community Programs • Amy Foss, Vice President, Conservation Legacy • Karen Domerski, General Counsel, Jumpstart

5. Nebraska Economic Opportunity Corps(AmeriCorps State and National) Support | Educate | Empower The Nebraska Economic Opportunity Corps is an AmeriCorps State program of Lutheran Family Services of Nebraska, Inc. Our mission is to improve the economic opportunities of at-risk youth, new Americans, and veterans through education and employment programs that empower individuals towards self-sufficiency.

6. NEOC Performance MeasuresSelf-Sufficiency Matrix Outcomes are measured through the use of an adapted version of the Snohomish County Matrix developed by the Snohomish County Self-Sufficiency Taskforce in 2004*. • Assessed domains are tied to factors impacting improved employment and education prospects. • Beneficiaries scored on a five level scale from “in- crisis” to “thriving.” *The taskforce was collaborative partnership of the county’s human services community including the Snohomish County Community Action Division of the Human Services Department, United Way of Snohomish County, and community partners.

7. NEOC Self-Sufficiency Matrix

8. Benefits of the Matrix Flexibility – Customize for individual host sites intervention method. Collaboration – Design encourages a collaborative effort among providers in defining improvement. Case Management – Can be used for both long term and short term interventions. Self-Assessment – Keeps the member focused on the goal of self sufficiency. Management – Assists in determining what is or is not working. Communication – Helps communicate provision of service. Measurement – Measures beneficiary progress.

9. Challenges • Lack of understanding of how to complete the Matrix • Technology failures • Collecting reports • Basic math issues • Counting assessments

10. Overcoming Challenges • More training and communication of requirements • Researched and experimented with different reporting methods, which led to using Google for Nonprofits

11. Google for Nonprofits Free service: Nonprofits qualify for business level service at no cost. Improved communication: Multiple tools including email, chat, and Google Hangouts. Less paper: Assessment forms are included on a separate tab for easy access. Security: Encryption of mobile devices using our account. Reduces reporting obstacles: Members may complete assessments through their phone.

12. Example Google Sheet

13. Going Forward • Improve methods to capture outcomes. • Work out any issues that arise from this year. • Create process and assessment of beneficiaries’ perspective on services provided. • Potentially using online survey methods or randomized follow-up calls from program staff.

14. CalSERVES: AmeriCorps Programs ofNapa County Office of Education

15. CalSERVES AmeriCorps Programs • After School: Local Direct Service, 34 FT/60 HT • Collective Impacts: Local Direct Service, 20 FT/36 HT Members • PREP: State-wide Capacity Building, 40 FT Members • VIP: State-wide Capacity Building, 100 FT Members

16. What We Did • Develop Evaluation Goals and Questions • Employ an Evaluation Firm that Understands AMC • Determine the Method of Evaluation • Determine Strategy for Comparison Group • Conduct Analysis • Report Findings • Discuss Program Changes and Next Steps for Research

17. Evaluation questions • Does participation in the CalSERVES VIP program increase organizational capacity to utilize volunteers compared to organizations that did not take part in the program? • Does participation in the CalSERVES VIP program improve organizational capacity concerning volunteer recruitment, training, and retention compared to organizations that did not take part in the program? • Does participation in the CalSERVES VIP programimprove organizational capacity to create and sustain successful volunteer programs compared to organizations that did not take part in the program?

18. VCA Instrument • To measure the effect of program participation on organizational and volunteer capacity over time, data were collected using the Volunteer Capacity Assessment (VCA), a survey that includes three sections with items focused on: organizational capacity, volunteer recruitment, and elements of a successful volunteer program. • The VCA is designed to assess how well a non-profit or educational organization is prepared for recruiting, training, and utilizing volunteers to achieve its mission and goals. The instrument asks for organizations to report how well their practices align with the best practices listed on the survey.

19. Propensity Score Matching List of Characteristics Used in Propensity Score Matching

20. What the Results Showed Average Pre/Mid VCA Score and Average Post VCA Scores, Matched Comparison

21. Conclusion “Overall, the partner sites that participated in the CalSERVES VIP program reported strong, positive changes over time on the items included in the VCA instrument. Organizations that did not participate in the program reported much smaller (and sometimes negative) changes on the VCA items. Because the matching procedure diminishes the likelihood that factors other than participation in the program influenced the change in reported scores over time, the evidence strongly suggests that the CalSERVES VIP program positively impacts organizations’ capacity concerning volunteer programs.”

22. Challenges • Finding the right evaluation firm • Willingness of comparison sites to participate • Resources to fund evaluation

23. Successes • Kept costs reasonable using comparison group design. • Able to develop and implement a comparison group design that allowed for the development of some real understanding of impact. • Able to use data to inform the program and determine next steps.

24. Next on Our Research Agenda • Examine the type of organizations (both in size and scope of work) that benefit the most from the CalSERVES VIP program. • Explore the specific types of services that are most beneficial to these organizations.

25. Conservation Legacy: A Progressive and Phased Approach Amy Foss

26. Overview – Our Projects and Past Evaluations What problems do our land management agencies face? • Multiple uses that impact the land, how to maintain natural resources that communities and human use, preservation, habitat and species protection Our Projects/Intervention • Trail improvement, forest fuels reduction, habitat restoration, invasive species removal, surveys and research Our Evaluation History • Measurement of member outcomes via the Public Lands Service Coalition Study • Project specific measurement of outcomes via agency experts • Challenges/What we are pursuing: Wanting to delve deeper into these questions and research additional ways to make these measurement more evidence based

27. Challenges We Face • Outcomes are so varied and broad, not possible to measure everything that we do. • We don’t have a singular activity with a singular output. • Outcomes are not immediate, can take years to materialize. • Don’t know where we are going to end up!

28. Phased Approach to New EvaluationThree Phases • Phased Approach • Three phases over multiple years • Independent Research Team working in partnership with us throughout the three phases. • Research team consists of professors and graduate students at NC State and BYU. NC State and BYU – NC State is the lead. • Researchers also working on similar studies including one focused on the community impacts of Rails to Trails.

29. Phase 1 Literature Review 2 Goals of Literature Review • First, it was designed to provide a summary of scientific evidence about the impacts of conservation activities similar to those conducted by conservation corps. • Second, it was designed to identify potential strategies and indicators to measure impacts in future research and evaluation with conservations corps.

30. Phase 2 • Find a few things that conservation corps do from the literature review and then use similar measurement tools (as were used in the literature review) and apply them to our activities to determine possible impacts. • Will be a trial of different measurement approaches to determine best possible connections, impact, and outcomes that are reliable. • Identify how to measure multi-year impacts or change in condition in partnership with land management agencies. • Determine what is feasible and cost effective without over burdensome cost.

31. Step 3 • Expand upon Step 2 and incorporate additional conservation corps in different areas across the country. • Final assessment of the results. • Figure out the final implementation costs and determine reality.

32. Our Goal • Multi-year approach to finding best possible measurement tools using experts in the field and researchers to help us. • Never possible to have a fully experimental evaluation. • We don’t know exactly where we are going to end up!

33. Contact Info Amy Foss Conservation Legacy amy@conservationlegacy.org 970-749-1151

34. Jumpstart provides small grants ranging between US $7,500 and $10,000 for promising research that addresses significant questions on the processes and impact of Jumpstart. Provides researchers with the opportunity to investigate the influence of Jumpstart (1) on the lives of young children who live in low-income and high-stress communities, and (2) on adult volunteers who implement the Jumpstart program. Up to three grants will be awarded each year. Funded studies may be carried out using any research method or approach as long as the focus of the project is on examining the Jumpstart program. Priority will be given to applications that use rigorous research designs and methodologies. Jumpstart Learning Collaborative

35. 1. In what ways does Jumpstart contribute to children’s kindergarten readiness, specifically social emotional and executive functioning skills? 2. In what ways does Jumpstart support the development of children’s phonological awareness, oral language, and/or books and print knowledge? 3. In what ways does Jumpstart’s implementation fidelity (e.g. fidelity to curriculum, Corps member training) affect 1) Jumpstart children’s kindergarten readiness and 2) Corps members’ learning, knowledge, and overall Jumpstart experience? 4. In what ways does Jumpstart’s Corps member training affect children’s kindergarten readiness? 5. Does the Jumpstart Program impact children with different demographic backgrounds in different ways (e.g., Dual language learners or entering language levels)? 6. What impact does Jumpstart have on our Corps members? Research Questions

36. A Proposal to Study the Impact of Participating in Jumpstart on Corps Members Impact that participation in Jumpstart has on Corp members’ civic engagement, knowledge of child development, work readiness, and teamwork and collaboration skills, when compared to a similar group of students who are not involved in Jumpstart. Proposed Evaluation Study of Full-Class Delivery of Jumpstart in Chicago Sites Quasi experimental evaluation - researchers will be able to compare changes over time in a large sample of preschoolers receiving the full-class delivery model, to demographically matched children attending similar preschool programs, but not receiving Jumpstart. Assessment of Language, Literacy, and Social-Emotional Development Among Children Attending Preschools Served by Jumpstart This study will pilot a comprehensive assessment battery including direct measures of cognitive and social skills targeted by Jumpstart. A Multi-system Approach to Examining Effects of Jumpstart on Children’s Stress Response The purpose of the current project is to explore the effects of Jumpstart involvement on children’s stress physiology. We hypothesized that enrollment in Jumpstart would decrease children’s levels of cortisol and alpha-amylase across the preschool day. Examples of Funded Studies

37. Bi-annual Convening of Researchers Goals of Research Consortium To leverage learning from external research studies for program improvement To expand our knowledge on establishing and maintaining university research partnerships To determine avenues for future research through the Jumpstart Learning Collaborative Jumpstart Research Consortium

38. What questions do you have?