Net Impact Evaluation Framework For Minnesota With preliminary results as of May 2014

1. Net ImpactEvaluation Framework • For Minnesota • With preliminary results as of May 2014 • A Standardized

2. Getting Started Nick Maryns Senior Policy Analyst Governor’s Workforce Development Council Nicholas.maryns@state.mn.us Raymond Robertson Professor of Economics Macalester College robertson@macalester.edu

3. Standardized Net Impact Evaluation Framework Motivations, History, Partners Overview and Basic Parameters Evaluation Design Pilot Project Preliminary Results

4. Motivations, History, Partners Motivations History Advisory Group “Based on our rough calculations, less than $1 out of every $100 of government spending is backed by even the most basic evidence that the money is being spent wisely.” • Peter Orszag and John Bridgeland,The Atlantic Monthly, July 2013

5. Motivations, History, Partners Motivations History Advisory Group The National Conversation Pew-MacArthur Results First Initiative Results for America Social Impact Bonds / Pay for Success

6. Motivations, History, Partners Motivations History Advisory Group Apples and Oranges Approaches across the State

7. Motivations, History, Partners Motivations History Advisory Group The UPAM law required the development of uniform ROI measure.

8. Motivations, History, Partners The GWDC’s Role Motivations History Advisory Group (d) Functions. The State Board shall assist the Governor in— (6) development and continuous improvement of comprehensive State performance measures, including State adjusted levels of performance, to assess the effectiveness of the workforce investment activities in the State as required under section 2871 (b) of this title; • Section 111 of the Workforce Investment Act Subd. 3. Purpose; duties. (c) “Advise the governor on the development and implementation of statewide and local performance standards and measures relating to applicable federal human resource programs and the coordination of performance standards and measures among programs” • Minnesota Statute 116L.665 Subd. 3c

9. Motivations, History, Partners Motivations History Advisory Group State Agencies • Department of Employment and Economic Development • Department of Corrections • Department of Education • Department of Human Services • MN State Colleges and Universities Local Workforce Boards • City of Minneapolis Employment and Training Program • Minnesota Workforce Council Association • Workforce Development, Inc. Community Organizations • Greater Twin Cities United Way • Lukeworks • Twin Cities RISE! Business / Employers • Dolphin Group • MN Chamber of Commerce Researchers / Evaluators • Anton Economics • Invest in Outcomes • Macalester College • Minneapolis Federal Reserve Bank • Wilder Research

10. Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Goal A framework for measuring and understanding the net impacts and social ROI of all publicly-funded workforce programs that is standardized and credible, and that informs strategy and continuous improvement

11. Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Supportive Policies and Infrastructure Cost-Benefit Analysis Net Impact Analysis Oversight / ManagementFramework

12. Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Manageable, feasible to administer Useful, relevant, timely Credible, transparent, trusted Adaptable, sensitive to change

13. Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Improving Services, Driving Value“What works, and for whom?”“What disparities exist?” Making Smarter Investments“How do current investments align to what works, and to disparities in our community?” Communicating Value“How do workforce services benefit participants and taxpayers?” Standardizing the Approach Strengthening Transparency/Accountability

14. Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs Publicly-administered and funded workforce programs DEED and other state agencies Non-profit passthroughs Public education (elements of K-12 and PS) State and federal competitive grants and special initiatives Programs serving targeted populations (e.g. people with disabilities, veterans) Long-Term Vision Independent nonprofits and education providers Other service areas

15. Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs United Way Wilder Research Invest in Outcomes/State Pay for Performance National Governors Association

16. Overview and Basic Parameters Framework Design Values Objectives Scope of Programs Collaborations Considerations and Trade-Offs One methodology, many programs Ensuring usefulness to program managers and policy makers (Unintended) incentives created by measure

17. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Policy Framework Management Framework Cost-Benefit Analysis Net Impact Analysis

18. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Break-Even Point $ Benefits Time Costs ROI = (Benefits – Costs) / Costs (Return) (Investment)

19. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Employment Income / Fringe Benefits Taxes Income / Payroll / Sales Public Assistance Savings MFIP / SNAP / UI Healthcare Savings MinnesotaCare / Medical Assistance Incarceration Avoidance

20. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Program Costs Time-weighted / Service-weighted (where possible) Cost to Participant

21. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Benefits and Costs to Participants + Benefits and Costs to Taxpayers = Total Social Benefits and Costs

22. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Some public benefits Subsidized housing costs Prescription Drug Program costs Child Support payments Other important but difficult-to-quantify effects Change in mental and physical health Change in worker productivity Reduction in criminal activity Economic multipliers

23. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Cloning Randomized Trials Causality / True Attribution Approaching Kernel Density Propensity Score Matching Difference-in-Difference Estimator “As good as random”

24. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Foundation: administrative data at the individual level Avoid broad assumptions wherever possible Not Used: Self-reported program performance indicators, e.g. entered employment rate six-month retention rate earnings change

25. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Earnings Not the relevant comparison Treatment Group Net Impact Comparison Group Time

26. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Accounts for Many Factors Personal Characteristics Geography Local Economic Conditions Services Received Also allows us to analyze performance by these categories

27. Evaluation Design Cost-Benefit Analysis Benefits Costs B/C Perspectives What’s Not Included Net Impact Analysis Data, Not Assumptions Net Impact Other Features Contextualized Performance GoalsAdjusted for population served, local conditions Leading IndicatorsFor near-term relevance; based on statistical relationships between near-term indicators and long-term outcomes

28. Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe Purpose Primarily for internal use, to test concept, methodology, data process The pilot evaluation comprises 950,000 individuals and roughly 50 million data points

29. Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe Initial Cohorts (2007-08 and 2009-10) WIA Adult Program WIA Dislocated Worker Program Twin Cities RISE! New Cohorts (2010-11) FastTRAC I&B Grantees MFIP / DWP Employment Services Adult Basic Education SNAP Employment and Training

30. Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe Registrants at WorkForce Centers and on MnWorks.net Unemployment Insurance Applicants

31. Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe

32. Pilot Project Scope of Programs Treatment Comparison Data Sharing Timeframe New round of Data Sharing Agreements recently finalized Data are currently coming in Results this Fall

33. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Preliminary results address earningsand employment impacts across two programs: • WIA Adult • Dislocated Worker (both WIA and MN) Treatment cohorts are defined as such: Additionally, some initial findings on FastTRAC data are also provided. See disclaimer to the left.

34. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next The analysis that has produced the following preliminary results has been guided by the GWDC Net Impact Advisory Group and is still under development. The preliminary results that follow have been reviewed by program directors and relevant staff at DEED, who emphasized the value of the findings and voiced their support for the continuation of the effort.

35. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.

36. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.

37. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.

38. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: Treatment and control groups have been matched along a number of variables. Tables 1-4 show how similar the cohorts are. The main difference is with regard to race; in all cohorts, treatment cohorts have a lower percentage of white individuals.

39. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 1a: WIA AD 0708 Kernel Density Distribution of Pre Wages INTERPRETATION: These charts show how similar pre-enrollment earnings are between treatment and control. For WIA Adult, wages are slightly lower than the controls; for Dislocated Worker, the match is closer. Figure 1b: WIA AD 0910 Kernel Density Distribution of Pre Wages

40. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 2a: DW 0708 Kernel Density Distribution of Pre Wages INTERPRETATION: These charts show how similar pre-enrollment earnings are between treatment and control. For WIA Adult, wages are slightly lower than the controls; for Dislocated Worker, the match is closer. Figure 2b: DW 0910 Kernel Density Distribution of Pre Wages

41. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next INTERPRETATION: This table tells us the average time in program is between three quarters and a year, with a lot of variation.

42. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 3: Unmatched Wage Distribution: WIA Adult 0708 INTERPRETATION: In the earnings charts that follow, 0 represents time of enrollment. We worked to match earnings in the pre-period. The net impact on earnings is the average difference in the post-period, specifically quarters 5-8. Matched Average Net Impact

43. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 3: Unmatched Wage Distribution: WIA Adult 0708 INTERPRETATION: For WIA AD 0708, the results at right translate to a net 30% increase in earnings for program participants, controlling for other observable factors. The statistical significance of the result is still in progress.

44. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 4: Unmatched Wage Distribution: WIA Adult 0910 INTERPRETATION: For WIA AD 0910, the results at right translate to a net 31% increase in earnings for program participants, controlling for other observable factors. The statistical significance of the result is still in progress.

45. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 5: Unmatched Wage Distribution: Dislocated Worker 0708 INTERPRETATION: For DW 0708, the results at right translate to a net 52% increase in earnings for program participants, controlling for other observable factors. The result is statistically significant.

46. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 6: Unmatched Wage Distribution: Dislocated Worker 0910 INTERPRETATION: For DW 0910, the results at right translate to a net 31% increase in earnings for program participants, controlling for other observable factors. The result is statistically significant.

47. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 7: Unmatched Employment Distribution: WIA Adult 0708 INTERPRETATION: For AD 0708, the results at right translate to a net 30% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.

48. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 8: Unmatched Employment Distribution: WIA Adult 0910 INTERPRETATION: For AD 0910, the results at right translate to a net 29% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.

49. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 9: Unmatched Employment Distribution: Dislocated Worker 0708 INTERPRETATION: For DW 0708, the results at right translate to a net 6% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.

50. Preliminary Net Impact Results Summary Statistics Earnings Impacts Employment Impacts FastTRAC: Initial Findings What’s Next Figure 10: Unmatched Employment Distribution: Dislocated Worker 0910 INTERPRETATION: For DW 0910, the results at right translate to a net 5% increase in the likelihood of employment, controlling for other observable factors. The result is statistically significant.