AEL: Then, Now, Soon

1. AEL: Then, Now, Soon Adam.Leonard@TWC.state.tx.us

2. Background – TWC’s Origins • Texas Workforce Commission created in 1995 • Originally Consolidated 28 programs from 10 agencies • Goal: Demand-driven system with seamlessly integrated service delivery • Local Workforce Development Boards provide service with local flexibility • State provides oversight and ensures accountability • Mission: • To promote and support a workforce system that creates value and offers employers, individuals, and communities the opportunity to achieve and sustain economic prosperity

3. Background – TWC Keys to Success • Clear Vision • Strong Leadership • From Locals on Up • From Governor on Down • Partnerships with Stakeholders • Willingness to Change • Commitment to Excellence • Belief in Data • Commitment of Resources to Use of Data • Monthly Performance Reporting • Quarterly Open Performance Briefing

4. Background – AEL comes to TWC • Texas Sunset Commission Reviewed AEL in Texas as part of their 2012-13 Sunset Review of TEA • Sunset listed AEL as ISSUE 2 out of 13, stating that • Texas Lacks Clear Leadership on Adult Education • This is Threatening the State’s Ability to Meet Future Workforce Demands • The Program is Misplaced at TEA • Key Findings • AEL doesn’t serve a sufficient portion of the need • Educating adults is not part of TEA’s core mission • AEL and WF Development have inadequate coordination • AEL in TWC would better position the state to ensure adults have the basic skills necessary to succeed in the workplace

5. Digging into Sunset Findings • AEL Doesn’t Serve a Sufficient Portion of Need • Individuals without basic skills cannot compete for higher paying jobs to support their families nor function well on the job and in society • Educating adults is not part of TEA’s core mission • TEA lacks a clear focus/adequate oversight of AEL program • TEA largely outsourced the program and didn’t have any single staffer who focused exclusively on AEL • Existing Providers were extended for 10 Years • No new Procurement • No action where failing to perform (30-50% of providers failed to use all funding in EACH of the 3 years prior to Sunset Review) • TEA never pushed grantees beyond basic federal minimums

6. Digging into Sunset Findings • AEL and WF Development have inadequate coordination • Many AEL Providers focus exclusively on long-term goals at the expense of accelerating job-readiness • Lack of Data Sharing between AEL and WF programs • Lack of strong relationship between AEL Providers & WF Boards • AEL in TWC would better position the state to ensure adults have the basic skills necessary to succeed in the workplace • Students would benefit from stronger partnerships • Data Sharing would be enhanced (no FERPA issues)

7. And then there was WIOA . . . • Increase access to and opportunities for the employment, education, training, and support services to succeed in Labor Market – especially for those who haven’t historically had success • Align workforce investment, education, and economic development systems in support of a comprehensive, accessible, and high-quality workforce development system • Provide America's workers with the skills and credentials necessary to secure and advance in employment with family-sustaining wages and to provide America's employers with the skilled workers the employers need to succeed in a global economy • Improve structure and delivery of services through the workforce development system to better address the employment and skill needs of workers, jobseekers, and employers • Increase the prosperity of workers and employers, the economic growth of communities, regions, and States, and the global competitiveness of the United States.

8. AEL under WIOA is to Assist: • Adults to become literate and obtain the knowledge and skills necessary for employment and economic self- sufficiency • Adults who are parents or family members to obtain the education and skills that-- • Are necessary to becoming full partners in the educational development of their children; and • Lead to sustainable improvements in the economic opportunities for their family • Adults in attaining a secondary school diploma and in the transition to postsecondary education and training, including through career pathways • Immigrants and English language learners in-- • improving their reading, writing, speaking, and comprehension skills in English; and mathematics skills • acquiring an understanding of the American system of Government, individual freedom, and the responsibilities of citizenship

9. Now it Makes Sense! • AEL doesn’t address enough need? • TWC focused initially on Enrollment Levels • AEL largely an Outsourced Program? • Created AEL Department staffed with Policy and TA staff • No competition for AEL providers? • Reprocured Program Operators (twice) • No push to move beyond basics? • Emphasized Expansion of IET and Intensive Models • AEL & WF Development have inadequate coordination? • Engaged with DOL & US ED to better align WIOA programs • Pushed AEL Providers and WF Boards to work more closely together • Inadequate focus on short-term goals? • Again, IET and Intensive models offered more employment-focused options to students • Lack of focus on Employment? • Leaned hard to focus on WIOA Employment Outcomes

10. And Then . . . • In 2013, Legislature CLEARLY demanded change in AEL to better address workforce preparedness • In 2014, Congress CLEARLY demanded similar changes thru WIOA • In 2019, Senate Finance Committee Asked: • Have you just turned this into a Job Training Program? • Only 7% in IET in PY17 • 86% of all students are at <=5th educational functioning level • Are you teaching people basic skills? • 47% of those Served have Gains • 84.52% of those Tested show Gains • So where did these questions come from? • Our Measures? • Our Reports? • Are we not doing a good job communicating?

11. WIOA AEL Measures

12. Performance Measures • Promote Accountability • Communicate Information • Highlight Successes • Identify Areas for Improvement • Influence Behavior • Good Measures have a positive influence • Good behavior improves performance • Bad Measures have a negative influence • Bad behavior “improves” reportable performance • Selective enrollment • Dedicating excessive resources to admin/tracking • Foster Innovation

13. Careful What You Measure (& How) • Google: “You Are What You Measure” • CEO Pay tied to Stock Price? • Expect laser-like focus on Stock Price • Why so many Online Top 10 Lists as “Photo Galleries?” • PAGE VIEW Measures • Has Customer Service gotten better? • Call centers moved from Call Duration to Customer Satisfaction • What happens when Teacher Jobs are Tied to Standardized Test Results? • Teaching the Test rather than How to Think? • Quantified-Self Movement

14. Effective Performance Measures • Mission-Aligned & Meaningful • Otherwise they’re viewed as compliance requirements • Accessible by all stakeholders, not just analysts • Hard to ensure Accountability in absence of information • Transparent & Understandable to stakeholders • Hard to get buy-in with “Trust us, you’re failing” • Lack of Understanding leads to disengagement • Provide Accurate, complete, reproducible numbers • Stakeholders have to know they can believe what they see • Available Timely • Hard to timely take action on 15 month old data • Efficient • Gathering data to PROVE the result takes resources from doing that which helps you ACHIEVE the result

15. AEL Performance Trends

16. Drilling into Employed/Enrolled Trends

17. Drilling into Other Exit-based Trends

18. ABE-EFL Performance Trends

19. ESL-EFL Performance Trends

20. Drilling into MSG Performance Trends

21. What’s Next? • Strategic Plan in the Spring provides Opportunities • Can Modify or Clarify Measures • Can rethink the story we tell about AEL in Texas • Better detail on those we serve and what we accomplish • REAL WIOA Performance Accountability Begins • WIOA §116 has EXTENSIVE Negotiation Provisions • State Negotiations must consider: • Other State Levels of Performance • Statistical Adjustment Model that addresses • Economic conditions • Characteristics of Participations Served (Casemix) • Continuous Improvement to ensure Optimal ROI • National Government Performance and Results Goals

22. Why does WIOA Require Models? • Guide Negotiations and Ensure Fairness • Ensure that “Creaming” Doesn’t get Rewarded • Serve Highly Educated People with lots of Work History & Relevant Skills? Target Goes Up • Serve Less Educated People with many Other Factors that May Negatively Impact Employment? Target Goes Down • Answer the Arguments • “We have more Hard-to-Serve participants” • “We’re serving more Hard-to-Serve Participants Now” • “Our Economy is Worse”

23. Statistical Adjustment Model Requirements • Economic Conditions minimally including • Unemployment Rate • Job Losses/Gains in various Industries • Participant Characteristics • Poor work history / Lack of work experience, • Lack of educational or occupational skills attainment • Dislocation from high-wage and high-benefit employment • Low Levels of Literacy or English proficiency • Disability status • Homelessness • Ex-offender status • Welfare dependency • Gender, Age, Race, Ethnicity

24. More AboutStatistical Adjustment Modeling • Statistical Models require LOTS OF DATA to create • Historic results under the new Measures • Participant Casemix of those included in results • Economic Conditions relevant to the results • DOL, OCTAE, RSA all having to develop models • DOL had lots of historic data • OCTAE & RSA did not • OCTAE’s Model will be built using • 2 Years of Data (of varying degrees of completeness) • Summary Level Data Only • Data that doesn’t align well with the actual Cohorts

25. What can we Learn from DOL Models? • DOL & OCTAE & RSA all coordinating on Model Development • DOL had Great Statistical Technique • Tons of Data Elements that went WAY beyond model attempts under WIA • Recognition that Regression Models estimate “Average Effects” but there are differences they can’t account for: • Drop in Movie Ticket Sales hurts California more than Texas • Rise in Oil Prices helps North Dakota more than Maine • ESL has lesser impact in Texas than Michigan • Fixed Effects Model provided 2 predictions • Target (what an “Average State” should achieve) • Predicted (what “Your State” should achieve) • RSA using Fixed Effects; hopefully OCTAE will too

26. How do the Models Work? • Each model will have several dozen data elements • DOL Models had ~50 Elements per Measure • Sex, 4+ Age Ranges, 5+ Races, Ethnicity, Disability, Veteran, 6 Education Levels • 4 Prior Employment/Earnings • 9 “Other Factors Impacting Employment” • 7 Related to types of Services • Unemployment Rate and Changes in Employment: • Natural Resources • Construction • Manufacturing • Information, Finance, Technology • Education, Health Care • Leisure, Hospitality, Entertainment • Government • Other

27. Other Factors Impacting Employment • WIOA requires reporting detailed information on ALL Participants – • Displaced Homemakers • Low-income individuals • Indians, Alaska Natives, and Native Hawaiians • Individuals with disabilities • Older individuals • Ex-offenders (which now means pretty much anyone arrested) • Homeless individuals • Foster Youth or those who have aged out of the foster care system • Individuals who • Are English Language Learners, • Have Low Levels of Literacy, and • May face Substantial Cultural Barriers • Eligible migrant and seasonal farmworkers (and their dependents) • Individuals within 2 years of exhausting lifetime eligibility for TANF • Single parents (including single pregnant women) • Long-term unemployed individuals • Other groups identified by the Governor

28. Application of the Model • Each Factor in the Model has a Weight • Factor and Weight basically tells us how much performance should go up or down “all other things being equal” • DOL Adult Employed Q2 Example: • Female Impact is .07973 • As % of Exiters who are Female goes up by 1 percentage point, performance should increase .07973 • As % of Exiters who are Female goes down by 1 percentage point, performance should decrease by .07973 • Calculation of the Target involves: • Calculating each Element (% female, % 26-35, etc.) • Multiplying by the Element’s Weight to get the impact of that element on performance • Taking the Fixed Effect and Adding all Impacts

29. Demographics with Biggest Impact • Serving Higher % of Participants with Other Factors that Impact Employment will Reduce Performance Target • Failure to Document Factor will generally result in HIGHER Targets

30. Economic Factors with Biggest Impact

31. WIOA Target Setting has 2 Phases • Targets initially negotiated using Statistical Models that focus on ASSUMED Casemix & Economic Conditions • Targets are later retroactively adjusted Down or UP based on the ACTUAL Casemix & Economic Conditions • If actual Casemix or Economic Conditions are “better” than expected, targets go up at the end of the year • “Easier to Serve” Casemix? Better Economic Conditions? Targets goes Up

32. Radically Different • WIOA Performance Accountability is NOT about TARGET SETTING • WIOA Performance Accountability is about PERFORMANCE EVALUATION • We don’t set a future Target & Aim for it • Even without the Lag, the negotiated targets end up being changed based on actual Casemix & Economic Conditions • The Key to WIOA Performance Accountability is Obtaining, Learning From, and Using DATA • What Data is required? • What does the Statistical Adjustment Model account for & what does it miss? • What factors might exist or be changing that suggest past performance will not be indicative of future performance?

33. EVALUATIONS not just for Targets • WIOA §116 requires Evaluations to: • Promote, establish, implement, and utilize methods for continuously improving core program activities • Achieve high-level performance within, and high-level outcomes from, the workforce development system • Moving into Evaluation Phase in all WIOA Programs • TWC accepted to be part of national AEL Evaluation Workgroup • Some of the key areas of focus: • Effective means of Gathering Data • Impact of Onboarding on Engagement & Achievement • Dream is to have models that help staff choose the best approach to serving each Participant

34. “Performance Management” • Performance is to be Achieved, not managed • If Performance is seen as something to be “managed”, people become focused on “getting credit” • Culture of Credit undercuts real mission • Puts focus on trying to LOOK excellent rather than trying to BE excellent • Often incents short run decisions with long run costs

35. Culture of Credit vs. Culture of Excellence • CoC spends inordinate resources on follow-up to “get credit” for results • Diminishing Returns? • CoE focuses resources on the thing that creates the results: quality services • CoC more concerned about what is reported than what it MEANS • Doing well on a BAD measure ≠ good • In a CoE, bad measures are • Resisted if possible • Tolerated when not • NEVER Celebrated, even if Target met

36. Continuous Improvement • World Class Organizations all strive for Continuous Improvement • Government increasingly expected to as well • Not a Red or Blue Issue • Efficiency • Effectiveness • Evidence-informed • Analytics provides opportunity to meet these expectations