Improving UI Integrity: Strategies and Policies to Reduce Work Search Errors

1. Get a Job! 2012 UI Integrity Summit March 13, 2012

2. Today's Panel Moderator: Brad Wiggins Integrity Project Coordinator, Office of Unemployment Insurance U.S. Department of Labor, Employment and Training Administration Panelist: Andrew Clarkwest Senior Researcher Mathematica Policy Research

3. Our Agenda • Review of DOL’s Work Search Strategy • Analysis of State UI Work Search Policies • Mathematica Policy Research • Group Discussion

4. UI Improper Payment Rate

5. Root Causes of UI Overpayments

6. UI Integrity Strategic Plan • DOL has been focused on prevention, detection and recovery in three major root cause areas: • Claimants continuing to claim after returning to work (Benefit Year Earnings (BYE) • Untimely/insufficient separation information from employers and Third Party Administrators • Employment Service (ES) registration • Focus in 2012 on New Strategies to Address Work Search Errors

7. Work Search Strategy • Research: Analysis of State UI Work Search Laws, Policies, and Practices • Pilot: Development of a Portable Web-based Tool to Capture, Organize and Share UI Claimant Work Search Records • Working Group: Small Group of States (12) to Review Research and Frame National Strategies to Reduce Work Search Errors

8. Analysis of State UI Work Search Policies

9. Study Background

10. Why Focus on Work Search Error? • Violation of design and mission of UI program • Temporary support for those to those with involuntary job loss • Assumes claimants are trying to obtain new employment • Fiscally costly

11. Why Focus on Work Search Error? • Legal • IPERA: Goal to reduce improper payments by $50 billion across federal programs • UI identified as a program with unacceptably high rates of improper payments • Political: Refocus UI to emphasize job search

12. Two Fundamental Questions • How many benefits are being paid to claimants who do hold up their end of the bargain by actively searching for work? • How can that number be reduced?

13. Research Questions • Which state policies and practices are correlated with higher or lower rates of work search improper payments (IPs)? • How can measured rates of work search IPs be improved to more accurately reflect actual rates of non-compliance?

14. Data Sources

15. State Policies and Practices Dataset • Quantitative dataset characterizing state policies on: • Work search requirements • Recording and reporting requirements • Verification and enforcement

16. BAM Data • Work search error measures • State practices observed in data • Frequent ERP interviews • Most common filing method • Benefit certification frequency • Work search exemption frequency • Labor exchange registration requirement • Staff job search assistance • Use of warnings

17. In-Depth Primary Data Collection • In-depth primary data collection from a subset of states • Phone (7 of 9 responded): FL, IL, ME, NY, OH, TX, WI • Selected for geographic variation, large dollar amount of work search improper payments • Also a subset with very low work search error rates • Written responses by OUI site visit staff (4 of 6 responded): AZ, CO, LA, VA • Improper Payment High Priority States for fiscal year 2012 • Work search a major cause of IPs in each state

18. Variation in State Policies

19. Work Search Requirements Source: SPPD

20. Documentation and Reporting • Documentation: Many states require claimants to keep a log of activities and provide it on request • Reporting: Few states require claimants to regularly submit details of the log • Required reporting of contacts appears to have decreased with technological advances • Verification: Some states perform random audits of work search (other than BAM audits), but proportion of claimants audited tends to be very low

21. Variation in Work Search Error Rates

22. Work Search Error Rates Across States

23. Variation in Work Search Error Rate • What factors explain variation in work search error rates? • Challenge: Measured work search error rates may not be comparable across states • How much of the observed variation in work search error rates reflects actual variation in work search compliance?

24. Measuring Work Search Error Rates

25. Measuring Work Search Error Rates

26. Issues in Measurement of Work Search Error Rates • We only observe “measured” error rates • BAM is the most comprehensive and reliable data source on error rates • BAM data provide clues on differences between measured error rates and rates of claimant non-compliance • Finding: much of the variation in measured error rates appears to be unrelated to actual rates • Subsequent analyses must be interpreted with caution

27. Issues in Measurement of Work Search Error Rates • Differences in treatment of nonresponse • In 5 states, more than half of nonrespondents have work search error • In 39 states, no nonrespondents have work search error • Differences in formal warning rules • 15 states determined that most work search errors were “technically” proper due to formal warning rule

28. Issues in Measurement of Work Search Error Rates • Differences in exemptions from work search requirement • Across states, 1 to 58 percent of claimants are exempt from work search requirement • Conversations with states provide evidence of BAM practices that are at times inconsistent with state policies • Independent assessment of claimant status in states with numeric work search requirement suggests higher error rates than reported

29. Reported vs. Independently Assessed Error Rates

30. Analysis of Predictors of Work Search Errors

31. Overview of Analysis • Goal: Identify policies and practices that are associated with higher or lower work search error rates • Methods: • Compare error rates across states with different stringency of work search rules • Conduct correlational analysis to identify significant predictors of work search error

32. Error Rates by Stringency of Policies • Do states with more stringent policies have higher or lower work search error? • Our approach: • Construct a stringency measure based on state policies and practices • Compare work search error rates across categories • Findings: • No significant differences across categories

33. Error Rates by Stringency of Policies Note: Differences in work search error rates across categories are not statistically significant.

34. Predictors of Work Search Error • Are specific policies and practices associated with high work search error rates? • Our approach: • Examine correlations between each policy and work search error rates • Use a regression analysis to determine which policies were the best predictors of error rates • Findings: • A small number of policies predict work search error • Results depend on the outcome considered • Associations between policies and error rates do not prove a causal effect

35. Work Search Error Outcomes • We considered several error rate outcomes: • Reported error rate • Reported error rate, excluding states with very low rates (less than 0.5 percent) • Indicator for rate less than 0.5 percent • Error rate treating formal warnings as errors • Assessment of error rate based on numeric work search requirement

36. Policies Associated with Work Search Error * Denotes that the policy has a statistically significant correlation with the error rate or other outcome.

37. Predictors of Work Search Error • States allowing claimants to submit work search information via the Internet have higher work search error rates • When excluding states with very low error, states requiring claimants to log names of individuals contacted have higher error rates • Possible that more stringent reporting requirement leads to lower burden of proof for SWA in detecting errors

38. Predictors of Work Search Error • States requiring work search “customary for occupation” or having exemptions from requirement for some claimants are less likely to have very low work search error. • When counting formal warnings as errors, states with formal warning policies have higher work search error rates. • One explanation is that formal warning policies may reduce incentive to satisfy requirements.

39. Recommendations for Improving Data Consistency

40. Improving Data Consistency: Motivation • Understanding causes of work search error is difficult if data are not complete and consistent • How can data be improved in order to better target work search error? • Ensure that BAM audits apply state policies correctly • Consider policies that promote more complete data

41. Consistency of BAM Audits • BAM data are sometimes inconsistent with state policy • Some states requiring claimants to submit work search logs grant clemency to BAM nonrespondents • Lack of employer contacts sometimes contrasts with determination of work search error • Develop BAM procedures based on state policy • If state policy requires work search log, code nonrespondents as having work search error • Check employer contacts against state requirement

42. Policies Promoting Higher Quality Data • Updating work search policies may enable better quality data collection • Better data lowers the SWAs’ burden in verifying work search requirements • Suggested policies: • Require claimants to keep a log • Require claimants to provide work search information as part of continuing claims process • Clarify rules on activities that may substitute for employer contacts • Eliminate formal warning rules

43. Work Search Compliance Recommendations: Ideas from Conversations with State Workforce Agencies

44. Recommendations • Enhance work search support through increased collaboration with the local One-Stop Career Centers • Encourage or require more job search assistance • Worker Profiling and Reemployment Services program is an example • Collaborate on verification of work search activities

45. Recommendations • Augment verification processes • Require claimants to submit log of work search activities • Implement a system that automatically checks completeness of reporting • Provide increased funding for audits • Impose harsher penalties for failure to meet work search requirements • Disqualify claimants after a work search error

46. Group Discussion Questions?

47. Identified challenge areas: BAM Training Automated work search verification BAM methodology Work search messaging Other challenges? Promising practices? Group Discussion

48. THANKS!