Measuring Business Impact in Workforce Development

1. Measuring Business Impact in Workforce Development Lisa Soricone Commonwealth Corporation lsoricone@commcorp.org www.commcorp.org NAWDP Conference May 19, 2011

2. Presentation Overview • Audience • Definitions • Our Experience • Steps Involved in Measuring Business Impact • Examples from Programs in MA • Q & A

3. What do we mean by Business Impact? • How does the business or employer benefit from a workforce development intervention? • Rooted in Kirkpatrick’s model for evaluation of training 1: Reaction 2: Learning 3: Behavior or application of learning 4: Results or business impact • Jack Phillips’ additions • Return on Investment • Intangibles • Continuum: from good stories to ROI

4. Where do we get our insights? • 10 years experience with sector strategies --- workforce development with dual benefit (workers and employers) • BEST • Bay State Works • Workforce Competitiveness Trust Fund (incl. case studies) • Extended Care Career Ladder Initiative • Research literature and other efforts in measuring business impact or employer impact • Kirkpatrick, Philips • Aspen Institute ‘s Business Value Assessment • Skills for Work project in the U.K.

5. Challenges to Measuring Business Impact • History of focus on individual outcomes • WD professionals’ discomfort engaging employers • Different needs/measures for different industries • Lack of time • Many employers don’t manage with this kind of data • Measuring the value of training is challenging work

6. How can Business Impact be measured successfully? 5 key steps in the process: • Defining the problem • Identifying business impact measures • Gathering data • Analyzing and interpreting data • Using and sharing data

7. Step 1: Defining the Problem What is the problem/goal to be addressed through the training intervention? • Ideally, part of the project design phase • Best pursued jointly by employer and WD staff • Requires some understanding of employer context/trends- market- how training fits their needs • Often involves multiple individuals from the employer organization(s)

9. Defining the Problem:  Things to Remember • Consider business impact as early on in a project as possible. • Set up a meeting with management/other staff at each employer. • Prepare a set of questions to learn about your employer(s). • You needn’t be an industry expert, just be curious! Working with multiple employers: • Look for common goals • Assess employer interest in engaging in the process • Consider the proportion of workers who’ve been trained at each employer

10. Step 2: Identifying Business Impact Measures Once the problem to be solved is identified , how will you know you’ve solved it? • Define “success” in the employer context. • What would change? • Where can you see change that is related to training? It depends…

11. Organizational Benefit Work Processes Improved Application of Knowledge & Skills New Knowledge & Skills Training

12. Finding suitable measures • Talk to employer staff at different levels:Leadership, Training/HR, Operating staff, Supervisors, Participants • If possible, use data that the employer is already collecting and using. • Focus on meaningful and feasible measures.- Do the measures relate to training?- Might the measures be strongly impacted by other factors?- Can you get data on measures with reasonable time and effort?

13. Sample Business Impact Measures

14. Intangible Measures Some things don’t convert to numbers but still matter: • Improved work climate • Increased organizational commitment • Improved teamwork • Improved cooperation/reduced conflict • Reduced employee stress • Improved communication • Improved labor/management relations

15. Case Study: Arwood ManufacturingUsing an existing measure to show impact • Business Problem: reduce production and inspection errors due to misinterpretation of blueprint specifications and production validation • Training Solution: Geometric Dimensioning & Tolerancing and Blueprint Reading • Indicator: error rate in documentation and inspection • Results: a 61% reduction in error rates through improved quality, inspection and documentation processes

16. Example: Food Manufacturing Workplace Education ProgramFocus on Application of Skills Survey to determine employees’ improvement in 5 key areas: • Expressing ideas and asking questions in English • Correctly identifying ingredients, tools, equipment used regularly • Using safety gear (gloves, goggles) as instructed • Using correct procedures to request time-off /call-in sick • Identifying problem areas (safety hazards, wrong ingredients) Scale: greatly improved to not at all Space for additional comments This approach may seek individual or collective feedback.

17. Example: Early Childhood EducationUsing a cross-agency survey to gather data on impact • Context: State policy increasing educational requirements for staff • Project goals include:-- Reduce staff turnover-- Increase # of teachers pursuing credentials / higher ed degrees-- Increase agency ability to meet staff qualifications for funding and other opportunities that increase quality and teacher compensation • Cross-employer Survey (see handout)- organizational impact – significant to minimal - on: employee stability, attitude/motivation, understanding of child development, family relationships, instruction and more- captures baseline and interim/final results

18. Identifying Business Impact Measures:  Things to Remember • Work with employers to understand the context and determine meaningful measures. • Choose measures that relate to training. • Go as far toward organizational benefit as you can. • Intangibles are ok – but require evidence. • Keep it simple. • The point is to demonstrate value.

19. Step 3: Gathering Data Once you have defined your measures, how do you get data to see if desired change has happened? • Comparisons: pre- / post- training or comparison group • Timing: When do you need data? When are data available? When should you collect your own? • Data sources: What do you have? What can you get? What tools/agreements will you need? • Roles and responsibilities: Who does what? When?

20. Examples from MA projects • Surveys:baselineinterim for project feedbackpre-post comparisons at project end • Collection of employer data:quarterly updatesannual updatesfinal reporting* keep in mind employer data calendars

21. Healthcare Technician Training 1– Dollarizing Example

22. Gathering Data:  Things to Remember • Establish a plan and timelines for data collection and reporting.For each measure: • How will data be gathered? ( e.g., survey) • When will data be collected? • Who will be responsible? • Develop your plan with employer data timeline in mind. • Don’t be afraid to send reminders! • Do a test run if new data is being collected or a survey is being used. • Set confidentiality parameters: what can be shared/aggregated at organization/project/community levels?

23. Step 4: Analyzing and Interpreting Data For each measure, review the data: what do they tell you? • What kind of changes took place? • Did participant skills increase? • Are people working differently? • Did the employer experience a cost savings? • To what extent did the changes that were expected to result actually occur? • How significant is the change(s) observed? • Could this change be due to other factors/events? • To what extent do changes meet the project’s business impact goals ?

24. Healthcare Technician Training 2 Example

25. Analysis of Data • Reduction in - vacancy rates - overtime expenses • No change in- orientation time- retention • Other factors influenced outcomes but training was a contributor • Need more time to see impact on overtime and retention • What about the program content may have affected outcomes? What could be changed going forward? • Additional benefit: opportunity to experiment with staffing model for a high demand position

26. Analyzing and Interpreting Data:  Things to Remember • Focus on one measure at a time. • Ask “What happened?” before you jump to determining what results mean for business impact. • Involve stakeholders in the process of analyzing data at an appropriate time. • If you didn’t find the changes you’d hoped for, discuss findings within your project; there may be a lesson in there somewhere.

27. Step 5: Using and Sharing Data What can you do with all this new information? • Business Impact data can be used for: • Accountability and reporting to stakeholders • Continuous improvement • Feedback to trainers and project staff • When sharing your data, place them in the context of the overall goals the project sought to achieve and also what else is happening in the environment.

28. Using and Sharing Data:  Things to Consider • Consider your audience: - What do they really want to know?- What context will they need to understand findings? - What level of detail will be most appropriate? • What’s the best mechanism for communicating? - written report- presentation • Respect your confidentiality/permission agreementsat organization/project/community levels

29. Value of MBI • Demonstrates the value to employers of investing in their employees • Improved employee skills • Cost savings and/or quality improvements • Address skill shortages or deficiencies in the workforce pipeline • Improved relationships with education/training providers and workforce development agencies • Shows how public sector investment in workforce development improves competitiveness of companies • Builds both a culture of and capacity for examination of value of such investments Page 29

30. Resources • Getting Started Document • Commcorp website: www.commcorp.orgResources Tools  Assessing Business Impact- Sample surveys, tools and presentations • Business Value Assessment: -- Tips for using questionnaires (and more)http://www.aspenwsi.org/wsiwork-bvatool.asp • How to Measure Training Results by Jack Phillips and Ron Drew Stone, McGraw-Hill, 2002.

31. Questions?