Measuring the Impact of Career Guidance: State of Practice in Canada

1. Measuring the Impact of Career Guidance: State of Practice in Canada Who Cares About Career? Career Guidance Week Centre for Guidance Studies, University of Derby June 2007 Bryan Hiebert On behalf of the Canadian Research Working Group for Evidence-Based Practice in Career Development (CRWG)

2. Overview • Background • State of practice: Canadian studies • Taking action • Addressing current trends • Evaluation framework • Current projects • Future directions

3. Background and Rationale • International symposia and pan-Canadian Symposium • Participants included career practitioners, policy makers, and employers • Participants agreed • Need more effective evaluation systems • Need evidence reflecting the impact • to inform public policy pertaining to individuals, families, organizations, society, etc.

4. Background and Rationale A challenge by Canadian Policy Makers: “You haven’t made the case for the impact and value of career development services”

5. To meet the need … In 2004, a research team was formed to follow-up on recommendations from Working Connections, the Pan-Canadian on Career Development and Public Policy • The Canadian Research Working Group for Evidence-Based Practice in Career Development (CRWG) • Robert Baudoin, Lynne Bezanson, Bill Borgen, Liette Goyer, Bryan Hiebert, Kris Magnusson, Guylaine Michaud, Vivian Lalande , Céline Renald, Reginald Savard, Michel Turcotte • First task: Assess the current state of affairs

6. Research Project Goals Better Understand the State of Practice of Career Development Evaluation in Canada • Importance of evaluation • Scope of evaluation practices • Types of outcomes identified and/or reported • Types of outcomes desired but not measured

7. Research Participants • Completed surveys from: • 173 agencies (147 English and 26 French) • 214 practitioners (168 English and 46 French) • Telephone Interviews • 9 policy makers (out of 41 contacted) • 7 employers (out of 23 contacted) • 10 - 35 minutes • 2 interviews conducted in French

8. Evaluation is important In some sectors • 87% of agencies say it is important to evaluate work with clients • 92% of practitioners say it is important to evaluate work with clients Importance varies across sectors

9. Importance of Measuring Outcomes(Agency Responses, n=137) 2 = 25.04; p = .02

10. Importance of Measuring Outcomes(Practitioner Responses, n=166) 2 = 40.8; p < .01

11. Evaluation is important But counsellors do not evaluate their work with clients • 1993 study • 40% never evaluated their work with clients • 35% evaluated with client during interview • 2005 study • 33% did not answer • 56% reported data • Client flow, counsellor time use, etc.

12. Major Disconnect • 84% of agencies say they report the outcomes or impacts of their services • 29% of practitioners say they report the outcomes or impacts of their services How can this be?

13. 2005 Study:Agencies & practitioners What are the 3 most important outcomes you report? • Change in employment or educational status of the client and marginally • Skill development; financial independence, connectedness, self-confidence • Number of clients served • Client satisfaction • Programs completion • Service delivery • Cost-benefit

14. Data reported don’t match mandate • The best way to increase employment is to create more jobs, BUT • That is not the mandate of the agencies

15. What outcomes are you achieving that are going unreported or unmeasured? • Client empowerment • Client skill development (e.g., personal self-management skills), • Client increased self-esteem, • Client changes in attitudes (e.g., about their future, or about the nature of the workforce), • Client knowledge gains • Financial independence • Creation of support networks • More opportunities for clients and marginally • Community benefits; • Client satisfaction; Increased queries; • Political lobbying (agency)

16. Bottom Line … • Everyone wants more and better evaluation • Agencies • Practitioners • Policy makers • Funders • Employers BUT • It’s not being done in a way that permits making a connection between • what counsellors do and • the client changes that take place.

17. An attempt to meet the need • Canadian research Working Group for Evidence-Based Practice in Career Development (CRWG) • Create and validate a framework for evaluating the effectiveness of career development interventions • Intended for • Discussion • Feedback Would this work for you? How would this fit in your work place?

18. Approach for Demonstrating Value InputProcessOutcome

19. Approach for Demonstrating Value InputProcessOutcome • Indicators of client change • 1. Learning outcomes • Knowledge and skills linked to the intervention • 2. Personal attribute outcomes • Changes in attitudes, • Intrapersonal variables (self-esteem, motivation), • 3. Impact outcomes • Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training, etc. • Social and relational impact • Economic impact

20. Approach for Demonstrating Value InputProcessOutcome • Activities that link to outputs or deliverables • Generic interventions • Working alliance, microskills, etc. • Specific interventions • 1. Interventions used by service providers • Skills used by service providers • Home practice completed by clients • 2. Programs offered by agency • 3. Involvement by 3rd parties 4. Quality of service indicators • Stakeholder satisfaction.

21. Approach for Demonstrating Value InputProcess Outcome Activities that link to outputs or deliverables Specific interventions Career decision making Work-specific skills enhancement 3. Work search 4. Job maintenance 5. Career-related personal development 6. Other

22. Approach for Demonstrating Value InputProcessOutcome • Resources available • 1. Staff • Number of staff, level of training, type of training • 2. Funding • Budget • 3. Service guidelines • Agency mandate 4. Facilities 5. Infrastructure 6. Community resources

23. Projects currently in process-A Within organization mobility in SMEs • Applied for funding from SSHRC • Rejected • Applied for funding from CURA • Rejected • Applied for program development funding • Validation of evaluation model embedded inside program development • Approved, 3 years funding ($1.2 million)

24. Projects currently in process-B • Training program for government service providers • Validation of evaluation model embedded inside program development • Approved, 3 years funding ($800,000) • Competency-based training program • Develop evaluation tools in tandem with curriculum • Data will link professional training implementation (change in practices) client outcomes

25. Projects currently in process-C University-to-work transition program • University careers service • Existing program • Wanting external validation • 10 month project • Using evaluation framework • Creating process monitoring tools • Tracking employment status

26. Future Possible Directions Concepts under development Evidence Policy makers can relate to • Employment equivalence • Return on investment Food for thought and discussion

27. Employment equivalence • Consider a client who receives careers guidance and • Decides to return to school so he can • Find a better job that pays more money and less likelihood of unemployment • Employment status does not change • Considered a failure

28. Employment equivalence • Consider instead • In Canada, men 30 years old are 34% more likely to be employed if they have high school education (compared to those with no high school education) • Employment equivalence for taking training is .34 • Consider also, men described above earn on average $6,000 more money per year • Represents a return on investment

29. Return on Investment for Post Secondary Student Services Post secondary leavers vs. completers • 50% more likely to have difficulty keeping up with work load • leavers reported being unsure of what they wanted to do, • #1 reason for leaving school was “lack of fit.” • Completers were 50% more likely to report having a career plan that was a good match for their program • PSE graduates earn on average $6,500 more, return on investment is $3,250 per person per year.

30. Possible Employment Equivalence

31. Possible Employment Equivalence

32. Return on Investment for High School Career Education Programs School funding is based on student enrolment (person-courses) • 2 years after implementing career education program • Completion rates increased by 15% • number of students in their Registered Apprenticeship Program • Increased funding provided • 1.5 additional staff • more preparation time for teachers • perceived more positive work climate

33. General observations Agencies and practitioners agree: • Evaluation is important But • People don’t do it Perhaps these ideas will help integrate evaluation into service delivery

34. Measuring the Impact of Career Guidance: State of Practice in Canada Thank you Bryan Hiebert on behalf of the Canadian Research Working Group for Evidence-Based Practice in Career Development Robert Baudoin, Lynne Bezanson, Bill Borgen, Liette Goyer, Bryan Hiebert, Kris Magnusson, Guylaine Michaud, Vivian Lalande , Céline Renald, Reginald Savard, Michel Turcotte For more information: hiebert@ucalgary.ca