Presenters Rod Wolford and Beth Larson

1. OSHA Trainers Exchange Conference 2013 Level 3 Training Impact Evaluations Presenters Rod Wolford and Beth Larson FOF COMMUNICATIONS

2. This educational material was produced under grant number SH-222242 from the Occupational Safety and Health Administration, U.S. Department of Labor to the Roofers & Waterproofers Research and Education Trust It does not necessarily reflect the views or policies of the U.S. Department of Labor, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

3. 4 Key Session Objectives At the end of this session, you will be able to: • Identify at least 3 of 5 essential elements of successful Level 3 evaluation • Indicate likelihood of incorporating essential elements into future Level 3 evaluation plan • Describe at least one specific way in which the elements could be adapted in an evaluation plan • Say you learned at least one thing you can use in your work FOF

4. Brief Review of Harwood Best Practices Kirkpatrick Training Evaluation Model • Level 3 – Behavior/Impact:changes in practices and/or conditions at least 3 to 6 months after training • Level 2 – Learning: immediate knowledge testing • Level 1 – Reaction: immediate perceptions of training quality/usefulness • (Ex: Rate instructor, materials, facility, etc.) Evaluating Training Programs, Donald Kirkpatrick, 1975 Preparing Instructional Objectives, Robert Mager, 1975Report from 1999 National Conference on Workplace Safety and Health Training, DHHS (NIOSH) FOF

5. Level 3: Assess Impact Level 3 plans must be tailored for Harwood grants Susan Harwood Training Grant Program • Tight budgets • Short timeframes • Big workplace hazards and risks • Possible nationwide scope of many Harwood grant programs FOF

6. Level 3 Planning Think of S & H training as a type of INTERVENTION • Delivers a TREATMENT to trainees • Primary purpose is to produce desired IMPACT(s) When should Level 3 evaluation begin? Should the training and the evaluation be designed simultaneously? Yes No I Don’t Know FOF

7. Level 3 Planning Overview of basic Level 3 evaluation tasks 1. Select a modelorlogic (rationale for training/evaluation design) 2. Develop a feasible hypothesis based on a credible premise 3. Design/administer measurement instruments (pre/post/follow up) 4. Conduct statistical analysis of data 5. Report findings to OSHA / provide feedback for training program FOF

8. TASK 1. Model or Logic for Training / Evaluation Possible default model Some better models • Constructivist Model – learner actively builds new ideas by exploring within a framework led by instructor as facilitator • OSHA Regs/Hazards Instruction Trainee Knowledge Safety • Extended Parallel Process Model (EPPM) – presents threat, empowers trainees to decide how to diffuse/eliminate it • Other Examples • Theory of Planned Behavior • Stages of Change Model • Health Action Process Approach FOF

9. TASK 2. Hypothesis Based on Credible Premise Draw credible premise(s) from the literature HYPOTHETICAL CREDIBLE PREMISES • More worker participation on safety committees less injury/illness • Systematic inspection/donning of PPE less injury/illness • Increased adoption of a specific control measure less injury/illness • Adoption of a specific safety policy less injury/illness Develop feasible Level 3 hypothesis to be tested CORRESPONDING HYPOTHESES • Harwood training more worker participation on safety committees • Harwood training systematic inspection/donning of PPE • Harwood training increased adoption of the specific control measure • Harwood training adoption of the specific safety policy FOF

10. Level 3 Measurable Variables Basic Before/After Design Level 3Follow Up Pre-Training Training Post-Training 90-180 Days Reaction/LearningMeasure BaselineMeasure ImpactMeasure Intervention • Perception • Knowledge • Attitudes • Beliefs • Intended Actions • Actual Actions • Knowledge • Attitudes • Beliefs • Incident Rates • Demographics • Prior training • Perception • Knowledge • Attitudes • Beliefs • Past Actions • Incident Rates • Motivation • Attitudes • Beliefs • Knowledge • Desired Actions FOF The feasible measures are PROXIES for reduced injury/illness.

11. Level 3 Measurable Variables ACTIONS CONDITIONS Change KnowledgePersists AttitudesPersist BeliefsPersist FOF

12. Identify ‘Unit of Analysis’ The major entity analyzed in evaluation tion • Individual – trainee, worker, supervisor • Group – crew, shop, job title • Organization – company, non-profit, training center, other • Combination Collect data • From this entity • About this entity FOF

13. Identify ‘Unit of Analysis’ EXAMPLE: Individual trainee as ‘unit of analysis’ FOF

14. Identify ‘Unit of Analysis’ EXAMPLE: Training center as ‘unit of analysis’ FOF

15. Define/Rank Variables Operationally define the variables • Independent variables (not likely impacted by training intervention) • Job status, age, education, work experience, prior training, etc. • May function as ‘confounders’ • Dependent variables (changes to be measured) • Motivations, attitudes, beliefs, knowledge, desired actions/conditions • Must correlate exactly with instructional objectives • May also function as ‘confounders’ Rank/prioritize/select variables • Not every variable is relevant – especially independent variables • Too many may suppress response rates FOF • Too many may yield a finding of change when there is none

16. TASK 3. Design/Administer Instrument(s) Pre-Training (usually on site data collection – captive audience) • Paper questionnaires? (key entry or scanned) • Audience Response System (ARS)? Post-Training (usually on site data collection – captive audience) • Paper questionnaires? (key entry or scanned) • Audience Response System (ARS)? Follow Up • ‘Paper’ questionnaires? • Internet entry? • Phone survey by interviewer? • Focus groups? • Subsequent trainings on different topic delivered to same trainees? FOF

17. Response Rates and Random Samples Pre/Post-Training (usually on site data collection – captive audience) • 100% response rates expected Follow Up (90 – 180 days after training) • 100% response rate from all trainees may be difficult to achieve • Random sample can be best feasible option regardless of survey method • 100% response from a valid random sample is superior to 70% or lower response from all trainees • pollsters use random sample sizes of 1,000 to 1,800 for accurate representation of all American voters • for several hundred trainees, a random sample of 100 or even 50 (>10%) can be a reliable representation of all trainees • avoid self-selected sample and/or “faux random sample” FOF

18. How to Draw a Valid Random Sample Ideally, sample size should be defined by ‘power calculation’ http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3 Some ‘Rules of Thumb’ • If training several hundred ‘comparable’ trainees, then >10% is safe* • If training 51 to 200 trainees, survey 20 minimum and up to 25%* • If training <50 trainees, survey all trainees Random sample must be generated by valid computation • Random Number Generator (RNG) • SPSS-PC • Excel (RAND) • Web-based calculators – select with care • Random Numbers Table – last resort FOF *Assumes valid random sample

19. Level 3 Example Survey Questions Fillable PDF Email surveys administered by state associations to a random sample of trainees FOF *Note: Incentives

20. Level 3 Example Result *Note: Error Bars Here FOF

21. Level 3 Example Survey Questions Stems 1. You are setting up a ladder. How likely are you to make sure it extends 3 feet above the upper landing? 2. If a particle breaks free from a grinding wheel, I can close my eye fast enough before it strikes to prevent injury. 3. If I bring a safety problem to my supervisor’s attention, I could be fired. 4. Your employer tells you to go into an empty tanker to make a quick 15-minute repair, even though you’ve had no confined space entry training. How likely are you to do the repair? Response Scales – easy to read Strongly Somewhat Somewhat Strongly Agree Agree Disagree Disagree Very Somewhat Somewhat Very Likely Likely Unlikely Unlikely Always Often Rarely Never FOF

22. TASK 4. Conduct Statistical Analysis Test null hypothesis that training has no impact FOF

23. Example Results: Two-sample t-test Error Bar shows percents of non-Hispanic and Hispanic strongly agreeing targeted work activities are hazardous The error bars are derived from Two-sample t-tests RED = PRE GREEN = POST FOF

24. Example Results: Correlations FOF

25. Level 3 Example Results: ANOVA FOF

26. Level 3 Example Results: Odds Ratios EXAMPLE: Odds Ratios CHANGES IN KEY PRACTICES AMONG CONCRETE WORKERS AT SIX-MONTH FOLLOW UP FOF

27. TASK 5. Report Findings / Feedback Executive Summary • Overview of program and evaluation • General results, conclusions, recommendations Describe Methods • Design of evaluation • Instruments (e.g., questionnaire) • Sampling procedures • Data collection procedures • Data analysis procedures Results • Description of findings • Charts and graphs of findings Discussion • Explanation of findings • Interpretation of results Conclusions/Recommendations • Conclusions about effectiveness • Recommendations FOF