Evaluation Procedures

1. Evaluation Procedures Dr. Steve Training & Development INP6325

2. Perspectives on Training • Trainee – What if trainee does well in training, but gets fired or laid off for being ineffective on the job? • Trainer – What if trainee develops what s/he thinks is effective training, but supervisors won’t let workers use newly learned techniques/skills? • Organization – What if organization spent big money on training program, but employees are still not effective?

3. Applied Questions • Training specialists interested in: • Whether criteria (performance, $ savings) indicate improvement following training • Whether improvement was a result of training • Whether training will be equally effective with different group of trainees • Whether training will be beneficial to other organizations.

4. Research Questions • Which of two or more training programs is the best and why? • Which type of training works best with which type of worker? • What type of organization can benefit most by this type of training?

5. Types of Training Researcher (Randall) • Negativists – Evaluating training is either impossible or unnecessary • Often decisions made based on anecdotal evidence • Positivists – Only rigorous scientific evaluation is worthwhile • If there’s no experimental data, it’s a waste of time • Often don’t have resources for full-blown experiment • Frustrates (Activists) – All training should be evaluated, but must recognize that evaluation quality varies depending on constraints.

6. Training Evaluations Training Evaluation only as good as the: • Question that is asked • Appropriateness of the criteria • Evaluation design

7. Training Evaluations: Ethical Issues Ethical issues of empirical study • Use of control group means some may not get training • Eliminates promotion opportunities • Control group may feel slighted • Trainees act differently if know in experiment • Hawthorne Effect

8. Formative vs. Summative Evaluation • Formative Evaluation – evaluate whether training program is operating as planned BEFORE it is implemented • Focus on process criteria – how training should be implemented • Summative Evaluation – Does training produce the expected outcome? • Trained vs untrained group comparisons’ • Comparative summative evaluation – which of 2 or more training approaches produces greatest benefit

9. Threats to Validity of Training • Internal Validity – did training make a difference in this situation? • Training Validity – Does trainee’s training performance meet criteria for training program? • Transfer Validity – Does trainee’s job performance meet criteria? • External Validity – Can training results be generalized to other trainees or settings? • Intraorganizational Validity – Will training be effective on new group of trainees? • Interorganizational Validity – Can training program be used successfully at other organizations?

10. Threats to Internal Validity • History – Events that happen between pre- & post- tests have nothing to do with training • Contamination of results (ex: layoffs, accidents, holidays, etc.) • Maturation – Biological or psychological effects of time on training • Effects of fatigue or disinterest

11. Threats to Internal Validity • Testing – pre-test sensitizes trainee to upcoming post-test • Trainee prepares for it, or learn from pretest • Ex: GRE score  GRE course  GRE score improves • Instrumentation – questionable reliability of test measure • If fluctuation in scores is normal, can’t attribute change to training

12. Experimental • Control Pretest Posttest Threats to Internal Validity • Statistical Regression – Extreme (hi or lo) pre-test scores will often regress to the mean on post-test • Differential Selection of Participants – if allow participants to volunteer, experimental group may be more highly motivated • Random selection, random assign, or matched groups

13. Threats to Internal Validity • Experimental Mortality – differential loss of participants between groups • Trainees who did poorly on pre-test may get discouraged, fired, quit, etc. • Training may coincide with other project deadlines • Interactions – differential effects that one threat might have on another • Ex: testing effects different in each group due to selection

14. Threats to Internal Validity • Diffusion or Limitation of Treatments – members of one group share information with the other • Compensatory Equalization of Treatments – to avoid perceived differential treatment, some alternative might be given to control group • No longer true control, but 2nd experimental group

15. Threats to Internal Validity • John Henry Effect – competition between experimental and control group increases motivation of control group to work harder • Demoralization of Control Groups – passive-aggressive response to not being selected for training

16. Threats to External Validity • Reactive Effect of Pre-testing (sensitization)- effects of pre-test lead to increase sensitivity of instruction • Increased attention to material that was seen in pre-test • Interaction of Selection & Experimental Treatment (representative sample) – characteristics of the group chosen for evaluation may be different than in future groups

17. Threats to External Validity • Reactive Effects of Experimental Settings – Experimental group knows they’re being observed causing them to react differently from future groups • Hawthorne, Guinea pig, or Pygmalion Effects • Multiple-Treatment Interference (carryover)– problem with within subjects experiments • participant is exposed to more than one treatment - previous treatments will affect later ones

18. Training Evaluation Designs • Pre-Experimental Designs • One Shot Case Study: • No control group X T2 • One Group Pre-Test/Post-Test: • Before and after comparison, no control group T1 X T2 • Static Group Comparison: X T2 -- T2

19. Training Evaluation Designs • True Experimental Designs • Pre-test/Post-test Control Group: • Controls for most internal validity threats (except diff treatment) R T1 X T2 R T1 -- T2 • Solomon 4-Group Design: • Controls for both internal and external threats R T1 X T2 R T1 --- T2 R -- X T2 R -- --- T2 • Post-Test Only Control: • Like static group, but with randomization R X T2 R T2

20. Training Evaluation Designs • Quasi-Experimental Designs • Time Series • Longitudinal method rules out maturation, testing, stat regression T1 T2 T3 T4 X T5 T6 T7 T8 • Non-Equivalent Control Group • Same as pre-test/post-test control, but not random selection T1 X T2 T1 --- T2

21. Other Evaluation Methods • ANCOVA – use pre-test as covariate in comparing post-test means • Example covary measure of ability prior to training • Useful when small r between pre & post test R T1 X T2 R T1 --- T2 • Correlations – correlate training performance to on-the-job performance • Doesn’t necessarily mean training had effect • Content Validity – SMEs opinion that KSAs covered in training were those identified in needs assessment