Session Topics • Purpose and Benefits of Evaluating Training • Measurement Process • Issues in Assessment Development • Formative vs. Summative Evaluation
Session Topics • Kirkpatrick’s Four Levels • ROI: Methodology and Calculation • Three Measures of Central Tendency • Uses of Analysis
Purpose of Evaluating Training What is the Purpose of Evaluating Training? • Determining business impact • Improving the design of the learning experience • Determining the content’s adequacy
Benefits of Evaluating Training What are some benefits of evaluating training? • Secure client support and build relationships w/client • Allows practitioner to see whether the results of learning are consistent w/business analysis and needs assessment • Helps focus the training
Benefits of Evaluating Training What are some benefits of evaluating training? • Validates performance gaps and learner needs • Determines where training is the solution to a gap
Measurement Process List the basic steps of evaluation. • Identify evaluation goals • Develop an evaluation design and strategy • Select and construct measurement tools • Analyze data • Report data
Issues in Assessment Development Validity vs. Reliability • Validity: measuring what you intended to measure • Reliability: ability of the same measurement to produce consistent results over time
Issues in Assessment Development How do Validity and Reliability Relate to Each Other? • Validity: concerns what is being measured • Reliability: concerns how consistently the measure is measuring what it is intended to measure
Issues in Assessment Development What are the five types of validity? • Content validity • Construct validity • Concurrent validity • Criterion validity • Predictive validity
Issues in Assessment Development What is content validity? • Extent to which the items on the evaluation reflect a representative sample of that which is to be measured • Do the items on the evaluation encompass what was taught?
Issues in Assessment Development What is construct validity? • Extent to which the evaluation we implemented contained the content we intended it to and the outcome we measured was what we intended to measure • Do the items on the evaluation test for what we wanted to test for?
Issues in Assessment Development What is concurrent validity? • Extent to which an assessment’s results agree with the results of other assessments administered at approximately the same time to measure the same characteristics • Does the assessment bear out or conflict with known and accepted standard measures or criteria?
Issues in Assessment Development What is criterion validity? • Extent to which a test predicts some criterion (measure of performance), usually in the future • Does doing well on this assessment predict that the person will do well elsewhere later in life?
Issues in Assessment Development What is predictive validity? • Extent to which the results obtained by the evaluation agree with results obtained from more direct and objective measurements taken later on • Does doing well on this assessment predict that the person will do well when tested on the same material later on?
Issues in Assessment Development If you don’t have a statistician at your disposal, what other source do you have for validating your evaluation? • SME
Formative vs. Summative Evaluation • What is Formative Evaluation? Assessment of the effectiveness of the training program while the program is still under development. • What is Summative Evaluation? Assessment of the effectiveness of the training program after the program has been delivered.
Kirkpatrick’s Four Levels What are Kirkpatrick’s 4 Levels of evaluation? • Reaction • Learning • Behavior • Results
Kirkpatrick’s Four Levels List some of the tools you could use to conduct a Level 1 evaluation. • Reaction sheets • Word-of-mouth feedback
Kirkpatrick’s Four Levels Which level assesses the overall impact of the program on the organization? • Level 4
ROI: Methodology and Calculation • What are some ways to assign monetary value to program improvement? • Benchmarking • Expert input • Estimates • Control groups • Customer input
ROI: Methodology and Calculation What is the formula for deriving ROI? ROI = Total Benefits – Program Costs X 100 Program Costs
Three Measures of Central Tendency • What is another word for “measures of central tendency?” • Averages • What are the three “measures of central tendency” we are concerned with today? • Mean • Median • Mode
Three Measures of Central Tendency • Define Mean • The sum of all numbers divided by the number of values that make up the sum • 64 + 73 + 82 + 98 = 317 (sum of values in list; aka Σx) • 317 ÷ 4 = 79.25 (mean; Σx ÷ n, where n is the number of values in the list)
Three Measures of Central Tendency • Define Median • The number separating the higher half of a sample from the lower half. The median of a finite list of numbers can be found by arranging all the observations from lowest value to highest value and picking the middle one. If there is an even number of observations, the median is not unique, so take the average of the two middle values.
Three Measures of Central Tendency • Define Median • Less sensitive to extreme scores than the mean is, so the median is a better measure than the mean for highly skewed distributions.
Three Measures of Central Tendency • Define Median • 64, 73, 82, 98 (scores from previous slide) • 73 + 82 = 155 (even number of scores, so add the two in the middle and… • 155 ÷ 2 = 77.5 …divide the result in half to get the median)
Three Measures of Central Tendency • Define Mode • The most frequently occurring score in a distribution. • Greatly affected by sample fluctuations, so not recommended for use as the only measure of central tendency. • Distributions can be multi-modal; contain more than one mode.
Three Measures of Central Tendency • Define Mode • 3 + 8 + 5 + 3 + 9 + 2 + 6 + 3
Three Measures of Central Tendency • Define Mode • 3 + 8 + 5 + 3 + 9 + 2 + 6 + 3 • Mode = 3
Uses of Analysis • What are some of the uses for ROI evaluation, cost-benefits, utilities, and forecasting? • To measure a program’s economic contribution • To measure the degree of change in retention • To isolate the effects of training
Uses of Analysis Case Problem Activity • An organization has been running the same new hire orientation program for over a year. • The HR director wants to determine the program’s effectiveness, with a desire to improve it. Also, HR staff want to know why they seem to be answering the same questions repeatedly from new hires.
Uses of Analysis Case Problem Activity • Your boss asks you, the primary trainer, to identify the strengths and weaknesses of the program. • Company management has indicated they would like to offer the new hire orientation program online, in connection with links to the other helpful information that HR provides.
Uses of Analysis Program Evaluation Activity As a program evaluation consultant, you have been hired to evaluate a training program at Horizons, Inc. Complete the handout to perform your evaluation.
Session Review • Purpose and Benefits of Evaluating Training • Measurement Process • Issues in Assessment Development • Formative vs. Summative Evaluation
Session Review • Kirkpatrick’s Four Levels • ROI: Methodology and Calculation • Three Measures of Central Tendency • Uses of Analysis
Next Session • Selecting and Planning Your Work Product