Presented by: Elizabeth Childs Tammy Dewar Dave Whittington

1. Creative Approaches to Evaluating E-Learning Presented by: Elizabeth Childs Tammy Dewar Dave Whittington

2. Workshop Goals • Explore the strengths and limitations of the Kirkpatrick model and discuss e-learning implementation examples at each level. • Review a number of Return on Investment (ROI) models, their strengths and limitations, and how to choose one to fit your organizational context. • Develop a creative strategy for evaluating e-learning that can be customized to organizational goals and benchmark data from other organizations.

3. Bingo Game “We shouldn’t try to do something better until we first determine if we should do it at all” – Dwight D. Eisenhower –

4. Levels of Evaluation: • Level 1: Reaction and Planned Action • Level 2: Cognitive Learning and Retention • Level 3: On-The-Job Application • Level 4: Performance Results • Level 5: Return-On-Investment (ROI) • * Jack Phillips / Donald L. Kirkpatrick

5. * Jack Phillips / Franklin Covey Co.(from a large Telco client) Evaluation Frequencies:

6. Jack Phillips – The Need for ROI • Phillips advises trainers to ask themselves these questions: • What is the life cycle of the training program? • How many people will use it? • How much money is involved? • The Phillips rule of thumb: Big projects, use ROI. Small ones, forget it. But when you do it, do it right.

7. Travel time/costs Class time-participants Class time-instructors Opportunity costs-participants Records administration Development time Software/hardware costs Maintenance costs Measurable performance improvements Support costs-printing, mailing, facilities Typical ROI characteristics

8. Program Benefits • Benefit-Cost Ratio = Program Costs  Net Program Benefits • ROI = 100 Program Costs Calculating ROI

9. Different Ways of Representing ROI • As a percentage. • If the customer gains benefits of $1,000,000 in 12 months on a total investment of $250,000 in the same time period, their ROI expressed as a percentage can be calculated as follows: • If, Return = Payback – Investment ROI = [(Payback – Investment)/Investment)]*100, or in this case: [($1,000,000-$250,000)]/$250,000*100 = 300% • As a ratio • Return/Investment = $1,000,000/$250,000 = 4:1 • As a time to break-even. • ($250,000/$1,000,000)*12 months = (0.25)*12 = 3 months or 90 days

10. Different decisions required = different focus for ROI Source: Jay Cross “A Fresh Look at ROI” www.learningcircuits.org/2001/jan2001/cross.html

11. ROI – Best Practices • Know what you’re measuring! (isolate data from other factors) • Sample, don’t saturate! (pick a few impressive factors) • Show them the money! (both tangible and intangibles-travel time, more job satisfaction) • Compare apples to apples! (pre- and post- estimates for same factors) • Source: Jack Phillips

12. ROI Model # 1Docent’s approach • In order to collect the data required to substantiate a 400% or more ROI claim, Docent prefers to work directly with the “CXOs” and line of business managers, and not the traditional “training manager.” • The traditional training department metrics will typically yield ROIs of roughly 100% for a web-based training project.

13. Docent ROI Approach White Paper (2002) • When developing the ROI estimate, the analysis should consider the following four categories of potential benefits: • Hard cost savings vs. alternative solutions • Hard revenue impacts vs. alternative solutions • Soft competitive benefits vs. alternatives • Soft benefits to individuals vs. alternatives

14. ROI Model # 2Collis and Moonen, 2001 • Used to predict ROI and provide information about how to implement e-learning for various stakeholders • They suggest concentrating on a more intuitive (but still systematic) calculation that emphasizes the relative comparison of ROI for various stakeholders. • They compare costs (investments) and benefits (returns) on a small number of key issues that matter most to those involved.

15. Collis and Moonen ROI Model • Keep focus local • Identify major actors • Concentrate on project outcomes relating to economic and monetary terms, qualitative terms (often attitudes) and efficiency terms (time or effort) • For each actor, identify perceived impact in each of the project outcomes using a scale from –5 (very negative) to +5 (very positive)

16. ROI Prediction for prospective student WWW site

17. ROI Model # 3Doughty and Shaw, 2000 • A model to help inform investment decisions • Based on the main concerns - Quality, Time and Cost - the method does not reduce these to a single measure • Respects multiple stakeholder perspectives • Considers levels of confidence and risks • Used to predict/estimate ROI

18. Doughty and Shaw, 2000 Simplified Process • Identify the Problem/Opportunity • Identify Stakeholders • Record their different perspectives • Identify alternatives (maybe go/no go) • Record perceived costs and benefits and where the evidence will come from • Estimate levels of confidence and risks associated with errors • Review alternatives

19. ROI Model # 4 Value Creation – Forman (2002) • ROI not just about cost reduction • Relies on the definition of the term “value” within the organization • Categories of value measures that can be defined and used as priorities in the evaluation are based on what is real and vital for the organizations success. They commonly include: • Business strategy measures • Time measures • Human capital measures • Customer measures

20. ROI Model # 5 The Value Proposition -Rosenberg • Focus of evaluation is on ability of elearning to deliver value to the business • Elearning value = the sum of (cost efficiency + quality + service + speed) where: • Cost efficiency is measured by delivery cycle time and productivity • Quality is measured by assessing the increase in knowledge/skill/ability intended by the program. • Service is measured by assessing how accessible is the elearning program • Speed is measured by assessing how responsive is the system is to the changing demands of the work.

21. Return on Expectations • Benedet suggests that organizations considering implementing an ROI strategy start the process by delivering an ROE (return-on-expectation) survey. “It should be highly visible, and should pave the way for future sponsorship and participation in real ROI projects.” She advises companies to look at an opportunity and determine whether measures are in place. “If not,” she advises, “do a return on expectations.” • Source: Brenda Benedet, Director of e-learning strategies for SkillSoft

22. Choosing Your ROI Factors • Have you defined what “counts” as return on investment in your organization? • Can you show a baseline for comparisons? • How well do you track current costs of training development and implementation? • How closely can performance data be identified, isolated, and linked to the training?

23. References Doughty and Shaw http://www.elec.gla.ac.uk/TILT/cbtalk/ ASTD website – www.astd.org Docent website – www.docent.com Flexible Learning in a Digital Age, Betty Collis and Jef Moonen, Kogan Page Ltd, London, 2001. Handbook of Training and Evaluation and Measurement Methods (3rd ed.), Jack J. Phillips, Gulf Publishing, Houston Texas, 1997. In Action: Implementing Evaluation Systems and Processes, Jack J. Phillips, (Editor), ASTD, Alexandria, Virginia, 1998.

24. References Continued Evaluating Training Programs (2nd Edition), Donald L. Kirkpatrick, Berrett-Koehler Publishers, San Francisco, California, 1998. Evaluating Professional Development. Thomas R. Guskey. Corwin Press: Thousand Oaks, CA (2000). e-Learning. M. J. Rosenberg. McGraw-Hill, New York. (2001). ASTD E-Learning Handbook. McGraw-Hill, New York. (2002).

25. Resources • Course Evaluator • “More than a year ago, MVU began developing rigorous standards to guide the design and evaluation of online course quality. Based on decades of research and the work of the best minds in the field of Instructional Design, we have recently completed a comprehensive set of standards that can now be used to design and evaluate online courses. http://standards.mivu.org/ • Working through evaluating elearning http://www.astd.org/virtual_community/research/What_Works/e-learning/top_10.html

26. Contact Info • Elizabeth Childs – elizabeth.childs@royalroads.ca • Tammy Dewar – tammy@calliopelearning.com • Dave Whittington – dave@calliopelearning.com