Evaluation Roundtable Indirect Sales Program Analysis

1. Evaluation RoundtableIndirect Sales Program Analysis Katie BauerEvaluation & Metrics, AnalystHRD - Performance Center Katie.bauer@nextel.com Desk 703-264-4152 NEXTEL. Done. | ™ Danny BrownEvaluation & Metrics, ManagerHRD - Performance Center Daniel.brown@nextel.com Mobile 571-436-4576 NEXTEL. Done. | ™

2. Agenda • What are the evaluation learning levels? • What is Metrics That Matter (MTM)? • NU Evaluation Forms • Supplier Report Review • Small Group Brainstorming • Ideas for Action • Appendix • Grading Scale Confidential.

3. Kirkpatrick/Phillips Methodology:Five Levels Learning • Reaction/Satisfaction (Level 1) • Instructor • Environment • Courseware • Learning (Level 2) • Learning Effectiveness • Application (Level 3) • Job Impact • Business Impact (Level 4) • Business Results • Return on Investment (Level 5) • Expectations Confidential.

4. Level 1 Did they like it? Level 2 Did they learn? Level 3 Do they use it? Level 4 What were the results? Level 5 What’s the ROI? Metrics That Matter Balanced Scorecard for Measurement MTM is a systematic, replicable and comparable ROI model. Confidential.

5. Reasonable Learning Indicators Independent studies have shown that there is an enormously high cost to data accuracy versus obtaining a reasonable indicator. Organizations should balance when to take the extra time and money to yield highly statistical results. Given how executives make decisions, often times reasonable data provided in a more timely manner outweighs data with more precision delivered in a less timely manner. In your organization, what best describes the level of accuracy that analytics must have in order to be perceived as useful for information decision-making purposes of senior management? Source: KnowledgeAdvisors Learning Analytics Best Practices Research Study, 2004. Confidential.

6. Review Evaluation Forms • Post Event Instructor Led Training • 60 Day Follow Up Evaluation • Scaling Use in Evaluations 7-Point Lykert Scale • 7 = Strongly Agree • 1 = Strongly Disagree • For comparative analysis, the following HRD Performance Goals have been benchmarked based on the following scale: • Overall Favorable: > 5.60 • Scoring: • B+ 6.2 - 6.29 • B 5.8 - 6.1 • B - 5.6 - 5.7 Confidential.

7. Business Impact Level 4 Estimation Isolation Adjustment Reaction Level 1 Learning Level 2 ROI/ROE Level 5 Application Level 3 Post Event Evaluation Confidential.

8. Level 2 Level 3 Level 5 Level 4: Estimation, Isolation & Adjustment Follow Up Evaluation Confidential.

9. How We Measure Training – “Metrics That Matter” • Post event & 60 day Follow-up Evaluations • Performance to Goals • Report Card • Supplier vs. NU benchmarking • Human Capital ROI Score Card • Monetary Benefits, Benefit to Cost Ratio & ROI percentage • Time to Impact • Trend Analysis Confidential.

10. Response Rates • Most organizations report average post event response rates of 65% • Nextel average 77% • Commsult 21% • Most organizations report average follow-up response rates of 38% • Nextel average 10% • Commsult 41% Confidential.

11. Performance to Goals survey results compared to user, pre-set goals INSERT P2G REPORT Confidential.

12. Program Report Card a summary of learning performance by survey question category and by survey question benchmarked against all Nextel University for the timeframe specified Confidential.

13. Human Capital ROI Score Card • Comparison of post event to follow up data • Estimation • Isolation • 65 % Adjustment – Tennessee Valley Authority study • Conservatism • Bias • These principles form a powerful model in tabulating a systematic, replicable and comparable ROI model Confidential.

14. Business Results “This training will improve my job performance and productivity.” Job Performance Change: Adjusted Percent of Improvement (API) due to Training The industry standard for API is around 2-3%. Confidential.

15. Time to Job Impact 88.7% participants applied training within 4 weeks! Confidential.

16. Learning Effectiveness: Trend Analysis Forecast Performance Improvement 6.18 5.89 5.60 5 % increase overtime Learning Effectiveness - out of 7 Training Industry • Post Event • immediately after class Follow Up - 60 days later Time Confidential.

17. Cost Information Confidential.

18. Salary & Monetary Calculations Confidential.

19. Analyst Worksheet A tool enabling the learning team to conduct more comprehensive business impact and ROI analysis. The tool allows you to store multiple business results tied to multiple programs and show trainings impact and ROI. Confidential.

20. Vendor Management • Poor performance identified quickly • Instructor removed from program before any other occurrences • Continuous feedback loop between Nextel and Supplier Confidential.

21. Small Group Brainstorming • What seems to be working? • Areas for improvement? Confidential.

22. Ideas for Action • Where do we go from here? • What are your take-aways from today? • Shall we do this, again, in 3 months, 6 months, etc.? Confidential.

23. Thank You! Katie BauerEvaluation & Metrics, AnalystHRD - Performance Center Katie.bauer@nextel.com Desk 703-264-4152 NEXTEL. Done. | ™ Danny BrownEvaluation & Metrics, ManagerHRD - Performance Center Daniel.brown@nextel.com Mobile 571-436-4576 NEXTEL. Done. | ™

24. Appendix

25. Grading Scale Confidential.