1 / 16

Training 2014 Session 501 February 4, 2014 10:45 a.m. – 11:45 a.m. ROI Basics

Training 2014 Session 501 February 4, 2014 10:45 a.m. – 11:45 a.m. ROI Basics. Learning Objectives. Explain why ROI is needed today Identify the five levels of evaluation Identify ten steps in the ROI Methodology Determine where and when to use ROI. • • • •.

alma-kirkland
Télécharger la présentation

Training 2014 Session 501 February 4, 2014 10:45 a.m. – 11:45 a.m. ROI Basics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Training2014 Session501 February4,2014 10:45a.m.–11:45a.m. ROIBasics LearningObjectives ExplainwhyROIisneededtoday Identifythefivelevelsofevaluation IdentifytenstepsintheROIMethodology DeterminewhereandwhentouseROI • • • • SpeakerName:PattiPhillips,Ph.D. Company:ROIInstitute,Inc. Phone:2056788101 Email:patti@roiinstitute.net Website:www.roiinstitute.net 1

  2. TheROIcalculationissimple. ProgramBenefits BCR= ProgramCosts NetProgramBenefits* ProgramCosts **** ROI= X100 $750,000 $425,000 BCR= = $750,000-$425,000 $425,000 ROI= X100 = *Netprogrambenefits=ProgramBenefits–ProgramCosts WhatmakesagoodROI? • • • • Setthevalueatthesamelevelasotherinvestments–15% Setslightlyaboveotherinvestments–25% Setatbreakeven–0% Setatclientexpectations WhatispositiveaboutanegativeROI? 2

  3. TomakeROIwork,fivepiecescometogethertocompletetheROIpuzzle. Evaluation Framework Case Applicationand Practice Implementation Strategy Operating Standardsor Guiding Principles AProcess Model EVALUATIONFRAMEWORK Thefive-levelevaluationframework*categorizesdatatoensurequestionsimportanttoall stakeholdersareanswered. Level Inputsand Indicators Reactionand PlannedAction Learning Application BusinessImpact ROI MeasurementFocus Measuresofinvestmentinthe trainingprogram Measuresparticipantsatisfaction withtheprogramorprocessand capturesplannedactions Measureschangesinknowledge, skills,andattitudes Measureschangesinperformance oraction Measureschangesinkeybusiness measures Comparestheprogrambenefitsto thecosts KeyQuestion Howmanypeopleattendedtheprogram?Howmany hoursoftraining?Howmanyofferings?Whatarethe costsperperson? Istheprogramorprocessrelevant,important,useful, andhelpfultotheparticipantandthejob environment? Didparticipantsincreaseorenhanceknowledge, skills,orperceptions?Dotheyunderstandthe informationshared?Dotheyhavetheconfidenceto dowhattheyneedtodo? Areparticipantsapplyingthe knowledge/skills/information?Ifyes,whatis supportingthem?Ifno,whynot? Howdoestheapplicationimpactoutput,quality,cost, time,andsatisfaction?Howdoweknowitwasthe programthatcausedthisimpact? Dothemonetarybenefitsoftheprogramexceedthe investmentintheprogram? 0 1 2 3 4 5 3

  4. Theevaluationframeworkservesrolesbeyondcategorizingevaluationdata.TheframeworkalsoTheevaluationframeworkservesrolesbeyondcategorizingevaluationdata.Theframeworkalso guidesthedevelopmentofprogramobjectivesandensuresstakeholderneedsareclearlyidentified. • • • Bybeginningtheneedsassessmentatthehighestlevelofneed,anappropriatesolutioncan beidentified.Thisisthefirststeptowardaligningyourprogramswithbusinessresults. Bydevelopingprogramobjectivesateachlevel, theprogramcanbepositionedforsuccess. Programobjectivesrepresentthemeasurestobe takenduringtheevaluationprocess,thereby, eliminatingtheguessworkwhenasenior executivesays“ShowMetheMoney!”Thisis yoursecondsteptowardaligningyourprograms withbusinessresults. Byevaluatingateachofthefivelevels,results importanttoallstakeholdersarereportedand businessimpactandROIcanbeconnectedtowhatpeopledowithwhattheylearned. Isolatingprogrameffectonresultsduringtheevaluationprocessisthethirdsteptoward linkingyourprogramswithbusinessresults. MatchingObjectiveswithLevels AnExercise Objective:Participantswilldemonstratetheirknowledgeofthelevelsofevaluationbycorrectly identifyingtheleveltowhicheachobjectiveshouldbemeasuredwithin3secondsfromthetimethe objectiveappearsonthescreen. Objective 100customerswillrequestadditionalinformationfromthewebsitewithintwo weeksofthepromotionalbriefing. Givenfivestepstocommunicateeffectively,participantswillrankthemin orderofapplication. Participantswillimplementthechangeprocesswithin3monthsofcompleting thecourse. Withintwoweeksoftheconferences,exhibitorswillfollowupwithtenqualified prospectsthatvisitedtheirbooth. Sixmonthsaftertheprogram,grievanceswillgodownonaverage7per monthduetotheprogram. Givenbenefitsof$250,000andcostsof$150,000,theROIwillbe67%. Level ________ ________ ________ ________ ________ ________ 4

  5. BusinessAlignment StartHere EndHere PayoffNeeds 5 ROIObjective 5 ROI BusinessNeeds 4 ImpactObjectives 4 Impact Initial Analysis Measurement andEvaluation PerformanceNeeds 3 ApplicationObjectives 3 Application LearningNeeds 2 LearningObjectives 2 Learning PreferenceNeeds 1 ReactionObjectives 1 Reaction 0 0 InputNeeds BusinessAlignmentandForecasting InputObjectives Program Inputs TheROIProcessModel 5

  6. SampleLinkageBetweenNeeds,Objectives,andEvaluation Level 5 4 3 2 1 Needs $1.6millioncostsduetofalsely rejectedsyringes Falserejectsisaproblem;false rejectsdefinedassyringesrejected whentheyareactuallyusable. Inspectorsareincorrectlyidentifying syringesasunacceptable Deficiencyinskillstorecognize unacceptablesyringes One-day workshop and introduction to newjobaid Level 5 4 3 2 1 Evaluation Programcostscomparedto monetarybenefitsofprogram Monetarybenefitdeterminedby costssavingsofreducednumber offalserejects Monitorfalserejectsforsix months Participantestimatesusedfor isolation Follow-upquestionnaireto participantstocheckfrequencyof skillapplicationandbarriersthree monthsafterthetraining. Unscheduledauditsoversix months Demonstrateabilitytoidentify acceptableandunacceptable syringes Indicateknowledgeand understandingbycompleting learningassessment Reactionquestionnaire administeredattheendofthe workshop Objectives 25%ROI Reducethenumberoffalse rejectsby10percentwithinsix months Followthefivestepprocess during100%ofinspections Utilizejobaidasneeded Identifybarrierstofollowingthe fivestepprocess Understandfive-stepprocessand jobaid Knowthedifferencebetween acceptableandunacceptable syringes Understandconsequencesof incorrectlycategorizingsyringes Programreceivesfavorablerating of4outof5on: oRelevanceofworkshop contentandjobaid oImportanceoffollowing thefive-stepinspection process oIntenttousejobaid duringinspections oOthermeasures importanttodesignand deliveryofcontent • • • • • • • • • • • • • • • • • 6

  7. SampleLinkageBetweenNeeds,Objectives,andEvaluation Level 5 4 3 2 Level 5 4 3 2 Evaluation Programcostscomparedtomonetarybenefitsof program=$123,008. Monetarybenefitdeterminedbytimesavings fromreductioninnumberofpeopleinmeetings, numberofmeetings,andlengthofmeetings. Timevaluebasedonaveragesalaryand benefitsoftimeofthoseinmeetings= $715,327.65. Timeinmeetings,numberofmeetings,and numberofpeopleinmeetingscapturedby meetingleaderswhoattendingthetraining.A meetingprofilewasdevelopedbyparticipants duringtheworkshopandthenagainthree monthsposttraining. Isolatingtechniqueisestimationduetothefact thatcontrolgroupswerenotfeasible,andtime andnumberofmeetingshadnotbeentracked historically.An81%confidencefactorwasused toadjustforerroroftheestimated.Thisisbased onrespondentaverage. Follow-upquestionnaireadministeredto participantsthreemonthsaftertheprogramto determineachievementwithobjectives. Facilitatorassessmentofmeetingprofile Writtentest Skillpracticeobservation Needs Thousandseachmonthdueto excessive,unproductive meetings. Teamspendstoomuchtimein meetings.Toomanypeople,too muchtime,toomanymeetings. Managersandsupervisorsare notmanagingtheirmeetings. Toomanypeopleattend meetings;noagenda;nofollow- up;nostructure;nodiscussion management Managersandsupervisors need todevelopmeetingmanagement skills;learningisimportant,but doing(Level3)ismore important. Objectives 25%ROI Threemonthsaftertheprogram,therewillbea reductionincostduetotimeinmeetingsas measuredby: •Reducednumberofmeetings •Reducednumberofpeopleinmeetings •Reducedlengthofmeetings •Increaseinproductiveuseoftimedueto fewermeetings Uponreturningtothejob,participantswill: •Developadetailedagendaoutliningthe specifictopicstobecoveredfor100%of meetings. •Establishmeetinggroundrulesatthe beginningof100%ofmeetings. •Followuponmeetingactionitemswithin threedaysfollowing100%ofmeetings. Duringtheworkshop, participantswilldemonstrate theirabilityto: •Developameetingprofile •Developameetingagenda •Selecttheappropriateparticipants • • • • • • • • Managediscussions Handleconflict Managefollow-upactivities • • • 1 1 One-dayworkshopand introductiontonewjobaid Programreceivesfavorableratingof4outof5 from80%ofparticipantson: •Contentrelevancetothejob Reactionquestionnaireadministeredattheend oftheworkshop • Intenttoapplycontentimmediatelyupon returntojob Othermeasuresimportanttodesignand deliveryofcontent • • 7

  8. THEROIPROCESS! CalculatingtheReturnonInvestment ofPerformanceSolutions HRSolutions TrainingSolutions/E-Learning MajorProjects TechnologySolutions OrganizationDevelopmentSolutions ChangeInitiatives ! ! ! ! ! ! CAPTURE COSTSOF SOLUTION EVALUATIONPLANNING DATACOLLLECTION DATAANALYSIS REPORTING DEVELOP EVALUATION PLANSAND BASELINEDATA COLLECTDATA DURING SOLUTION IMPLEMENTATION CONVERT DATATO MONETARY VALUE DEVELOP/REVIEW OBJECTIVES OFSOLUTION COLLECTDATA AFTERSOLUTION IMPLEMENTATION ISOLATE EFFECTS OFSOLUTION CALCULATE THERETURNON INVESTMENT GENERATE IMPACTSTUDY REPORT 5.ROI 1.SATISFACTION/ REACTION 2.LEARNING 3.APPLICATION/ IMPLEMENTATION 4.BUSINESSIMPACT IDENTIFY INTANGIBLE MEASURES ROICalculation-Example EVALUATIONFRAMEWORK Level 1.Reaction,Satisfaction &PlannedAction MeasurementFocus Measuresparticipantsatisfactionwiththe program/projectandcapturesplannedaction. Net Program/Project Benefits Program/ProjectCosts INTANGIBLE BENEFITS ROI = 2.Learning 3.Application 4.BusinessImpact 5.ReturnonInvestment Measureschangesinknowledge,skills, andattitudes. Measuresimplementationandchangesin behaviorintheperformancesetting. Measureschangesinbusinessimpact variables. Comparesbenefitstothecosts. Costsofprogram/project$230,000 Benefitsofprogram/project(1styear)$430,000 ROI==0.87x100=87% $230,000 $430,000-$230,000 8

  9. OPERATINGSTANDARDS/GUIDINGPRINCIPLES GUIDINGPRINCIPLE 1.Whenahigherlevelofevaluationisconducted,data mustbecollectedatlowerlevels. 2.Whenanevaluationisplannedforahigherlevel,the previouslevelofevaluationdoesnothavetobe comprehensive. 3.Whencollectingandanalyzingdata,useonlythe mostcrediblesources. 4.Whenanalyzingdata,choosethemostconservative alternativeforcalculations. 5.Atleastonemethodmustbeusedtoisolatethe effectsofthesolution. 6.Ifnoimprovementdataareavailableforapopulation orfromaspecificsource,itisassumedthatlittleorno improvementhasoccurred. 7.Estimatesofimprovementsshouldbeadjustedforthe potentialerroroftheestimate. 8.Extremedataitemsandunsupportedclaimsshould notbeusedinROIcalculations. 9.Onlythefirstyearofbenefits(annual)shouldbeused intheROIanalysisforshort-termsolutions. 10.CostsofthesolutionshouldbefullyloadedforROI analysis. 11.Intangiblemeasuresaredefinedasmeasuresthat arepurposelynotconvertedtomonetaryvalues. 12.TheresultsfromtheROIMethodologymustbe communicatedtoallkeystakeholders MEANING Tellthecompletestoryofprogram success. Conserveresourcesforthehigherlevelof evaluations. Usethemostcrediblesources. Choosemostconservativealternative. Givecreditwherecreditisdue. Makenoassumptionsfornon- respondents. Adjustestimatesforerror. Omittheextremes. Reportfirstyearbenefitsonlyforshort- termprograms Accountforallcosts. Reportintangiblebenefits. Communicateanduseyourevaluation data. 9

  10. Intheend,theROIMethodologygeneratessixtypesofdata.ThesedatarepresentthechainofIntheend,theROIMethodologygeneratessixtypesofdata.Thesedatarepresentthechainof impactthatoccursaspeopleareinvolvedinprograms,viewthemasrelevantandimportanttotheir jobs,acquireknowledgeskillandinformationneededtomaketheprogramwork,applywhatthey learn,andasaconsequenceimpactthebusinessintermsofimprovedoutput,quality,cost,time,job satisfaction,customersatisfactionandworkhabits.Weknowthatimprovementisduetotheprogram becauseweisolatetheeffectsoftheprogram.Toensurethebenefitsexceedthecostofthe program,theROIiscalculated. Remember:WhileROIistheultimatecomparisonofinvestmentinaprogramandthebenefitsofthe program,withoutthelowerlevelsofevaluation,wecannotexplainhowwedevelopedtheROIwe reported.Asimportant,withoutallofthedata,wehavenobasisforprogramandprocess improvement. Reaction&PlannedAction Learning&Confidence Application&Implementation IsolatetheEffectsoftheProgram Impact ROI IntangibleBenefits Howcanyouusethesedata? 10

  11. CASEAPPLICATIONANDPRACTICE Theframework,processmodelandguidingprinciplessetthefoundationforROI.But withoutapplication,theyaremerelytheoretical.Hundredsoforganizationsareputting theorytopracticebyroutinelyapplyingROI.TomakeROIworkforyou,however,you needtodevelopyourowncasestudies. Program PerformanceManagement (RestaurantChain) ProcessImprovementTeam (AppleComputer) Skill-BasedPay (ConstructionMaterialsFirm) SexualHarassment Prevention(HealthCare Chain) SafetyIncentivePlan(Steel Company) Diversity(Nextel Communications) RetentionImprovement (FinancialServices) Absenteeism Control/ReductionProgram StressManagementProgram (ElectricUtility) ExecutiveLeadership Development(Financial) E-Learning(Petroleum) ExecutiveCoaching(Nortel Networks) CompetencyDevelopment (Veteran’sHealth Administration) FirstLevelLeadership Development(AutoRental) KeyImpactMeasures Avarietyofmeasures,suchasproductivity,quality, time,costs,turnover,andabsenteeism Productivityandlaborefficiency Laborcosts,turnover,absenteeism Complaints,turnover,absenteeism,jobsatisfaction Accidentfrequencyrate,accidentseverityrates Retention,employeesatisfaction Turnover,staffinglevels,employeesatisfaction Absenteeism,customersatisfaction Medicalcosts,turnover,absenteeism Teamprojects,individualprojects,retention Sales Severalmeasures,includingproductivity,quality,cost control,andproductdevelopmenttime Timesavings,workquality,fasterresponse Variousmeasures–atleasttwopermanager ROI 298% 182% 805% 1052% 379% 163% 258% 882% 320% 62% 206% 788% 159% 105%7 11

  12. IMPLEMENTATIONSTRATEGY D Thelastpieceoftheevaluationpuzzleistheimplementationstrategy.Conductingone ortwoROIstudiesdoesnotcreateasuccessfulaccountabilitypractice.Youneeda strategytoensureasustainablepractice. WhydoestheROIMethodologywork?Becauseit: • • • • • • Providesabalancedsetofmeasures Followsastep-by-stepprocess Bridgesevaluationdisciplines Balancesresearchandreality Isflexible Presentsacredibleapproachtodevelopingdatastakeholdersneed. InitialstepsyoucantakeasyoudevelopyourROIstrategyinclude: 1. 2. 3. 4. 5. AssessyourreadinessforROI Identifyyourstakeholdersandtheirdataneeds Determinethepurposeofyourevaluationpractice IdentifyprogramssuitableforImpactandROIevaluation Developcapability MyActionPlan Purposedrivestheinitiativetoachieveoutcomes. Implementingourevaluationstrategygivesustheinformationweneedtohelpdevelop ourpeople,improveourprocesses,andcontributetothebottom-line! Weensurethattaxpayerdollarsareallocatedtotheprogramsthatdirectlysupportthe agencymission. 12

  13. CriteriaforselectingprogramsforevaluatingprogramsatLevel4andLevel5.CriteriaforselectingprogramsforevaluatingprogramsatLevel4andLevel5. Thinkaboutprogramsyouoffer.Rankthem1-5againstthefollowingcriteria.Thosethat resultinthehighestscorearecandidatesforROI. Criteria Expensive Long-LifeCycle Identifiedthroughcomprehensiveneedsassessment Alignedtoorganization/missionstrategy Highlyvisiblewithintheorganization/community Largetargetaudience Largetimecommitment Seniormanagement/clientinterest DevelopCapabilities Program2 Program3 Program4 Program5 Program1 Learn to: !Identifyexpectations !Selectordesignappropriate strategies !Communicateandgainsupport fortheevaluationplan !PlanROIstudies !Managedatacollection !Analyzeandinterpretdata !Reportconclusions !Makerecommendations !Usethedata Resources: !Booksonmeasurement,evaluation,and ROI •ROIResourceCenter (www.roiresourcecenter.net) •SagePublications(www.sagepub.com) •Berrett-KoehlerPublications (www.bkconnection.com) •Pfeiffer/JohnWiley (www.pfeiffer.com) !WorkshopsandWebinars •ROIInstitute,Inc.(www.roiinstitute.net) •TrainingMagazine (www.trainingmag.com) •ASTD(www.astd.org) •Evaluator’sInstitute (http://www.evaluatorsinstitute.com) 13

  14. StakeholderDataNeeds Stakeholders DataNeeded DataProvided DataGap 14

  15. ROIReadiness: Readeachquestionandcheckoffthemostappropriatelevelofagreement. Disagree Agree 1 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 2 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 3 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 4 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 5 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 1.Myorganizationisconsideredalargeorganizationwith awidevarietyofprograms. 2.Mydepartmenthasalargebudgetthatreflectstheinterestofsenior management. 3.Myorganizationhasacultureofmeasurementandisfocusedonestablishinga varietyofmeasuresincludingROI. 4.Myorganizationisundergoingsignificantchanges. 5.Thereispressurefromseniormanagementtomeasureresultsofourfunction. 6.Mydepartmentcurrentlyhasaverylowinvestmentinmeasurementand evaluation. 7.Mydepartmenthasexperiencedmorethanoneprogramdisasterinthepast. 8.Mydepartmenthasanewleader. 9.Myteamwouldliketobetheleadersinprocessesthatsupportourfield. 10.Theimageofmydepartmentislessthansatisfactory. 11.Myclientsaredemandingthatourdepartmentshowbottom-lineresults. 12.Mydepartmentcompeteswithotherfunctionswithinourorganizationfor resources. 13.Thereisincreasedfocusonlinkingprogramsfromalldepartments tothestrategicdirectionoftheorganization. 14.Mydepartmentisakeyplayerinchangeinitiativescurrentlytakingplaceinmy organization. 15.Ouroveralldepartmentalbudgetisgrowing,andwearerequiredtoprovethe valueofourprograms. 15

  16. Scoring YouarenotyetacandidateforROI. YouarenotastrongcandidateforROI;however,itistimetostartpursingsometypeof process. YouareacandidateforbuildingskillstoimplementROI.Atthispointthereisnopressureto showROI;however,nowistheperfectopportunitytoimprovetheevaluationprocess. Youshouldalreadybeimplementingacomprehensivemeasurementandevaluationprocess, includingROI. Ifyouscored: 15–30 31–45 46–60 61-75 16

More Related