Evaluating California’s Web-based Training for Medical Assistants

1. Evaluating California’s Web-based Training for Medical Assistants Leslie Heyden, MA California Department of Public Health Immunization Branch Information and Education Section February 19, 2009

2. Overview • The purpose of the CDPH IZ Branch evaluation is to assess the use and effectiveness of the first two immunization skills elearning lessons: • Preparing Vaccines • Administering Vaccines • The elearning is custom developed, interactive web-based training. • The primary target audience is medical assistants in California VFC provider practices. • The lessons are available via eziz.org. 2

3. eziz.org • The new California VFC program website • A one-stop shop for California VFC providers and their staffs • A repository of a wide variety of immunization skills training and VFC program resources, including: • VFC forms (e.g., vaccine order and recertification forms, vaccine temperature and usage logs, patient eligibility screening forms) • Job aids, guidelines, posters, and up-to-the-minute news about vaccine availability 3

4. eziz.org—home page 4

5. Webinar learning objectives By the end of this webinar you will be able to: • List the objectives of the evaluation plan • Explain why CDPH IZ branch selected the EZ-IZ elearning lessons to evaluate • Describe the methodology CDPH developed to evaluate the training • Indicators • Targets • Methods, including data sources • Describe lessons learned 5

6. Why evaluate EZ-IZ’ elearning? • Is IZ Branch’s first elearning project • Has potential for high impact re: ability to reach unlimited number of learners with limited resources • Primary tool needed to conduct evaluation (LMS) was already being created • IZ Branch wanted to compare the “reach” of the elearning to existing classroom training 6

7. Goal of activity to be evaluated • To educate MAs in California VFC provider practices about best practices for vaccine preparation and administration 7

8. Evaluation objectives • By September 5, 2008 the first two elearning lessons of the EZ-IZ curriculum (Preparing Vaccines and Administering Vaccines) will launch, and the learning management system will begin tracking usage and learner performance. • By March 31, 2009, at least one person from 5% (approximately 180) of California’s active VFC provider sites (approximately 3,600) will have completed one of the elearning lessons (either Preparing Vaccines or Administering Vaccines) and have passed its posttest. 8

9. Evaluation objectives (cont’d) • By September 5, 2009, at least one person from 20% (approximately 720) of active California VFC provider sites will have completed the Preparing Vaccines and Administering Vaccines elearning lessons and passed the posttest for each. 9

10. Evaluation objectives (cont’d) • By November 30, 2008 all user feedback about the EZ-IZ website will have been gathered and analyzed, and recommendations made. • Recommendations concerning website usability will be forwarded to and resolved by the web developer and/or the web editor. • Recommendations concerning the EZ-IZ Curriculum and the lessons in it will be forwarded to and resolved by the instructional designer. • Recommendations about programming and usability issues within the elearning lessons will be forwarded to and resolved by the elearning vendor. • Recommendations concerning the learning management system (LMS) will be forwarded to and resolved by the LMS vendor. 10

11. Stakeholders • Management • John Talarico, Interim CDPH IZ Branch Chief • Tiffany Sutter, Section Chief: Information & Education • Natalie Nakahara, Project Manager and Creative Director • California VFC Program • Claudia Aguiluz, VFC Program Coordinator • VFC field staff and field services reps • VFC provider staff (particularly MAs who work in private provider practices) 11

12. Stakeholders (cont’d) • Immunization (federal, state, and local) • CDC (NCIRD) • Steve Nickell, Research Coordinator, IZ Branch • Sandra Jo Hammer, Nurse Consultant, IZ Branch • CDPH IZ Branch Evaluation team (Leslie Heyden, Natalie Nakahara, and Steve Nickell) • IZ Coordinators in LHDs • Others • California Medical Assistants Association (CMAA) and California medical assistant training programs 12

13. Table 3: Program Component/Activity Description OutcomesShort term Long term Inputs ActivitiesInitial Subsequent Outputs Funding Staff Time Review available materials & choose content Design training Employ vendor to program elearning and create learning management system Make training available via website Promote training Review of content by content experts Beta test on target audience Collect & assemble feedback Decide on changes Revise training Create a “one-stop shop” learning and resource portal for CA VFC providers, immunizers and other staff with clinical or administrative immunization-related responsibilities (Live) online elearning lessons, with learning management system tracking VFC providers will adapt to elearning as a training delivery method Increased skill and knowledge (pre- posttest comparison) about vaccination among medical assistants & other office staff Steady increase in the number of MAs who take the elearning lessons Fewer real-time “person hours” required to train MAs Fewer vaccine administration errors Higher coverage rates Fewer missed opportunities Fewer VPDs Increased provider/ user satisfaction

14. Primary data source—EZ-IZ LMS • An LMS (Learning Management System) is software that automates the administration of training. • The LMS registers users, tracks courses in a curriculum, and records data from learners • The LMS provides reports to management. 14

15. Evaluation question #1 • Q: Did the Preparing Vaccines and Administering Vaccines elearning lessons launch by September 5, 2008 (target)? • Indicator and method • The Preparing Vaccines and Administering Vaccines elearning lessons launch: request IZ branch staff to access lessons and report any problems accessing the lessons • Data source: feedback (verbal and email) from IZ Branch staff (and confirmation from LMS vendor) 15

16. Evaluation question #2 • Q: Did the learning management system (LMS) begin tracking usage and learner performance data by September 5, 2008 (target)? • Indicators and methods • LMS is operational: verify with vendor that LMS is operational • LMS is gathering usage data: obtain registration and login data from LMS report • LMS is gathering learner performance data: obtain pretest, lesson start, lesson completion, and posttest data from LMS report 16

17. Data source—registrationdata 17

18. Learning History Page 18

19. Data source—LMS Lesson Post-test Pre-test 19

20. Evaluation question #3 • Q: How many provider sites have at least one person who has completed one lesson and passed the posttest? • Target and indicator • 5% of provider sites have at least one person who has completed one lesson and passed its posttest by March 31, 2009 • Method • Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user • Data source: LMS 20

21. Data source—pre- and posttests 21

22. Evaluation question #4 • Q: How many provider sites have at least one person who has completed one lesson and passed the posttest? • Target and indicator • 20% of provider sites have at least one person who has completed one lesson and passed its posttest by September 5, 2009 (one year after launch) • Method • Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user • Data source: LMS 22

23. Evaluation question #5 • Q: What feedback has been received from EZ-IZ users that could indicate potential obstacles to accessing and completing the elearning lessons? • Targets and indicators • 100% of emails and 100% of all other feedback that could indicate technical problems with the website • 100% of emails and 100% of all other feedback that could indicate technical or any other problems with the lessons • Method • Emails about technical problems and overall feedback about the EZIZ website will be automatically sent to a specific (ezig.org) MS Outlook folder; phone calls will be logged and categorized by November 30, 2008 • Data source: emails submitted by users via links on eziz.org 23

24. Data source—feedback from users 24

25. Development of evaluation plan—Challenges • Very few challenges. IZ Branch Research Coordinator collaborated on and provided very valuable input into plan. • Due to limited resources, not able to build as robust an LMS as originally planned. • Proposed revisions to methodology during plan development required consulting with LMS vendor who then had to modify LMS database or do additional programming. 25

26. Lessons learned • Did not add question about MA’s experience; could not separate primary target audience from secondary target audience. • Learned that more data was needed than provided by LMS. Added question to QAR for field reps to administer. • The project team learned a lot during this entire pilot effort (including the evaluation plan) and is applying lessons learned during the current development of the next two lessons. 26

27. Added question to QAR 27

28. Questions? 28

29. Leslie HeydenHealth Educator & Instructional DesignerLeslie.Heyden@cdph.ca.gov(510) 620-3754www.eziz.org 29

30. This document: This document can be found on the CDC website at: http://www.cdc.gov/vaccines/programs/progeval/downloads/cali-eval-02-09.ppt