Evaluating Readiness: Course Content and Program Evaluation Bioterrorism Training and Curriculum Development Program Sep

1. Evaluating Readiness:Course Content and Program EvaluationBioterrorism Training and Curriculum Development ProgramSeptember 21, 2004 Lynn Rothberg Wegman, M.P.A. Director, Division of State, Community, & Public Health (DSCPH) Bureau of Health Professions (BHPr) Health Resources and Services Administration (HRSA) Department of Health and Human Services (DHHS)

2. Evaluation • Varying levels of evaluation • Individual health care provider’s level of preparedness • Is each individual course/module an effective training tool? • Is each grantee effective in overall training strategy for the geographic area? • Is the national BTCDP an effective means of training the Nation’s health care providers? • Ultimate question: Are the Nation’s health care providers better prepared to respond in an emergency?

3. Preparedness Needs of Healthcare Workforce • True need vs. perceived need • Training • Drills/Exercises • Retention/re-training • Level of competencies- who needs to know what? • Everyone needs to have basic level of training- who needs intermediate or advanced (who determines the need), how should it be delivered? • HCPs have different roles in different states

4. Issues to Consider in Training the Healthcare Workforce • Important to evaluate what’s working and what’s not working and why • Take developed courses and modify them for the audience • Different states, different institutions have different needs and procedures to provide training • Coordinate drills with HRSA Hosp Prep, DHS, and CDC as an evaluation strategy • JCAHO will allow tabletop drills as long as the community is brought into it • Full scale vs. tabletop drills

5. Issues to Consider in Training the Healthcare Workforce (cont’d) • Need evaluators trained to observe drills • Students (CD) should be involved in simulations rather than drills • Can learn from large scale drill if you can evaluate it properly (have enough staff, etc.) • Practicing HCPs don’t necessarily want public health-focused training and competencies

6. How Do We Measure Preparedness? Possible options to consider: • Core Competencies • 15 potential DHS scenarios of biological, chemical, nuclear, natural disasters, etc. • Universal Task List of general needs to respond to the 15 scenarios (not details of how to respond) • CDC’s 19 preparedness topics • Columbia’s Hospital Workforce competencies

7. Curriculum Development Perspective

8. CD Core Content • What level of training is required for health professions students? • Defined by the future role of each health care provider serving in an emergency • Basic competencies for all health care providers vs. targeted, advanced competencies for specific groups of providers • “Must knows” for all core concepts • Add discipline-specific material (same topics but varying levels of knowledge)

9. CD Evaluation • How are competencies being measured? • Competencies defined as a measurable skill/knowledge • Areas of student evaluation (pre and post tests): • Cognitive knowledge of preparedness • Attitudes toward preparedness • Prior exposure to preparedness content • National health professions licensing exams: what questions relate to emergency preparedness? • CE requirements- one state requires 2 hours of CE on emergency preparedness for licensure renewal • Problem-based learning (core) • Schools creating own evaluation

10. CD Program Evaluation • Needs Assessment • Course Content Validity • Faculty Evaluation and Development • Course Evaluation: for faculty, students, and instructional technology methods

11. Continuing Education Perspective

12. CE Content • There should be a basic core program that leads to more detailed or advanced training • What are the real basics that everyone needs to know? • Role • Piggyback on training on requirements (like JCAHO requirements on training for all workers) • Practicing professionals want to know what their responsibilities are and not all nuances about diseases

13. CE Content • Need to provide training for health professionals; not just first responder training • Match local training needs to national goals • Tiered educational programs are part of the answer • Just in Time training to help combat burn-out on terrorism preparedness training • Need to refer back to the four program goals in designing/developing content

14. CE Content • ½ doing NIMS; all doing HEICS • ½ doing “in depth” courses • All doing broad spectrum training • More than ½ are part of a state strategy for training healthcare professionals; all believe they should be part of a comprehensive state strategy

15. Evaluation: Competencies • Have to be based on training needs analysis for each group of trainees • CE grantees develop their own competencies based on variety of available competency sets • Competencies and objectives are derived from needs assessments • ½ of CE awardees believe no single set of national competencies exist that are easily adaptable to state and local needs • Curricula in the CE program use competencies drawn from a variety of sources: ODP/ERG, ACEP, CDC/Columbia, ASTM, etc.

16. Evaluation: Competencies • Competencies should not be developed without including appropriate groups: accrediting bodies, professional associations, federal agencies, provider groups • Everyone believes there needs to be consensus on national preparedness competencies for basic level training • Competencies must be broadly defined at national level, but with specifics developed at awardee level

17. Evaluation: Courses • Pre/Post testing alone is not sufficient • Tiered approach to evaluation • Future follow-up testing alone may not be sufficient to determine knowledge/competencies • Basic competencies “re-delivered” may be better than evaluations of retention of specifics

18. Evaluation: BTCDP Program • How can we help HRSA determine whether we have met the four program goals? • National impact: • Numbers trained by discipline • Contact hours • Geographic region • Population served • Special populations • By describing how we are helping the state meet preparedness goals

19. BTCDPQuestionnaire

20. CE Awardees Are you implementing a basic emergency preparedness orientation course? Number of CE Awardees: 19 Number of CE Respondents: 17 Number of Yes Responses: 16

21. Data from CE Awardees

22. Data from CE Awardees

23. CE Awardees Are you using core competencies in your bioterrorism and emergency preparedness courses? Number of CE Awardees: 19 Number of CE Respondents: 17 Number of Yes Responses: 14

24. CE Awardees Have you conducted any community-based drills as part of your curriculum or course content? Number of CE Awardees: 19 Number of CE Respondents: 17 Number of Yes Responses: 8

25. CD Awardees Are you implementing a basic emergency preparedness orientation course? Number of CD Awardees:13 Number of CD Respondents: 13 Number of Yes Responses: 8

26. Data from CD Awardees

27. Data FromCD Awardees

28. CD Awardees Are you using core competencies in your bioterrorism and emergency preparedness courses? Number of CD Awardees: 13 Number of CD Respondents: 13 Number of Yes Responses: 12

29. CD Awardees Have you conducted any community-based drills as part of your curriculum or course content? Number of CD Awardees: 13 Number of CD Respondents: 13 Number of Yes Responses: 3

30. Thank You Contact Information: Lynn Rothberg Wegman, M.P.A. Director, Division of State, Community & Public Health, HRSA, BHPr 301-443-1648 lwegman@hrsa.gov