Evaluation of Hospital Drills: Using the Tool

1. Evaluation of Hospital Drills: Using the Tool Amy Kaji, MD, MPH November 16th, 2005 Acute Care College Medical Student Seminar

2. Hospital Disaster Drills • Why are drills necessary? • Hospitals will be called upon to provide care to the ill, injured, exposed, and concerned • Joint Commission on Accreditation of Healthcare Organizations (JCAHO) Requirement • May help train employees • Tests aspects of hospital response

3. Hospital Disaster Drills • Types • Computer simulation • Tabletop exercises • Operationalized drills involving specific victim scenarios • Evaluations • Can help maximize the value of the drill • Based on accurate observation • Benefit of standardization

4. Johns Hopkins University Evidence-based Practice Center (JHU EPC) • Developed an evaluation tool for hospital drills • Assess impact of drill as hospital response progresses and develops • Presents topics for evaluation in a systematic manner • Identify strengths and weaknesses in hospital disaster preparedness • Promotes targeted efforts to strengthen preparedness • NOT intended to be used to determine whether a hospital passes or fails in its planning endeavors

5. The Evaluation Modules • Developed by a multi-disciplinary team of experts at JHU EPC • Based upon systematic review of published reports on hospital disaster drills and literature relevant to bioterrorism preparedness • Had “expert input” from federal, state, and local agencies • Piloted the modules in two multi-hospital regional disaster drills in the summer of 2003

6. Guiding Principles in Developing Evaluation Modules • Need for observing multiple hospital zones • Evaluation of a disaster drill requires an understanding of drill activities in all areas of the hospital • Four zones identified include: • Incident Command • Decontamination • Triage • Treatment • Addendum if Biological or Radiological Scenario • Decontamination Zone Module needed for radiation & chemical drills; not recommended in biological drill

7. Guiding Principles in Developing the Evaluation Modules • Need for documentation of time points • Recording time points of drill activities is a widely accepted method of evaluation • Labor-intensive if excessive number of time points and may detract from overall evaluation • Limited, specific time points thus predetermined

8. Guiding Principles in Developing the Evaluation Modules • Need for documenting clinical care outcomes • Track the volume of victims in each zone and adequacy of provisions made for them, including space, staff, supplies, etc. • Modules are not intended to collect individual victim level data • Modules monitor the zone and outcome for the zone as a whole, not for each victim

9. Guiding Principles in Developing the Evaluation Modules • Need for debriefing (after-action review) • Obtain feedback from participants, including organizers, staff, and victims • Allows for discussion of issues that span more than one zone • Evaluate and integrate cross-zone issues at a post-drill debriefing session

10. Guiding Principles in Developing the Evaluation Modules • Need for ease and flexibility of care • Designed to be readily understood, easy to use, and applicable to many different drill scenarios • Items on form are arranged by subject, and this is the same across all modules to facilitate analysis • Decision to include a module is dependent upon the specific scenario and drill

11. Guiding Principles in Developing the Evaluation Modules • Need for safety and security • Consider planned drill activities; i.e., use of unfamiliar equipment • Protect actual patients on the premises • Safety of drill victims and healthcare workers • Contingency plan to stop the drill in case of an actual emergency • Consider designating a safety officer to monitor the drill and its participants

12. Recommended Use of Evaluation Modules and Addenda ( X = USE IT!)

13. Internal Structure of Evaluation Modules • Zone forms for Incident Command, Decontamination, Triage, and Treatment have same structure and subject headers: • Time points • Zone description • Personnel • Zone operations • Communications • Information flow • Security • Victim decontamination and tracking • Victim flow • Personal protective equipment (PPE) and Safety • Equipment and Supplies • Rotation of Staff • Zone disruption

14. Internal Structure of Evaluation Modules • Numbering of questions on the forms • “C” denotes questions that are common to several zones • Questions that are zone-specific are identified by a unique letter code • 120 questions per module • Coding on the forms and use of comment boxes • Yes = Y; No = N; Unclear = U; Not applicable = NA • NA is not a substitute for missing information, negative information, or to avoid writing a comment • Color coding for the modules and addenda • Assists in organizing and tracking modules and addenda before and after a drill

15. Common Sample Questions from an Evaluation Module (Time Points) • C1. Time the drill began: ___AM/PM/U • C2. Time the hospital disaster plan was initiated in this zone: ___AM/PM/U/Not initiated • C3. Time this zone was ready to accept victims: ___AM/PM/U • C4. Time when this zone was notified that incident command was operational:___AM/PM/U/Not notified • C5. Time the drill ended in this zone: ___AM/PM/U

16. Common Sample Questions from an Evaluation Module (Zone Description) • C8. Was the boundary for this zone defined? Y/N/U • C9. If this zone had a defined boundary, how was it defined? (Check all that apply) • Barricade • Security personnel • Sign • Tape • Vehicle • Wall (permanent or temporary) • No boundary • Other (specify): __________________

17. Common Sample Questions from an Evaluation Module (Personnel) • C11. Did someone take charge of this zone? Y / N / U    • C12. If someone took charge of this zone, how many minutes after the drill activities in this zone began did this person take charge? (Check one)   • O < 10 min • O 10 - 29 min • O 30 - 59 min  • O 1 - 2 hrs • O > 2 hrs • O NA    • C13. If someone took charge of this zone, was it the officially designated person? • Y / N / U / NA

18. Common Sample Question from an Evaluation Module (Personnel) • C14. How was the person in charge of the zone identified? (Check all that apply) • a. [  ] Arm band • b. [  ] Hat • c. [  ] Name tag  • d. [  ] Verbal statement • e. [  ] Vest • f.  [  ] Not identified  • g. [  ] Other physical identification (specify): __________________________________________

19. Common Sample Questions from Evaluation Module (Zone Operations) • C29. Was the space allocated for the zone adequate? Y/N/U • C30. If not enough space for the zone, where did zone activities overflow to? (Check all that apply) • Adequate space allotted • Conference room • Hallways • Outside hospital • Treatment/victim care areas • Waiting rooms • No overflow • NA • Other (specify): ______________________________ • C31. Was this zone used for the same functions during non-drill operations? Y/N/U

20. Common Sample Questions from Evaluation Module (Zone Operations) • C33. Did clinical staff interact directly with families of victims? Y/N/U/NA • C34. Were families of victims referred to specially designated staff? Y/N/U/NA • C35. How was victims’ privacy ensured? (Check all that apply) • Curtains • Individual areas • Privacy screens • Not ensured • Other (specify): ____________________________

21. Common Sample Questions from Evaluation Module (Communications) • Communication device (s): If device not present , circle “N” in column “a” and go to the next line. • A. Was device present? Y/N/U • B. If present, # available: • C. If present, was it used in drill? Y/N/U • D. Comments (note problems) • C36. 2 way radio/phones • C37. Direct line • C42. Numeric paging • C44. Text paging • C45. E-mail and internet access • C47. Intercom • C49. Runner

22. Common Sample Questions from Evaluation Module (Information Flow) • C54. How was this zone notified of the event? (Check all that apply) • FAX • Runner • Telephone • Not notified • Other (Specify):__________ • C55. Who notified this zone of the event? • Drill organizer • Incident command center • Media • Other hospital staff • Outside source • Victims arriving • Not notified • Other (specify):____________

23. Common Sample Questions from Evaluation Module (Security) • C60. Were security present in this zone? Y/N/U • C62. If security personnel were present, what type of security? (Check all that apply and provide approximate numbers) • FBI ___ • Hospital Security ___ • Local Police ___ • State Police ___ • NA • Other (specify): ______________

24. Common Sample Questions from Evaluation Module (Victim Documentation and Tracking) • C74. Were all incoming victims registered and given a unique identification or medical record number? (check one) • Yes, before entering this zone • Yes, on entering this zone • No, not while in this zone • Unclear • C76. Was a central list of victims generated for this zone? Y/N/U • C77. Were the triage markers on the victims clearly visible? Y/N/U

25. Common Sample Questions from Evaluation Module (Victim Flow) • C81. Did a bottleneck develop in this zone? Y/N/U • C84. Were the paths leading to the next zone marked? Y/N/U • C86. Were the lowest acuity victims directed by staff to an area separate from higher acuity victims? Y/N/U • C87. What proportion of victims had treatment delayed because of zone staffing shortage? (Check one) • None • Less than half • At least half (but not all) • All • Unclear

26. Common Sample Questions from Evaluation Module (Personal Protective Equipment and Safety) • If needed, were these items for standard precautions available for the healthcare workers? Y/N/U • Used by staff? Y/N/U • Adequate supply? Y/N/U • Eye protection • Waterproof gowns • Isolation gowns • Gloves

27. Common Sample Questions from Evaluation Module (Equipment and supplies) TX15. Were medications needed for the treatment of victims available within the hospital? Y/N/U/NA TX16. Were medications requested from and outside source? Y/N/U Were needed medical supplies available? TX19. Bandages Y/N/U/NA TX20. Basic airway equipment Y/N/U/NA TX22. Blood drawing supplies Y/N/U/NA TX23. Burn Packs Y/N/U/NA TX24. Cleaning supplies for contaminated equipment Y/N/U/NA TX25. Crash carts Y/N/U/NA TX26. Intravenous fluids Y/N/U/NA

28. Common Sample Questions from Evaluation Module (Rotation of staff) • C98. Was there a staff rotation /shift change? Y/N/U • C99. If there was a staff rotation, did the officially designated person in charge of the zone change? Y/N/U/NA • C101. What method of shift changing was used? (Check one) • Group shift change • Staggered shift change • NA • Other (specify): _____________ • C102. How were incoming staff updated? (Check all that apply) • Group briefing • Individual briefing • Written notes • Not updated • NA • Other (Specify):_____________

29. Common Sample Questions from Evaluation Module (Zone disruption) • C103. Was there a plan to relocate this zone if necessary? Y/N/U • C104. Did this zone close at any time during the drill? Y/N/U • If no, disregard the remainder of this section. STOP. This zone module is complete… • C105. If the zone closed during the drill, what was the reason for closing? (Check all that apply) • Contamination • Other safety concerns • Space • Other (Specify): ________________

30. Description of Modules and Objectives • Pre-drill Module • Incident Command Center Zone Module • Decontamination Zone Module • Triage Zone Module • Treatment Zone Module • Group Debriefing Module

31. Pre-drill Module • Should be used in all disaster drills during the planning stages • Form is designed to collect the following: • Goals and objectives for the scope of evaluation • Background information • Information on areas that hospital wishes to evaluate • Resources required • If multi-hospital or regional drill, each site must work closely with overall coordinators

32. Incident Command Center Zone Module • Designed to reliably collect information about operations of the incident command system (ICS) • Should be used in all disaster drills when evaluating the ICS • Form is designed to assess the following: • Command structure in the zone • Adequacy of staffing in the ICS • Communication and information flow from hospital areas to the ICS • Communication with outside agencies • Adequacy of security, safety provisions, and physical space

33. Decontamination Zone Module • Designed to collect information re: functioning of decontamination area • Used if scenario involves radiation or chemical exposure and decontamination is needed • Form assesses the following: • Command structure in the zone • Communication and information flow in the zone • Victim and staff safety in the zone • Adequacy of staffing and physical space in the zone • Appropriateness of equipment and PPE • Victim flow in the zone

34. Triage Zone Module • Designed to collect information re: functioning of treatment areas • Should be used whenever drill objectives include evaluation of patient care activities beyond triage • Appropriate for use in emergency department-based treatment areas or in other clinical areas

35. Triage Zone Module • Form designed to assess the following: • Command structure in the zone • Communication and information flow in the zone • Victim and staff safety in the zone • Relation of physical characteristics of zone to treatment activities • Efficacy of treatment operations • Adequacy of materials and supplies in the zone • Victim flow in the zone

36. Debriefing • Debriefing is integral part of drill process • Main objective of debriefing is to identify issues not captured by evaluation modules • Facilitators should create an open, non-judgmental atmosphere • Should occur in all drills to obtain feedback from participants and observers

37. Debriefing • Different approaches to debriefing exist • One method: conduct a session with all participants and observers present and ask a series of general questions about the drill • Another method: conduct a group debriefing session with the participants from that zone

38. Group Debriefing Module • Contains open-ended questions designed to facilitate discussion after completion of drill • Questions may be added or deleted • Designed to cover all issues, including incident command structure, communications, security, decontamination, triage, and treatment • Documenting the debriefing • A scribe should be assigned to record • Videotaping / audiotaping debriefing may help capture comments but should not hinder open exchange

39. Group Debriefing Module Questions • Did you feel you were notified of the disaster in a timely fashion? • Did the incident command center work effectively? • Did anyone receive incorrect information from the incident command center? If not correct, what specifics do you recall about incorrect information? • Was the information from the incident command center received by other zones in a timely way?

40. Group Debriefing Module Questions • Were there problems with information flow within the hospital? • Were memorandums of understanding with outside agencies (e.g., police) activated? • Did nurses and physicians respond quickly to the disaster call? • Was the zone set up when the first mock victim arrived? • Was security in place before the first mock victim arrived?

41. Group Debriefing Module Questions • Did people have a good understanding of their roles, as defined in the disaster plan? • Did the decontamination system work effectively? • Did you have any problems with the decontamination equipment? Functioning properly? Adequate number of units? Participants used correctly? • Were there delays in decontamination? If so, what triggered those delays?

42. Description of Addenda • Four addenda are part of the hospital disaster drill evaluation: • Biological Incident Addendum • Radiation Incident Addendum • General observation and documentation addendum • Victim tracking addendum. • Used to supplement the zone forms • Example: for a radiation exposure drill, the Radiation Incident Addendum is added to Incident Command Center, Decontamination, Triage, and Treatment Zone Modules

43. Biological Incident Addendum • Designed to collect additional information that address response to a biological incident • Should be added to end of each Incident Command, Triage, and Treatment Zone modules • Should be used in all drills that address a biological incident

44. Biological Incident Addendum • Form is designed to assess the following: • Awareness that biological agent cause of illness • Whether appropriate personnel were contacted • Whether health and safety needs of staff were met • Whether health and safety needs of existing patients were met • Whether health and safety needs of victims were met • Availability of special medications and supplies

45. Biological Incident Addendum • BA11. Was the Center for Disease Control and Prevention (CDC) informed that a biological illness was present? • BA14. Was occupational health informed that a biological illness was present? • BA22. If the cause of the illness was not known prior to victim arrival, how long after the first victim arrived was the cause of the illness identified? (Check one) • <1 hour • 1-4 hours • 5-8 hours • >8 hours • Never identified • NA

46. Biological Incident Addendum • BA24. What resources were used to make the diagnosis? (Check all that apply) • Consultation with an in-hospital expert • Consultation with an expert from another hospital • Consultation with an expert from local health department • Consultation with an expert from state health department • Consultation with the CDC • History and physical exam by the treating physician • Microbiological data • Radiological data • NA • Other (specify): ______________

47. Biological Incident Addendum • BA26. Was isolation required for the suspected biological illness involved? Y/N/U (Isolation is required for smallpox, plague, viral hemorrhagic fever, certain pneumonias or rashes, and other symptoms suggestive of a contagious infectious outbreak) • BA27. If isolation was required, were victims transported into an isolation room? Y/N/U • BA28. If they were transported to an isolation room, was the room under negative pressure? Y/N/U/NA

48. Biological Incident Addendum • BA29. Were there enough isolation rooms? • BA30. If not enough isolation rooms, how were victims isolated? (Check all that apply) • Conversion of other rooms/area (specify):___________ • Existing isolation room in other area (specify):__________ • Overflow victims not isolated • Victims with the same suspected biological illness placed in the same isolation room • NA • Other

49. Radiation Incident Addendum • Designed to gather information in response to radiation-related incident • Should be added to end of each zone module • Should be used in all drills that address radiation exposure

50. Radiation Incident Addendum • Form is designed to assess the following: • Awareness that radiation was cause of illness • Whether appropriate monitoring personnel were contacted • Whether health and safety needs of staff were met • Whether health and safety needs of existing patients were met • Whether health and safety needs of victims were met • Availability of special supplies