Testing and Measuring Changes Review of QI 103: Testing and Measuring Changes with PDSA Cycles
Lecture Objectives • Describe how to establish and track measures of improvement during the “plan” and “do” phase of PDSA. • Explain how to learn from data during the “study” phase of PDSA. • Explain how to increase the size and scope of subsequent test cycles based on what you’re learning during the “act” phase of PDSA.
Lecture Outline • Planning for data collection • Operational definitions • Key questions • Testing changes and collecting data • Sampling • Studying data for improvement • Run charts • Building your degree of belief over time • Linking tests of change
Defining Measures: Three Types Use unambiguous operational definitions
Defining Measures: Two Levels • Level 1: Project-level measures • Representing overall goals and assumptions • Documented on the Project Charter • Level 2: PDSA-level measures • Representing the specific changes you are testing • Documented on the PDSA worksheet
Identify the type and level of each measure in this QI project: • Rate of occurrence of methicillin-resistant Staphylococcus aureus (MRSA) per 1,000 patient days • Percent of patient encounters in compliance with hand hygiene procedure • Number of hand sanitizer stations available
Identify the type and level of each measure in this QI project: • Your overall goal is to reduce rates of MRSA infection (project-level outcome measure) • Your change is to improve hand hygiene (project-level process measure and PDSA-level outcome measure) • Your first test is to try to improve hand hygiene by increasing the availability of hand sanitizer (PDSA-levelprocess measure)
How would you measure this outcome? • Access to health care
How would you measure this outcome? • There are many waysto measure “access” • Number of days to third next available appointment • Number of minutes from time of appointment to time to see clinician • Percent of “good” or “very good” answers on relevant patient satisfaction survey questions • Average daily clinician hours available for appointments
Which of these is a complete operational definition? • Rate of occurrence of methicillin-resistant Staphylococcus aureus (MRSA) per 1,000 patient days • Percent of patient encounters in compliance with hand hygiene procedure • Number of hand sanitizer stations available
Which of these is a complete operational definition? • Rate of occurrence of methicillin-resistant Staphylococcus aureus (MRSA) per 1,000 patient days • How do you define the occurrence of MRSA? What calculation will you use? • Percentage of patient encounters in compliance with hand hygiene procedure • How do you define a “patient encounter”? How will you calculate the percentage? • Number of hand sanitizer stations available • What area are you including in the count of sanitizer stations? How do you define “available” — what if the dispenser is empty?
Key Questions for Measurement • What is all the data you need to collect? (e.g., for a percent, you need both the numerator and the denominator) • Who is responsible for collecting the data? • How often will the data be collected? • How will the data be collected? • Make measurement as simple as possible!
Simplify Through Sampling • Simple random sampling • Proportional stratified random sampling • Judgment sampling
Planning for Data Collection: Review • Establish: What do you want to learn about and improve? • Determine: What measures will be most helpful for this purpose? • Define: For each measure, what is the operational definition? • Designate: who, what, when, where, how
Building Degree of Belief Iterative test cycles; can be concurrent Increase size: 5X rule Broaden scope: Test in many different conditions
Video https://youtu.be/Q4d7T_aBUPo http://www.ihi.org/education/IHIOpenSchool/resources/Pages/Activities/Provost-WhyShouldYouStartTestingChangesASAP.aspx
Discussion • Discuss the risks of starting a test of change earlier versus later. Which scenario do you think carries greater risk? • Provost implies that it’s important to see improvement quickly. Why do you think that is? • What do you think of the provocation “what can you do by next Tuesday?” Is it helpful? • Can you think of something to test by next Tuesday? http://www.ihi.org/education/IHIOpenSchool/resources/Documents/Facilitator_Provost-WhyShouldYouStartTestingChangesASAP.pdf
Exercise • Follow the instructions at http://www.ihi.org/education/IHIOpenSchool/resources/Documents/QI103_exercise.pdf • Form a team with at least one other person and pick a problem at your school or in your community to work on. The problem should lend itself to measurement. • Use the data collection planning checklist to help you create a data collection plan for the improvement project. Write down the answer for each question for a complete family of measures.