Quality Management of Case Surveillance Systems

1. Quality Management of Case Surveillance Systems

2. Session Overview • We will: • Discuss what defines Quality when considering case surveillance • Articulate what value Quality brings to case surveillance • Consider how Quality can be assessed, assured, and improved upon

3. Quality Management of Case Surveillance Systems • Quality Management typically entails:

4. But wait… Why are we talking about Surveillance and M&E? …. aren't they different things?

5. M&E vs. Surveillance M&E Surveillance • Aggregate counts • Measure of criteria • Measure vs. criteria • Shows service/process trends over time • Number of people tested • Number of HIV+ tests • Number of people on ART • Unique, unduplicated counts • Measure of disease burden • Measure of influencing factors • Shows disease-specific trends over time • Number of unique people w/HIV • Distribution of cases • Factors influencing transmission • Number of unique people on ART • Trends in ART access outcomes

6. But wait… Why are we talking about Surveillance and M&E? …. aren't they different things? Yes….. But…… How else will you know if your surveillance system gives you quality data?

7. M&E of Surveillance • Measure of criteria • Measure vs. criteria • Shows service/process trends over time • Do we have a high-quality system? • How is our surveillance system working? • How can we improve the surveillance system? • Do we have high-quality data? • Are the data reliable? • Should we trust our data? • How can we improve our data?

8. What is Quality, with respect to Case Surveillance? • Degree of Excellence of …What?

9. What is Quality, with respect to Case Surveillance? • Degree of Excellence of …What? Yes!

10. Why Monitor and Evaluate a Case Surveillance System? Success + Change Quality + Confidence

11. Why is Quality Important for Case Surveillance? • Being able to speak to the quality of • Collection- the data collected • Collation- the process used to collect, clean and manage the data • Analysis- the trends, and findings presented • Interpretation- the action items suggested • Dissemination- the data given to key partners Will increase your confidence in the data and the process, and that will then better guide how others will use and react to the data. Accuracy of data Accuracy of assumptions Accuracy of action taken

12. When should a quality management process be implemented? • Quality Management is a continuous process • It should be planned early on • It should begin the moment a program is initiated • It should be routine and ongoing • Its findings should be integrated into an adapting and evolving program

13. How to Monitor and Evaluate a Case Surveillance System?

14. How to Monitor and Evaluate a Case Surveillance System

15. A. Define criteria of interest • What is it that you want to know that will instill confidence in your system and your data? • Are all expected cases reported? • Have all expected sites reported? • Are all (key) fields entered on the case report form? • Are all fields entered correctly? • Are the data accurate? • Is reporting timely?

16. A. Define criteria of interestSample quality criteria for case surveillance Influence Data Quality Influenced by Process

17. A. Define criteria of interest Related to the Process Process Monitoring will allow you to understand: • What is working well • What are the gaps in the • Data collection process • Data reporting process • Data cleaning process • Data utilization process • Success or limitation in this area will influence data quality

18. A. Define criteria of interest Related to the Data • This will allow you to understand: • What is working well • What are the gaps in the • Data collection process • Data reporting process • Data cleaning process • Support and technical assistance in this area will allow you to have stronger confidence in using the data

19. B. Set measures by which to assess • How can you measure and perceive change in your defined criteria? • What are your objectives/standards?

20. B. Set measures by which to assessProcess Process Monitoring should be implemented in a routine manner • Is typically observational • Site level • Intermediate level • National level • Criteria/measures will be defined by the Standard Operating Procedures (SOP) • What is the gold standard for data collection and transmission? • What is actually happening?

21. B. Set measures by which to assessProcess • How are patient level data gathered? • Self report? Proxy? Any verification? • Ideally, patient-level data should be: • Self reported (not assumed or by proxy) • Validated (with identification, with questions for clarification) • Inaccurate data, pseudonyms render case matching difficult. • How are case data transcribed? • How many people touch each case? How many times are data copied? Any double entry or validation? • Ideally, data should be: • Copied or transcribed minimally (source is best!) • Validated if copied or transcribed • Errors of transcription create inaccurate data.

22. B. Set measures by which to assessProcess • How are cases reported? • Is responsibility clear? Are timelines clear? Is there supervision? Any validation? • Ideally, there should be: • Clearly defined roles and responsibilities • Clear timelines • Routine supervision that includes validation • If cases are not reported in a routine and timely way, data will be incomplete. • How are data managed? • Are data cleaned? Are cases matched or de-duplicated? At what level does this occur? How many people are responsible for this? • Ideally, data should be regularly: • Cleaned and deduplicate by few, but consistent people • Followed-up upon if there are questions • If cleaning is not routine, and feedback given, data will continue to be poor.

23. B. Set measures by which to assessData - Representativeness • Representativeness describes your ability to accurately describe the disease over time and by population and place • Are all expected cases a part of the national data set? • If 100 HIV+ tests were performed at a site, 100 HIV case report forms should be completed and submitted • Are all sites, types of service, regions represented? • If there are 100 testing/treatment sites, are all 100 reporting cases on a regular basis? • Examples: • 90% of cases from the last quarter were reported • 95% of eligible sites reported in the last quarter

24. B. Set measures by which to assessData - Representativeness A) Calculating Representativeness of reported cases • Are all cases that you expect to be in the system in the system? • Overall • By Region, Site, Network, etc. # of reported cases (in time period) # of expected cases (in time period) X 100 # of reported cases (in time period) # of expected cases (in time period) X 100 This could also be called system completeness

25. B. Set measures by which to assessData - Representativeness B) Calculating Representativeness of sites reporting cases • Are all sites that you expect to report to the system reporting? • Overall • By Region, Site, Network, etc. # of sites reporting cases # of expected sites X 100 # of sites reporting cases # of expected sites X 100

26. B. Set measures by which to assess Data - Completeness • Completeness describes what proportion of an expected count is ‘there’. • Are all expected data fields are complete? • What proportion of fields on the case report form are complete? • What proportion of cases are reported with the (X) case-defining variables complete? • Examples: • 95% of reported cases have all case defining variables complete • 95% of reported cases have Date of HIV Diagnosis complete

27. B. Set measures by which to assessData - Completeness

28. B. Set measures by which to assessData - Timeliness • Timeliness is a measure of the speed between steps in the process - time from diagnosis to reporting • Are the cases submitted and processed within a reasonable time? • Are you presenting this year’s data, or five years ago? • Example: • 85% of cases are reported within three months of diagnosis

29. B. Set measures by which to assessData - Timeliness • Can be measured two ways A) Median time between diagnosis and receipt of case report at MOH (or other authority) • Calculated by subtracting the months between the date of report and the date of diagnosis. • Reporting delay values can be summed and the median calculated B) Proportion of cases reported in a specified time period (some standards use 6 and 12 months ) • Number of cases diagnosed within a year and reported within 6 months of diagnosis • Number of cases diagnosed and reported for that diagnosis year • Before measuring timeliness, determine representativeness of reporting • Incomplete reporting will overestimate timeliness

30. B. Set measures by which to assessData - Accuracy • Accuracy describes the degree to which your data are ‘true’. • Your measure of accuracy will help you establish a degree of confidence (or not!) in your data • Data Accuracy is influenced by: • Collection of data from patient to forms • Does the patient tell you the correct data? • Does the counselor record the correct data with correct spelling? • Does the counselor ask all questions (rather than guessing)? • To assess the accuracy of data reported or collected from the patient, you will need to use process/observational methods

31. B. Set measures by which to assessData - Validity • Validity describes the degree to which your data are ‘true’. • Your measure of validity will help you establish a degree of confidence (or not!) in your data • Are males pregnant? • Are people 120 years old? • Were people diagnosed with HIV in 1958? • Do people have negative CD4 values? • Initial data checks can be done on data sets to look for anomalies • Data Validity is influenced by the: • Transcription of data from one form to another • Spelling, number sequence, correct code/box • Paper to paper and/or paper to computer • Example: • 95% of reported cases have a valid Date of Birth reported

32. B. Set measures by which to assessData - Validity

33. C. Gather Data • Data collection should be regular, routine, and ‘low barrier’ • Create templates and checklists • Train and support site-level staff to do routine monitoring and reporting • Use regional/national staff to do mentoring and supportive supervision of the process, and periodic monitoring • Start soon and start small • Pilot your M&E process and measures • See if you are able to find the answers you are looking for • Modify as needed

34. C. Gather Data - Design a Monitoring and Evaluation Process • Define the Details • How will the M&E Process be implemented? • This is your M&E Plan • Brief description of the project and the evaluation framework • Detailed description of the indicators • Data collection plan • Description of the data sources • Description of data collection tools • Plan for how monitoring and evaluation will occur • Dissemination and utilization plan for information gathered • Description of M&E plan update process SOP

35. C. Gather Data Local Monitoring Methods • Observation/supervision • Data validation • Selection of 5% of records • Can use re-abstraction methods • Data double entry • Data triangulation • Registers + Case Report + Patient Record + Lab Record

36. C. Gather Data Regional/National Monitoring Methods • Site visits/observation/supervision • Assistance with data validation • Support for data triangulation • National M&E Reports + Case Report Data + Other Aggregate Counts • Feedback!!!

37. D. Summarize findings • With respect to your defined criteria and measures • Use simple tools and outlines/templates to do this • Do this initially to document a baseline so you can measure/assess improvement • Do this frequently to ‘test your hypotheses’ (step E. Act on the Findings) and see if a change is worth it • Share your findings with all involved staff at all levels • This can motivate them • This can validate their effort

38. D. Summarize findings Use Data Locally; Regionally • Share findings and empower a clinical site to: • Improve the quality of surveillance activities • Quality and timeliness of data and reporting • Quality of patient interactions to capture complete data • Analyze and use surveillance data to improve clinical outcomes • Look at the clinical indicators collected; is this aligned with standards of care? • Better understand their local epidemic • Who needs to be reached? • Do programs need to be altered to do so?

39. D. Summarize findings Use Data Regionally; Nationally • Share findings and empower national/regional programs to: • Propose and/or implement national program changes • More/different investment in surveillance • More/different investment in HIV program activities • Clinical outcomes • Prevention for high-risk populations • Referral programs • Propose and/or implement national policy changes • Case reporting mandate • Revised clinical guidelines • Anti-discrimination policies

40. E. Act on the findingsExamples of Quality Improvement

41. E. Act on the findings Simple Quality Improvement process Identify Issue/Area for Improvement Measure Performance Brainstorm Cause of Issue

42. E. Act on the findings Step 1: Identify Issue/Area for Improvement • What is the issue that concerns you? • Break it down into small components • Which components can be modified? • Select one component and plan/implement a test of change

43. Example Not all sites are reporting cases Grand Anse is the most challenged

44. E. Act on the findings Step 2: Measure performance • Define and Measure • Who • What • Where • When • How • Set an objective for what you would like to see

45. Example Only 38% of sites in Grand Anse have reported an expected HIV case to HASS in the last year. Aim for 50% in one month; 75% in six months

46. E. Act on the findings Step 3: Brainstorm Cause of Issue • Gather information • Go meet with local staff • Understand the issue • Collectively come up with strategies for improvement • Enable the local staff to come up with solutions • Before you act… Consider what is actually in your control. It is likely more effective to focus on those!

47. Example • Issues • High turnover of nurses • Interruptions in supply of test kits (tests not done) • Case reports out of stock • Power outages • Staff have additional duties • Data collection incomplete • Unclear who is responsible • Possible Solutions • Cross-train staff • Increase staff mentorship • Define/assign roles and responsibilities • Create inventory checklist, implement use, and communication of needs • Procure battery back-up

48. Plan, Do, Study, Act Circle

49. E. Act on the findings Step 4: Generate and Plan Improvement Ideas • Select ONE area and ONE element • Brainstorm: • What is one thing we can do on one day to see improvement? • Who do we need to train/support to make this one change? • How will we measure if this creates an improvement? • Hints: • Start small • Start local • Start quickly

50. E. Act on the findings Step 5: Implement Change • For the ONE area and ONE element • Support Implementation: • Take a baseline measure • Train/orient the key staff • Implement the change for one hour/one day • Measure the result • Hints: • Start quickly • Don’t over think it