1 / 27

An Assessment of DMSS Immunization Records 1998-2004

An Assessment of DMSS Immunization Records 1998-2004. National Immunization Conference 2006 Daniel Payne, PhD, MSPH March 7, 2006. Objective of this data assessment. To estimate the agreement between chart abstracted AVA vaccination data and electronic DMSS vaccination data from 1998-2004.

base
Télécharger la présentation

An Assessment of DMSS Immunization Records 1998-2004

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Assessment of DMSS Immunization Records1998-2004 National Immunization Conference 2006 Daniel Payne, PhD, MSPH March 7, 2006

  2. Objective of this data assessment • To estimate the agreement between chart abstracted AVA vaccination data and electronic DMSS vaccination data from 1998-2004

  3. Why is this data assessment important? • “Analysis of DMSS data should be the primary approach for investigation of possible AVA-related health effects of medical significance that occur within the typical period of active duty following vaccination…”. Institute of Medicine. 2003. An Assessment of the CDC Anthrax Vaccine Safety and Efficacy Research Program. Washington, DC: National Academy Press.

  4. Data Assessment Project Timeline March 7, 2006Results presented Jan 19, 2006Preliminary analysis complete April-Dec 2005Medical charts abstracted Spring 2005Abstractors trained, data collection tools created and tested Jan 4, 2006CDC receives final abstracted dataset Jan-Apr 2005DoD service branch approvals obtained Dec 16, 2004QA Project proposed to TMA, SAP 2004 2005 2006 MAR JUN SEP MAR JUN SEP MAR

  5. Sampling Strategy • Representative sample of military treatment facilities (MTF) serving small, medium and large populations across Army, Marine Corps, and Navy service branches • Stratified random sample of MTF’s derived from PRISM data • Sampled 28 of the 146 MTF’s for these service branches (19%) • Geographical considerations: continental U.S.

  6. Geographical distribution of 28 facilities sampled Courtesy of Google Maps

  7. Abstraction Strategy • Records Coordinator at each MTF contacted and requested to pull pre-defined number of records • 2 abstractors reviewed each medical chart independently, with adjudication by a 3rd abstractor • Medical charts were reviewed to collect information on vaccination type and the date(s) of vaccination. • Data entered into a pre-tested PC-based abstraction instrument • SSN was collected to link an individual’s medical chart information to the electronic vaccination data in DMSS

  8. 4 Measures of Assessment Abstracted Records DMSS • Sensitivity = A / A+C • Specificity = D / B+D • PPV = Likelihood of A / Likelihood of A + B • NPV = Likelihood of D / Likelihood of C + D

  9. Methods of assessing AVA data in DMSS 1. Accuracy of DMSS data for personnel having one or more abstracted AVA vaccination(s) (person-level analysis) 2. Concordance of AVA vaccinations recorded in DMSS within different intervals of time, by branch and year (vaccine-level analysis) 3. Regression modeling analysis

  10. Chart Abstraction Results Population of service members 4,201 medical charts sampled from 28 MTF’s Less 27 with unmatched SSN’s (0.6%) 4,174 Approximately 45% had AVA vaccination(s) in medical charts 1,866 medical charts with AVA 1,842 Less 24 records with their only AVA before Jan 1, 1998 1,817 medical charts FINAL SAMPLE Less 25 records with inaccurate service branch-level match

  11. Demographics

  12. Demographics

  13. 1: Accuracy of DMSS AVA Data- Full Sample - Preliminary Results -- Do Not Distribute

  14. 1: Accuracy of DMSS AVA Data by Branch Preliminary Results - Do Not Distribute

  15. 2: Concordance within different reporting intervals Preliminary Results -- Do Not Distribute

  16. Preliminary Results -- Do Not Distribute

  17. Preliminary Results -- Do Not Distribute

  18. 3: Regression model results Preliminary Results -- Do Not Distribute

  19. 3: Regression model results Preliminary Results -- Do Not Distribute

  20. Limitations • Unforeseen circumstances were encountered at four facilities requiring adjustments/replacements to the original site list: • hurricane • construction at records facility • no active duty medical records • only dependent records • Medical records may not always be a gold standard: • Hand-written information in medical record occasionally illegible by all 3 abstractors • Some standard vaccination data collection forms in medical record were found to truncate or omit some data • Record coordinator indicated that, infrequently, some records may be collected electronically but not recorded on the medical chart, esp. during deployment preparation • Air Force medical records not able to be sampled

  21. Strengths • Robust sample • Considered generalizable to Army, Marine Corps, and Navy for 1998-2004 • Several analytical methods used • Effort to compare results to previous studies

  22. Conclusions • Compared with Honner W, et al.* (convenience sample of childbearing women in military), we found: • a higher sensitivity (93.4 vs. 61.5) • a lower specificity (89.5 vs. 97.5) • a higher PPV (88.9 vs. 76.6) • a similar NPV (93.8 vs. 95.0) • fewer differences between service branches (* estimates above are for same person-level methods used by Honner, et al.)

  23. Conclusions • Compared with Mullooly J, et al. ** (Vaccine Safety Datalink childhood vaccines), we found: • Service branch AVA sensitivities (range = 80.7 - 87.3) were close to those found in the 3 VSD databases for childhood vaccines (range= 82 - 98) • VSD noted that, « relaxing the vaccination date agreement to 7 days increased [the relative sensitivity] by 1-5 percent . » We found this increased μ=6.0% (** estimates above are for same vaccine-level methods used by Mullooly, et al.)

  24. Conclusions • Measures of agreement varied by service branch: • Person-level analysis: Army had highest sensitivity, Navy had best specificity and PPV • Vaccine-level analysis: AVA recorded on exact date was highest in Marine Corps, but agreement within +/- 7 days showed Army highest • Regression analysis: only statistically significant difference by branch was for sensitivity (exact match by date): Marines had 59% higher agreement than Navy • No statistically significant differences in agreement were detectable by year (1998-2004) • DMSS holds comparable estimates of vaccine quality compared with other vaccine safety databases and studies

  25. Acknowledgements Yujia Zhang, PhD Susan Duderstadt, MD, MPH Charles Rose, Jr., PhD Michael McNeil, MD, MPH Emily Weston, MPH Constella Inc.: Steve Wilkins, Tim Struttman DoD: Army Medical Surveillance Activity, MILVAX, Tricare Management Activity FDA: Center for Biologics Evaluation and Research National Vaccine Advisory Committee workgroup

  26. 7 Safeguards to assure data quality, consistency, and confidentiality REVIEW TRANSMISSION ABSTRACTION 1 Built-in abstraction instrument validity checks 2 Training program for abstractors 3 100% double-blind data entry 4 On-site 3rd person adjudication for data entry discrepancies Data Quality 5 Subset independently re-abstracted and re-adjudicated: 100% correct 6 Secure transfer of encrypted data for off-site storage 7 On-site informatics review

More Related