Deciding Whether To Complement a Systematic Review of Medical Tests With Decision Modeling

1. Deciding Whether To Complement a Systematic Review of Medical TestsWith Decision Modeling Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide www.ahrq.gov

2. Overview of a Medical Test Review • Analyze and Synthesize Studies • Assess Risk of Bias as a Domain of Quality • Assess Applicability • Grade the Body of Evidence • Meta-analysis of Test Performance Evidence With a “Gold Standard” — or — • Meta-analysis of Test Performance Evidence With an Imperfect Reference Standard • Decision Modeling Extract Data From Studies • Prepare Topic • Develop the Topic and Structure the Review • Choose the Important Outcomes • Search for and Select Studies for Inclusion • Search for Studies Research Sources Report Medical Test Review Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

3. Learning Objectives • Describe the utility of using decision modeling when performing systematic reviews of medical tests • Determine the appropriateness of using decision modeling when performing a systematic review Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

4. Background (1 of 2) • Most systematic reviews focus on test performancedue to limitations of what is reported in the literature. • Test performance is not sufficient to assess usefulness. • Links between testing, test results, and patient outcomes are complex. • Even with high test accuracy, doctors may not act according to the results, patients may not follow orders, and interventions may not yield a benefit. • Studies comparing test-and-treat strategies are ideal but rare. • Evidence often needs to be assembled from different studies. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

5. Background (2 of 2) • Modeling (either decision or economic analysis) can: • Link evidence from different sources • Explore the impact of uncertainty • Make assumptions clear • Evaluate tradeoffs in benefits, harms, and costs • Compare multiple test-and-treat strategies that have never been compared head to head • Explore hypothetical scenarios • Therefore, modeling can link testing to patient outcomes and aid in understanding and interpreting results of systematic reviews of medical tests. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

6. Challenges to Developing Decision Models • Developing a formal decision model is not always feasible due to: • Short time spans • Limited resources • Modeling requires: • Technical expertise • A good appreciation of the clinical issues • Resources, sometimes extensive • For these reasons, modeling should only be undertaken when informative and worthwhile. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

7. Decision Modeling: A Five-Step Approach • A five-step approach is used to determine whether modeling is informative and worthwhile within a systematic review. • Define how the test will be used. • Use a framework to identify test consequences and management strategies for each test result. • Assess if modeling is useful. • Evaluate previous modeling studies. • Consider whether modeling is practically feasible in the time frame given to create the review. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

8. Decision Modeling Step 1:Define How the Test Will Be Used • The PICOTS (population, intervention, comparator, outcomes, timing, and setting) typology establishes the context. It also: • Clarifies the setting of interest • Screening, diagnosis, treatment guidance, patient monitoring, or prognosis • Clarifies the intended role of the test • As the only test, as an add-on to previous tests, or as triage for further diagnostic workup • Is crucial for planning a meaningful decision analysis Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Samson D, Schoelles K. Developing the topic and structuring systematic reviews of medical tests: utility of PICOTS, analytic frameworks, decision trees, and other frameworks. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

9. Decision Modeling Step 2: Use a Framework To Identify Consequences and Management Strategies (1 of 2) • Medical test results are indirect. • Accurate diagnosis or changes in test performance do not change patient-relevant outcomes. • Instead, they influence downstream clinical decisions that eventually determine outcomes. • Test performance (e.g., sensitivity, specificity) is thus a surrogate end point. • Example: HIV testing • Direct effects: Emotional distress from the testing process, cognitive/emotional benefits of knowing carrier status, et cetera • Indirect effects: All downstream effects of treatment guided by test results, for example, the benefits/harms of treatment for true positive diagnoses Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

10. Decision Modeling Step 2: Use a Framework To Identify Consequences and Management Strategies (2 of 2) • Identifying the consequences of testing and its results is essential for interpreting summary testing performance (e.g., sensitivity, specificity). • The analytic framework used in the systematic review is a good starting point. • Is the basis for the decision tree showing test consequences and management options that depend on test results • Helps reviewers define clinical scenarios of interest, alternate strategies, and assumptions made by reviewers regarding test-and-treat strategies Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Samson D, Schoelles K. Developing the topic and structuring systematic reviews of medical tests: utility of PICOTS, analytic frameworks, decision trees, and other frameworks. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

11. Decision Modeling Step 3:Assess Whether Modeling Is Useful (1 of 5) • In most cases, decision modeling is useful when evaluating medical testing because of: • The indirect links between testing and health outcomes • The multitude of test-and-treat strategies that can be contrasted • Modeling is not useful when: • One test is a “clear winner.” • There is a very scarce amount of information. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

12. Decision Modeling Step 3:Assess Whether Modeling Is Useful (2 of 5) • There are two scenarios in which one test-and-treat strategy can be a “clear winner” over others. • Scenario A: The clear winner is indicated by direct comparative evidence that: • Evaluates all important test-and-treat strategies • Is obtained from well-run randomized trials or nonrandomized studies • Is applicable to both the clinical context and patient population of interest • Shows one dominant strategy with respect to both benefits and harms, with adequate power Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

13. Decision Modeling Step 3:Assess Whether Modeling Is Useful (3 of 5) • Scenario B: One test-and-treat strategy can be a clear winner as demonstrated by test accuracy alone if: • Patients have the same response to downstream treatments for all tests • The clear winner is preferable in all of the following categories: • Cost and safety • Sensitivity — correctly identifying patients with disease • Specificity — correctly identifying those without the disease Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

14. Decision Modeling Step 3:Assess Whether Modeling Is Useful (4 of 5) • Note: How to tell if patient groups should have the same response to treatment: • Randomized trials may suggest the same response. • Inference between tests • If the sensitivities of two tests are very similar, it is reasonable to expect that patients selected for treatment to be similar and thus respond to treatment similarly. • Extrapolation between tests • Tests operate on same principle, so the clinical/biological characteristics of the additional identified cases are expected to be the same. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

15. Decision Modeling Step 3:Assess Whether Modeling Is Useful (5 of 5) 2. The second case for not undertaking decision modeling is when information is very scarce. • There is not enough information regarding: • Which modeling assumptions are reasonable • Downstream effects of testing • Plausible values of multiple influential parameters • We do not understand the underlying disease processes well enough to credibly predict outcomes. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

16. Decision Modeling Step 4:Evaluate Previous Modeling Studies (1 of 3) • There are three steps to follow when evaluating previous modeling studies: • Judge the quality of existing models. • This step can be challenging when descriptions are unclear and necessary quantities are unknown. • If possible, validate the model against independent datasets that are comparable to the datasets used to develop the model. • External validation is important when unknown quantities have been assumed. • External validation can be difficult to accomplish. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

17. Decision Modeling Step 4:Evaluate Previous Modeling Studies (2 of 3) • Assess the applicability of models to the interventions and populations of interest (i.e., match the PICOTS [population, intervention, comparator, outcomes, timing, and setting]). • This assessment Includes consideration of whether methodological and epidemiological challenges are adequately addressed. • Explore the applicability of underlying parameters. • Previous models may use different estimates of diagnostic accuracy. • Knowledge of the natural history of the disease may have changed. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

18. Decision Modeling Step 4:Evaluate Previous Modeling Studies (3 of 3) • If existing models do not meet the needed criteria: • Developing a new model may be considered • May also consider working with the developers of existing models to address the key questions of interest Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

19. Decision Modeling Step 5: Consider Whether Modeling Is Practically Feasible • Feasibility considerations: • Time • Budget • Available personnel • Accessibility of pre-existing models • Modification needs for pre-existing models • Amount of out-of-scope literature required to develop/adapt a model • If a model is not currently feasible but would be useful, it may be done later as a secondary project. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

20. Illustrative Example: Positron Emission Tomography in Treating Alzheimer’s Disease (1 of 8) • Systematic review focus: the ability of positron emission tomography (PET) to guide the management of suspected Alzheimer’s disease (AD) in three different patient scenarios: • Scenario A: In patients with dementia, can PET be used to determine the type of dementia that would facilitate early treatment of AD and perhaps other dementia subtypes? • Scenario B: For patients with mild cognitive impairment, could PET be used to identify a group of patients with a high probability of AD so that they could start early treatment? • Scenario C: Is the available evidence enough to justify the use of PET to identify a group of patients with a family history of AD so that they could start early treatment? Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Matchar DB, Kulasingam SL, McCrory DC, et al. Use of Positron Emission Tomography and Other Neuroimaging Techniques in the Diagnosis and Management of Alzheimer’s Disease and Dementia. Available at www.cms.gov/deteminationprocess/download/id0TZ.pdf.

21. Illustrative Example: Positron Emission Tomography in Treating Alzheimer’s Disease (2 of 8) • Step 1: Define how positron emission tomography (PET) will be used • Focus is on the diagnosis of Alzheimer’s disease (AD) among the three scenarios • Only interested in PET as an “add-on” test to diagnose different severities or types of AD Cross-tabulation of PET Results and Actual Clinical Status Among Patients With an Initial Clinical Examination Suggestive of AD Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Matchar DB, Kulasingam SL, McCrory DC, et al. Use of Positron Emission Tomography and Other Neuroimaging Techniques in the Diagnosis and Management of Alzheimer’s Disease and Dementia. Available at www.cms.gov/deteminationprocess/download/id0TZ.pdf.

22. Those with AD benefit: • ↓Mortality • ↓Disease progression Test[Clinical exam or clinical exam and PET] Treat if test is positive [with AChE-I] • Those without AD do not benefit: • No major harms from treatment Illustrative Example: Positron Emission Tomography in Treating Alzheimer’s Disease (3 of 8) • Step 2: A simplified analytic framework was constructed in order to explicitly link PET testing with outcomes. • The framework shows the reviewers’ understanding of the test setting and the role of test-and-treat strategies of interest. AChE-I = acetylcholinesterase inhibitor; AD = Alzheimer’s disease; PET = positron emission tomography Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Matchar DB, Kulasingam SL, McCrory DC, et al. Use of Positron Emission Tomography and Other Neuroimaging Techniques in the Diagnosis and Management of Alzheimer’s Disease and Dementia. Available at www.cms.gov/deteminationprocess/download/id0TZ.pdf.

23. Illustrative Example: Positron Emission Tomography in Treating Alzheimer’s Disease (4 of 8) • Step 2 (continued) • All patients with positive results from positron emission tomography testing are treated for Alzheimer’s disease (AD). • Those with AD (true positives) benefit from treatment. • Those without AD (false positives) do not benefit from treatment and are exposed to treatment-related risks. • Effects on mortality and disease progression are only indirect, resulting from a downstream clinical decision of whether to treat patients with AD. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Matchar DB, Kulasingam SL, McCrory DC, et al. Use of Positron Emission Tomography and Other Neuroimaging Techniques in the Diagnosis and Management of Alzheimer’s Disease and Dementia. Available at www.cms.gov/deteminationprocess/download/id0TZ.pdf.

24. Illustrative Example: Positron Emission Tomography in Treating Alzheimer’s Disease (5 of 8) • Step 2 (continued): A decision tree is constructed to assess management options for positive and negative test results • Scenario B: Diagnosis of mild cognitive impairment • Describes classification (i.e., true positive, false positive, false negative, true negative) consequences and compares test-and-treat strategies AD = Alzheimer’s disease MCI = mild cognitive impairment PET = positron emission tomography *When applicable; testing harms not important enough to consider. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Matchar DB, Kulasingam SL, McCrory DC, et al. Use of Positron Emission Tomography and Other Neuroimaging Techniques in the Diagnosis and Management of Alzheimer’s Disease and Dementia. Available at www.cms.gov/deteminationprocess/download/id0TZ.pdf.

25. Illustrative Example: Positron Emission Tomography in Treating Alzheimer’s Disease (6 of 8) • Step 3: Assess whether modeling could be useful in the evidence report evaluating the use of positron emission tomography (PET) to confirm Alzheimer’s disease (AD). • No test-and-treat strategies have been compared head to head. • No “clear winner” could be identified on test performance alone. • Evidence exists to estimate the benefits and harms of treatment in patients with and without AD, but treatments are only marginally effective. • It is unknown whether subgroups of patients identified with PET may have a differential response to treatment. • Therefore, modeling was deemed useful. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Matchar DB, Kulasingam SL, McCrory DC, et al. Use of Positron Emission Tomography and Other Neuroimaging Techniques in the Diagnosis and Management of Alzheimer’s Disease and Dementia. Available at www.cms.gov/deteminationprocess/download/id0TZ.pdf.

26. Illustrative Example: Positron Emission Tomography in Treating Alzheimer’s Disease (7 of 8) • Step 4: Assess whether previous modeling studies can be utilized. • It is not stated whether the systematic reviewers searched for previous modeling studies. • Using search terms such as “model(s),”“modeling,”“simulat*,” or terms for decision or economic analysis may lead to potentially useful pre-existing models. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Matchar DB, Kulasingam SL, McCrory DC, et al. Use of Positron Emission Tomography and Other Neuroimaging Techniques in the Diagnosis and Management of Alzheimer’s Disease and Dementia. Available at www.cms.gov/deteminationprocess/download/id0TZ.pdf.

27. Illustrative Example: Positron Emission Tomography in Treating Alzheimer’s Disease (8 of 8) • Step 5: Consider whether modeling is practically feasible in the time frame given. • The reviewers concluded that they could achieve their goals for modeling in the available time frame. • The systematic reviewers performed decision modeling to: • Better contextualize their findings • Explore whether conclusions would differ if available treatment options were more effective • These “what if” scenarios can give information on the robustness of conclusions in a systematic review and also aid in communicating conclusions to decisionmakers. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Matchar DB, Kulasingam SL, McCrory DC, et al. Use of Positron Emission Tomography and Other Neuroimaging Techniques in the Diagnosis and Management of Alzheimer’s Disease and Dementia. Available at www.cms.gov/deteminationprocess/download/id0TZ.pdf.

28. Key Messages • Systematic reviewers should consider: • Clinical utility as well as test performance • Whether modeling would enhance the interpretation of test-performance data • Whether modeling may be helpful in obtaining insight into the interplay of various factors on decision-relevant effects • The proposed five-step algorithm can help determine whether decision modeling is appropriate for a systematic review. Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

29. Practice Question 1 (1 of 2) • Medical test performance, or accuracy, is an indirect outcome. • True • False

30. Practice Question 1 (2 of 2) Explanation for Question 1: The statement is true. Accurate diagnosis or changes in test performance are considered indirect outcomes, as they do not change patient-relevant outcomes.

31. Practice Question 2 (1 of 2) • A “clear winner” among multiple test options can be demonstrated by test accuracy alone. • True • False

32. Practice Question 2 (2 of 2) Explanation for Question 2: The statement can be true when certain conditions are met, including when patients, regardless of which test is used, have the same response to downstream treatments.

33. Practice Question 3 (1 of 2) • Generally speaking, modeling is especially useful in which situation: • When there are clear direct links between testing and health outcomes • When a multitude of test-and-treat strategies can be contrasted • When there is one test that is clearly superior to others • When there is a scarce amount of information

34. Practice Question 3 (2 of 2) Explanation for Question 3: The correct answer is b. Modeling is usually not useful when there is scarce information available. It is most helpful when there are only indirect links between testing and health outcomes, when one test is not clearly superior, and when there are multiple test-and-treat strategies to explore.

35. Practice Question 4 (1 of 2) • Test performance (e.g., sensitivity and specificity) should be considered direct outcomes for assessing the clinical effectiveness of medical tests. • True • False

36. Practice Question 4 (2 of 2) Explanation for Question 4: The statement is false. Test performance numbers such as sensitivity and specificity are considered indirect for clinical outcomes, as performance of the test itself is not a direct clinical outcome. The results influence downstream clinical decisions that eventually determine outcomes.

37. Authors • This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center. • The module is based on Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 10.1-13. AHRQ Publication No. 12-EHC017. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

38. References (1 of 5) • Bossuyt PM, McCaffery K. Additional patient outcomes and pathways in evaluations of testing. Med Decis Making 2009 Sep-Oct;29(5):E30-8. PMID: 19726782. • Claxton K, Ginnelly L, Sculpher M, et al. A pilot study on the use of decision theory and value of information analysis as part of the NHS Health Technology Assessment programme. Health Technol Assess 2004 Jul;8(31):1-103, iii. PMID: 15248937. • Habbema JD, Van Oortmarssen GJ, Lubbe JT, et al. The MISCAN simulation program for the evaluation of screening for disease. Comput Methods Programs Biomed 1985 May;20(1):79-93. PMID: 3849380. • Janssen MP, Koffijberg H. Enhancing value of information analyses. Value Health 2009 Sep;12(6):935-41. PMID: 19432841. • Karnon J, Goyder E, Tappenden P, et al. A review and critique of modelling in prioritising and designing screening programmes. Health Technol Assess 2007 Dec;11(52):iii-iv, ix-xi, 1-145. PMID: 18031651.

39. References (2 of 5) • Lord SJ, Irwig L, Bossuyt PM. Using the principles of randomized controlled trial design to guide test evaluation. Med Decis Making 2009 Sep-Oct;29(5):E1-12. PMID: 19773580. • Lord SJ, Irwig L, Simes RJ. When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials? Ann Intern Med 2006 Jun 6;144(11):850-5. PMID: 16754927. • Matchar DB, Kulasingam SL, McCrory DC, et al. Use of Positron Emission Tomography and Other Neuroimaging Techniques in the Diagnosis and Management of Alzheimer's Disease and Dementia. Technology Assessment (Prepared by the Duke Evidence-based Practice Center under Contract No. 290-97-0014, Task Order 7). Rockville, MD: Agency for Healthcare Research and Quality; December 2001. Available at www.cms.gov/ determinationprocess/downloads/id9TA.pdf.

40. References (3 of 5) • Meltzer DO, Hoomans T, Chung JW, et al. Minimal Modeling Approaches to Value of Information Analysis for Health Research. Methods Future Research Needs Report No. 6 (Prepared by the University of Chicago Medical Center through the Blue Cross and Blue Shield Association Technology Evaluation Center Evidence-based Practice Center under Contract No. 290-2007-10058). Rockville, MD: Agency for Healthcare Research and Quality; June 2011. AHRQ Publication No. 11-EHC062-EF. Available at www.ncbi.nlm.nih.gov/ books/NBK62146. • Oostenbrink JB, Al MJ, Oppe M, et al. Expected value of perfect information: an empirical example of reducing decision uncertainty by conducting additional research. Value Health 2008 Dec;11(7):1070-80. PMID: 19602213. • Samson D, Schoelles K. Developing the topic and structuring systematic reviews of medical tests: utility of PICOTS, analytic frameworks, decision trees, and other frameworks. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 2.1-14. AHRQ Publication No. 12-EHC017. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

41. References (4 of 5) • Tatsioni A, Zarin DA, Aronson N, et al. Challenges in systematic reviews of diagnostic technologies. Ann Intern Med 2005 Jun 21;142(12 Pt 2):1048-55. PMID: 15968029. • Trikalinos TA, Dahabreh IJ, Wong J, et al. Future Research Needs for the Comparison of Percutaneous Coronary Interventions With Bypass Graft Surgery in Nonacute Coronary Artery Disease: Identification of Future Research Needs From Comparative Effectiveness Review No. 9. Future Research Needs Papers No. 1 (Prepared by the Tufts Evidence-based Practice Center under Contract No. 290-2007-10055-I). Rockville, MD: Agency for Healthcare Research and Quality; September 2010. AHRQ Publication No. 10-EHC068-EF. Available at www.ncbi.nlm.nih.gov/books/NBK51079. • Trikalinos TA, Siebert U, Lau J. Decision-analytic modeling to evaluate benefits and harms of medical tests: uses and limitations. Med Decis Making 2009 Sep-Oct;29(5):E22-9. PMID: 19734441.

42. References (5 of 5) • Trikalinos TA, Kulasingam S, Lawrence WF. Deciding whether to complement a systematic review of medical tests with decision modeling. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 10.1-13. AHRQ Publication No. 12-EHC017. Available at www.effectivehealthcare.ahrq.gov/ medtestsguide.cfm.