Acknowledgments

1. Multicriteria decision analyses for healthcare decisionmaking: feedback from the field on the value of the EVIDEM framework in Canadian and South African settings 25 April 2010, Boston Mireille M Goetghebeur PhD,1Jacqui Miot PhD, 2 Michele Tony*, Monika Wagner PhD,1 Hanane Khoury PhD,1 Donna Rindress PhD,1 Paul Oh MD 3 1BioMedCom, Montreal, Quebec, Canada 2Division of Clinical Epidemiology, University of Pretoria, South Africa 3Endocrinology Toronto Rehabilitation Institute, Toronto, Ontario, Canada *MSc Candidate, University of Montreal

2. Acknowledgments Committee members • Drug Advisory Committee, Workplace Safety Insurance Board (WSIB) of Ontario, Toronto, Canada • Clinical Policy Unit, Discovery Health, Johannesburg, South Africa Web developers • Patricia Campbell BSc & Peter Melnyk PhD, BioMedCom, Montreal, Canada Funding • Internal sources of support provided by BioMedCom, WSIB and Discovery Health

3. Disease severity Patient reported outcomes System capacity Safety Efficacy Quality of evidence Historical context Unmet needs Personal values Cost Ethics Priorities Budget impact Expert opinion Structuring the natural thinking process Baltussen& Niessen. Cost Eff Resour Alloc. 2006;4:14

4. Background: MCDA & EVIDEM • Multicriteria decision analysis (MCDA) • Reflection on criteria at play in healthcare decisionmaking • Reflection on perspectives, values and priorities • Simultaneous consideration & quantification of a wide range of criteria of decision (beyond cost-effectiveness model) • EVIDEM Collaboration • Framework*: Comprehensive set of standard criteria of decision with tools and processes to consider them • Collaborative Registry: open web based interactive relational database with synthesized evidence for decisionmaking contexts globally *Goetghebeur M, Wagner M, Khoury H, Levitt R, Erickson LJ, Rindress D. BMC Health Serv Res 2008; 8:270.

5. Standard criteria of decision Ethical framework • Goals of healthcare - utility • Opportunity costs – efficiency • Population priority & access – fairness Disease impact • Disease severity • Size of population affected by disease Ethical framework • Goals of healthcare - utility • Opportunity costs – efficiency • Population priority & access – fairness Disease impact • Disease severity • Size of population affected by disease Context of intervention • Clinical guidelines • Comparative intervention limitations Context of intervention • Clinical guidelines • Comparative intervention limitations Other system-related criteria • System capacity and appropriate use of intervention (e.g., infrastructure) • Stakeholder pressures (e.g., media) • Political/historical context (e.g. precedence, national priority) • Interventionoutcomes • Improvement of efficacy/effectiveness • Improvement of safety and tolerability • Improvement of patient reported outcomes Other system related criteria • System capacity and appropriate use of intervention • Stakeholder pressures • Political/historical context • Interventionoutcomes • Improvement of efficacy/effectiveness • Improvement of safety and tolerability • Improvement of patient reported outcomes • Type of benefit • Public health interest (e.g., prevention, risk reduction) • Type of medical service (e.g., symptom relief, cure) • Type of benefit • Public health interest (e.g., prevention, risk reduction) • Type of medical service (e.g., symptom relief, cure) • Economics • Budget impact on health plan (cost of intervention) • Impact on other spending (e.g., hospitalization, disability) • Cost-effectiveness of intervention • Economics • Budget impact on health plan (cost of intervention only) • Impact on other spending (e.g., hospitalization, disability) • Cost-effectiveness of intervention Quality/uncertainty of evidence • Adherence to requirements of decisionmaking body • Completeness and consistency of reporting evidence • Relevance and validity of evidence Quality/uncertainty of evidence • Adherence to requirements of decisionmaking body • Completeness and consistency of reporting evidence • Relevance and validity of evidence 5 • *Defined using MCDA guidelines : non-redundancy (no double counting), can be considered independently, operationalizable, comprehensive

6. Objectives • Field test the framework using case studies selected by decisionmaking committees • Tramadol for chronic non-cancer pain for a public Canadian health plan • Liquid-based cytology for screening cervical cancer for a private South African health plan

7. Methods Discussion Extrinsic criteria Qualitative Tool Step 3 Impacts Committee members Systematic consideration of criteria of decision Intrinsic criteria MCDA Matrix Step 1- Weights Step 2 - Scores HTA Report (web based) Synthesized evidence Critical analysis of evidence Investigators Evidence organized to investigate each criteria of decision Evidence available Peer-reviewed literature HTA reports Registries Cochrane reviews Disease Association Other relevant resources Framework tools available from EVIDEM Collaboration – www.evidem.org

8. Data collection & analyses • Weights, scores and impacts: elicited during workshop sessions with committee members • Feedback on framework : elicited via survey & discussion during workshop sessions • MCDA estimate: linear model [sum of value contribution (Vx) of combined normalized weights (Wx) and scores (Sx) for applicable criteria (n) of the matrix]

9. Results

10. Web based interactive HTA reports • Open source software (Tikiwiki) • Access to evidence • Structured by criterion of decision • Synthesized evidence • High level synthesis (to get a quick grasp of issues) • Synthesis (with more details on studies and data) • Full text sources of data • Web links to data sources (peer-review journals, Cochrane reviews, reports etc) • Data entry & collection • Weights, scores, impacts, feedback • Data collection (MySQL database) Examples of prototypes: www.evidem.org/evidem-collaborative.php

14. Step 2 – MCDA scores tramadol for chronic non-cancer pain Canadian study • Step1 – MCDA weights values of committee • Step 3 – Qualitative impact of extrinsic criteria

15. For an intervention to achieve close to 1 on this scale, it would have to cure a severe endemic disease, demonstrate a major improvement in safety, efficacy and PRO compared to limited existing approaches, and result in major healthcare savings. An intervention that scores low would be for a rare disease that is not severe, with limited data showing small improvement in efficacy & major safety and PRO issues over existing alternatives, & resulting in major increases in healthcare spending ± depending on qualitative considerations

16. Step 2 – MCDA scores liquid based-cytology for cervical cancer screening South African study • Step1 – MCDA weights values of committee • Step 3 – Qualitative impact of extrinsic criteria

17. ± depending on qualitative considerations

18. Outcomes of applying framework • Transparent record of evidence for each criteria of decision • Quantitative considerations • Preferences (weights) of the group/committee • Performance of intervention (scores) • MCDA estimate • Specific to committee • Comprehensive measure • Qualitative considerations: positive, neutral or negative impact on value of intervention • Decision: ranking & striking a balance

19. Feedback from the field • Criteria of the framework • Most criteria should be systematically considered • Criteria questioned by some members • Intrinsic: Size of population, Disease severity, Improvement of patient-reported outcomes (PRO), Public health interest • Extrinsic: Stakeholders pressures, Political/historical context

20. Feedback from the field • Advantages • Systematic consideration of all aspects of decision • Consistency of decision process • Transparency & clarity • Understanding of intervention (ZA) • Understandability of decision by stakeholders (ZA) • Challenges • Language & understanding of some criteria of decision • Perceived difficulties in distilling information/evidence to populate framework • Lack of reference point for MCDA estimates (ranking)

21. Applications • Decisionmaking/priority setting • Knowledge translation • Communication • Research planning • Further testing and validation needed to develop MCDA approaches in healthcare decisionmaking

22. Collaborative studies Field testing (beta-testing package) Methodological validation International survey Web collaborative registry Open access to evidence on health care interventions for end users globally Ranking of interventions Optimize resources, decisions and health Plans for the Future

23. Framework tools available from the EVIDEM Collaboration www.evidem.org Thank you

24. Standard criteria of decision Disease impact • Disease severity • Size of population affected by disease Ethical framework • Goals of healthcare - utility • Opportunity costs – efficiency • Population priority & access – fairness Context of intervention • Clinical guidelines • Comparative intervention limitations Other system-related criteria • System capacity and appropriate use of intervention (e.g., infrastructure) • Stakeholder pressures (e.g., media) • Political/historical context (e.g. precedence, national priority) • Interventionoutcomes • Improvement of efficacy/effectiveness • Improvement of safety and tolerability • Improvement of patient reported outcomes • Type of benefit • Public health interest (e.g., prevention, risk reduction) • Type of medical service (e.g., symptom relief, cure) • Economics • Budget impact on health plan (cost of intervention) • Impact on other spending (e.g., hospitalization, disability) • Cost-effectiveness of intervention Quality/uncertainty of evidence • Adherence to requirements of decisionmaking body • Completeness and consistency of reporting evidence • Relevance and validity of evidence • *Defined using MCDA guidelines : non-redundancy (no double counting), can be considered independently, operationalizable, comprehensive