1 / 40

Interpreting Clinical Trial Results Betty Anne Schwarz MSc, BA, RN

Interpreting Clinical Trial Results Betty Anne Schwarz MSc, BA, RN. OHRI Clinical Research Conference Workshop October 16, 2012. Interpreting Clinical Trial Results. “I have no conflicts of interest related to this presentation”. Interpreting Clinical Trial Results. Workshop Applicability

senona
Télécharger la présentation

Interpreting Clinical Trial Results Betty Anne Schwarz MSc, BA, RN

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Interpreting Clinical Trial ResultsBetty Anne Schwarz MSc, BA, RN OHRI Clinical Research Conference Workshop October 16, 2012

  2. Interpreting Clinical Trial Results “I have no conflicts of interest related to this presentation”

  3. Interpreting Clinical Trial Results Workshop Applicability • Anyone who is working in clinical trials that is accustomed to reading published literature (RCTs)

  4. Interpreting Clinical Trial Results Objectives • Review what is considered statistically significant when interpreting clinical trial results • Illustrate how statistically significant is not the same as clinically significant • Interactive workshop will involve participation of the audience through review of specific trials to determine clinical significance with the tools provided

  5. Interpreting Clinical Trial Results Outline of Workshop • Review statistical features that are routinely expected within published trial results • Focus on the merits of considering MCID (minimally clinical important difference) Look at examples of positive vs. negative trials • Break into groups – utilize tools provided in reviewing specific trials to determine clinical significance IV. Review and Summarize

  6. Interpreting Clinical Trial Results Statistics & Clinical Trials • Interpreting trials – depends on level of comprehension and experience I. Key Statistical Terms • Choice of statistical test utilized to analyze data (interval or categorical, paired vs. unpaired) depends on the data being analyzed • Null hypothesis proposes ~ no difference between study groups with respect to variable's of interest Ref: (2007) McCluskey & Ghaaliq Lalkhen

  7. Interpreting Clinical Trial Results I. Key Statistical Terms • Unpaired vs. Paired Data ~ compare effects of an intervention within study samples imperative groups are as similar as possible • Randomization – large enough sample size • Ensures group differences within the groups cancel out as they may influence outcome of interest (weight, age, sex ratio, smoking habit) • Study contained independent groups – unpaired statistical tests • i.e., comparison of efficacy for two different drugs – HTN trial Ref: (2007) McCluskey & Ghaaliq Lalkhen

  8. Interpreting Clinical Trial Results I. Statistical vs. clinical significance • One should not be confused by another Scenario • Comparison of 2 hypotensive agents: mean arterial pressure after Rx. with drug A is 2 mm lower < Rx. with drug B. • If sample size is not large enough even a small difference between the 2 groups may be statistically significant with a P <0.05. • The clinical significance of a 2 mm Hg drop in mean arterial pressure is small and therefore not clinically significant Ref: (2007) McCluskey & Ghaaliq Lalkhen

  9. Interpreting Clinical Trial Results I. “Statistical Significance” and P Values • “Not everything that can be counted counts, and not everything that counts can be counted” Albert Einstein • Statistically significant simply means that a result is most likely caused by something other than chance • In other words ~ significant does not mean important • Segue into what’s considered clinically significant • Interpreting trial results requires a P value (conventional P < 0.05 statistically significant). P ≥ 0.05 not significant, P <0.01 highly significant Ref: (2004) libdoc.who.int.,(2007) McCluskey & Ghaaliq Lalkhen,

  10. Interpreting Clinical Trial Results I. Confidence Intervals (CI) • Range of sample data measures the probability that a population parameter will fall between two set values. The confidence interval can take any number of probabilities, with the most common being 95% or 99% • CI should always be presented for the relative risk (RR) and odds ratio (OR) • A RR or OR of 1 ~ no association between the risk factors and the disease under study • If RR > 1, but CI overlaps 1 – the increase in risk is not statistically significant and can attributed to chance Ref: (2004) libdoc.who.int

  11. Interpreting Clinical Trial Results I. Important Variables and Concepts • Study power ~ Type I and Type II errors • After data has been analyzed, the null hypothesis is either accepted or rejected based on the P value • If the null hypothesis is true and the P value (P < 0.05) is obtained; incorrect inference is drawn that the data collected from the sample groups is different : TYPE I STATISTICAL ERROR • From the data collected conclude null hypothesis is false but the P value obtained is ≥0.05 ~ conclude sample groups are similar but in fact missed a real difference: TYPE II STATISTICAL ERROR Ref: (2004) libdoc.who.int

  12. Interpreting Clinical Trial Results I. Important Variables and Concepts • Interpreting diagnostic tests ~ sensitivity and specificity • Comparing to gold standard or reference standard • Sensitivity – ability of a test to single out people who have disease. Low sensitivity ~ many false positives • Specificity – ability of a test to identify people who do not have a disease (negative). A low sensitivity ~ many false positives • Predictive Value (PV) – the frequency in which a positive test actually signifies disease • Confound • Bias Ref: (2004) libdoc.who.int Ref: (2004) libdoc.who.int

  13. Interpreting Clinical Trial Results II. MCID (minimally clinical important difference) • Interpretation of RCTs emphasis on statistical significance rather than clinical relevance • CONSORT (Consolidated Standards of Reporting Trials) statement developed to help authors improve their methods in reporting trials results with the aid of a checklist and flow diagram • Enable readers to comprehend the trial methodology and assess the validity of the results • Recently revised based on new evidence and address criticism Ref: (2001) Chan et al., (2001) Moher et al. Ref: (2004) libdoc.who.int

  14. Interpreting Clinical Trial Results II. MCID (minimally clinical important difference) • CONSORT revised – failed to recommend that authors specifically discuss the clinical importance of their results • Chan et al., (2001) randomly chose 27 (total 266) published RCTs from 5 major journals over a 1 year time period that were independently reviewed by 4 reviewers to discern if factors they considered as clinically relevant were stated in the study results Ref: (2001) Chan et al., (2001) Moher et al. Ref: (2004) libdoc.who.int

  15. Interpreting Clinical Trial Results II. MCID (minimally clinical important difference) • Specifically ~ primary outcome clearly defined, expected difference between the two groups reported used in the calculation of the sample size (delta value), whether the results were based on minimal clinically important difference of the intervention, the statistical significance of the results, presentation of confidence intervals pertinent to the study findings and finally the authors overall interpretation of the clinical relevance of the trial results Ref: (2001) Chan et al., (2001) Moher et al. Ref: (2004) libdoc.who.int

  16. Interpreting Clinical Trial Results II. MCID (minimally clinical important difference) • Defined as the smallest treatment effect required that would render a change in the management of a patient taking into account the side effects, costs and overall inconveniences • Key concept to be considered • Designing RCT Trial ~ calculate the sample size needed to detect magnitude of difference between the treatment groups that the study can reliably detect (delta value) • Key point ~ for a trial to have a reasonable chance of detecting clinically important effect size, the delta value needs to reflect the MCIDs for the trial interventions Ref: (2001) Chan et al., (2001) Moher et al. Ref: (2004) libdoc.who.int

  17. Interpreting Clinical Trial Results II. MCID (minimally clinical important difference) • Example: individuals without a previous history of MI or CVA; regular use of ASA will reduce the incidence of MI by 0.2% per year (from a baseline rate of 0.7%/year to 0.5%/year which is a RR reduction of about 25%) • Benefit is offset by a concomitant absolute increase in the chance of CVA by 0.02% per year (from 0.30%/year – 0.32%/year, which is now a relative increase of about 10%/year) and GI bleed of about 1% per year (from about 1%/year – 2%/year) • Thoughts? Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  18. Interpreting Clinical Trial Results II. MCID (minimally clinical important difference) • Weigh the benefits and disadvantages of taking ASA in this clinical situation; expert panel recommended not to take ASA in the prevention of MI as there was insufficient evidence to overcome the increased incidence of CVA, GI bleed in this low risk group. • Therefore; the efficacy of ASA in this particular scenario was insufficient to either meet or exceed its MCID Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  19. Interpreting Clinical Trial Results II. MCID (minimally clinical important difference) • Compare the actual study results ~ include point estimates and accompanying confidence intervals with the MCID values will render the clinical importance of the study results • Point estimate: can be either a single number or a range of scores. • Point estimates are not usually as informative as confidence intervals. Their importance lies in the fact that many statistical formulas are based on them Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  20. Interpreting Clinical Trial Results II. MCID (minimally clinical important difference) • If the MCID estimate is less than the value of the lower limit of the 95% CI ~ results are statistically significant and most likely clinically significant • If the MCID value is greater than the upper limit of the 95% CI, the results are very likely to be less clinically significant • If the results provided for the MCID are somewhere in the middle of the CI – the clinical importance is less clear Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  21. Interpreting Clinical Trial Results Methods • December 1, 1998 – November 30, 1999 of 266 articles randomly chosen - 27 RCTS were selected from 5 peer reviewed journals • Utilized standard data collection sheets to evaluate key features when interpreting the trial results and whether the results were clinically significant (MCID) • Disagreements were resolved collaboratively Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  22. Interpreting Clinical Trial Results Methods (hand-out) 1. Primary Outcome – clearly defined in the methods section and is also defined within the calculation of a sample size • No sample size reported – remainder of the article was reviewed for an explicitly stated primary outcome • No primary outcome – the outcome that was deemed of greatest clinical importance was chosen Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  23. Interpreting Clinical Trial Results Methods 2. Reporting of a Sample Size Delta Value • Magnitude of difference in outcomes between treatment groups that the trial was attempting to detect • Recorded as either the author’s interpretation of a MCID for the various study intervention, and whether or not the value was in absolute or relative terms 3. Statistical significance of results for the primary outcome • P value < 0.05% Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  24. Interpreting Clinical Trial Results Methods 4. Confidence Intervals • Primary outcome – whether the 95% CI surrounded the point estimate for the efficacy of a particular trial intervention 5. Authors overall interpretation of clinical importance found from their trial results • If the results were discussed – clinical relevance explicitly noted whereas; indirect reference to the results was considered implicitly reported Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  25. Interpreting Clinical Trial Results Methods Levels of Justification • Rating scale used to grade the level of strength that was developed to justify the reviewers overall interpretation of the clinical importance from the trial results obtained • Level 1: the clinical importance of the primary study result in relation to the work done to determine the MCID • Level 4: no accompanying justification Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  26. Methodological attributes important in the interpretation of study results from a clinical perspective Section Attribute Explicit primary outcome stated Expected magnitude of difference “delta value” stated Expected magnitude of difference identified explicitly as the MCID Delta/MCID value reported as an absolute value Ref: (2001) Chan et al., Methods

  27. Methodological attributes important in the interpretation of study results from a clinical perspective Section Attribute Statistical significance of primary outcome reported Confidence intervals for primary outcome reported Clinical importance of primary outcome discussed Discussion explicit or implicit Level of justification Appropriate clinical interpretation of results Results Discussion Ref: (2001) Chan et al.,

  28. Interpreting Clinical Trial Results Workshop Articles: • GISSI-Prevenzione Investigators. Dietary Supplementation with n-3 polyunsaturated fatty acids and vitamin E after myocardial infarction: results of the GISSI-Prevenzione trial. Lancet 1999;335 (9177):447-55. 2. Cleare AJ., Heap, E, Malhi GS, Wessely, S, O’Keane V, Miell J. Low-dose hydrocortisone in chronic fatigue syndrome. A randomized crossover trial. Lancet 1999;353:455-58. Ref: (2001) Chan et al., (1999) Cleare et al., Ref: (2004) libdoc.who.int

  29. Interpreting Clinical Trial Results Workshop Articles: • GISSI-Prevenzione Investigators. Dietary Supplementation with n-3 polyunsaturated fats acids and vitamin E after myocardial infarction: results of the GISSI-Prevenzione trial. Lancet 1999;335 (9177):447-55. • Chan et al., (2001) found that 26% studies did not report a sample size or a delta value • If reported – reported in relative instead of absolute terms • Likewise the delta value did not appear to reflect the MCID of the intervention Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  30. Interpreting Clinical Trial Results Workshop Articles: • GISSI-Prevenzione Trial • Assessed the effect of dietary supplementation with polyunsaturated fats demonstrated an absolute decrease of 1.3% (95% CI 0.1%-2.6%) in the primary outcome (combined end point of death, nonfatal MI and stroke) • In the sample size calculation; delta value was a 4% absolute difference between groups over 3.5 year period • Therefore: the efficacy of the intervention found by the study was significantly smaller than the sample size delta Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  31. Interpreting Clinical Trial Results Workshop Articles: • GISSI-Prevenzione Trial • If we assume that this value represents the MCID of the intervention, the magnitude of the intervention's effect is definitely not clinically important. • Authors reported “a clinically important and statistically significant benefit”, their estimation of the MCID of the intervention is not reflected in the sample size delta Ref: (2001) Chan et al., (2002) Hing et al., Ref: (2004) libdoc.who.int

  32. Interpreting Clinical Trial Results Workshop Articles: 2. Cleare AJ., Heap, E, Malhi GS, Wessely, S, O’Keane V, Miell J. Low-dose hydrocortisone in chronic fatigue syndrome. A randomized crossover trial. Lancet 1999;353:455-58. • Assessed the effect of low-dose hydrocortisone on chronic fatigue disorder, a 9 point reduction on a fatigue scale was deemed to be clinically important. • However; the delta value for the sample size was reported as a 4 point reduction on the same scale Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  33. Interpreting Clinical Trial Results Workshop Articles: 2. Hydrocortisone in chronic fatigue study • Therefore; the authors reported that the study result, a statistically significant 4.5 point reduction on the fatigue scale when comparing the intervention group with the control group , was not clinically important. Ref: (2001) Chan et al., Ref: (2004) libdoc.who.int

  34. Interpreting Clinical Trial Results Interpreting Study Results • Study results are regarded as statistically significant (P < 0.5) when values indicating the lower limit of the 95% confidence interval (CI) are greater than the null effect • Judging clinical relevance can be difficult • Clinical relevance can take 4 forms, pending the relationship of the MCID on the intervention to the point estimate (the best single value of the efficacy of the intervention that has been derived from the study results) and the 95% CI surrounding it Ref: (2002) Hing et al., Ref: (2004) libdoc.who.int

  35. Interpreting Clinical Trial Results Interpreting Study Results • Definite: when the MCID is smaller that the lower limit of the 95% CI • Probable: when the MCID is greater than the lower limit of the 95% CI, but smaller that the point estimate of the efficacy of the intervention • Possible: when the MCID is less than the upper limit of the 95% CI, but greater than the point estimate of the efficacy of the intervention • Definitely not: when the MCID is greater than the upper limit of the 95% CI Ref: (2002) Hing et al., Ref: (2004) libdoc.who.int

  36. Interpreting Clinical Trial Results Study Results of Definite Clinical Importance • Randomized placebo controlled trial that assessed the efficacy of the addition of spironlactone to the medication regime of patients with severe heart failure. • Results demonstrated that the study intervention reduced the chance of death by an absolute value of 11.4% (95% CI, 6.7% to 16.1%). The delta value calculated for the sample size was an absolute 6.5% reduction in mortality. • If the delta value is assumed to be the MCID, the results of this study can be interpreted as both statistically significant and clinically important Ref: (2002) Hing et al., Ref: (2002) Hing et al., Ref: (2004) libdoc.who.int

  37. Interpreting Clinical Trial Results Limitations of MCID • Methodological limitation: uncertainty in the choice of MCID • MCID values will vary from person to person depending on values/perspectives • Without widespread used of sample size deltas that accurately reflect the MCID is it appropriate to use the sample size deltas as benchmarks for determining the clinical relevance? • However……….. • Compare the relationship of possible MCID values to the point estimate of the efficacy of the intervention and it’s CI – this will allow determining the level of clinical significance Ref: (2002) Hing et al., Ref: (2004) libdoc.who.int

  38. Interpreting Clinical Trial Results Summary Formal statistical methods of analyzing and reporting clinical trial results are routinely performed and expected within the literature Interpreting and reporting trial results from the perspective of clinical relevance is not always done with the same emphasis Creates imbalance as clinically significant trials are often considered clinically significant and those that are statistically insignificant are also clinically unimportant Encourage evaluate both when interpreting clinical trial results with the tools reviewed today (MCID) Ref: (2004) libdoc.who.int

  39. References1.Chan, K. B. J. Y., Man-Son-Hung, M., Molnar, F. J. & Laupacis, A. (2001)  'How well is the clinical importance of study results reported? As assessment of randomized controlled trials', CMAJ, 165, (9), pp. 1197-1202.2. Man-Son-Hing, M., Laupacis, A., O’Rourke, K., Molnar, F. J., Mahon, J., Chan, K. B. Y. & Wells, G. (2002) ‘Determination of the Clinical Importance of Study Results’, JGIM, (17) pp. 469-476.3. McCluskey, A. & Ghaaliq Lalkhen, A. (2007) ‘Statistical IV: Interpreting the results of statistical tests’, Continuing Education in Anesthesia Care & Pain, [Online]. DOI: 10.1093/biaceaccp/mkm042ical, Volume 7, (6) Accessed: September 16, 20124. Moher, D., Schulz, K. F. & Altman, D. G. (2001) ‘The CONSORT statement revised recommendations for improving the quality of reports of parallel group randomized trials’, BMC Medical Research Methodology, [Online]. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-1-2.pdfAccessed: September 17, 2012.5. Moher, D., Dulberg,, C. S. & Wells, G. (1994) ‘Statistical Power, Sample Size, and Their Reporting in Randomized Clinical Trials’, JAMA, 272 (2) pp. 122-124.6. Library Doctor WHO International (1994) ‘Chapter 9 Interpreting research results’, Available from: http://whqlibdoc.who.int/emro/2004/9290213639_chap9.pdf (Accessed: September 21, 2012).

  40. Thank you!

More Related