Introduction to Critical Appraisal : Quantitative Research

1. Introduction to Critical Appraisal : Quantitative Research South East London Outreach Librarians January 2008

2. Learning objectives • Understand the principles of critical appraisal and its role in evidence based practice • Be able to appraise quantitative research and judge its validity • Be able to assess the relevance of published research to your own work

3. Daily Mail exercise • Would you treat a patient based on this article? Why? • Validity • Reliability • Transferable to practice

4. What is evidence based practice? Evidence-based practice is the integration of • individual clinical expertise with the • best available external clinical evidence from systematic research and • patient’s values and expectations

5. The evidence-based practice process. • Decision or question arising from a patient’s care. • Formulate a focused question. • Search for the best evidence. • Appraise the evidence. • Apply the evidence.

6. Why does evidence from research fail to get into practice? • 75% cannot understand the statistics • 70% cannot critically appraise a research paper • Using research for Practice: a UK experience of the barriers scale. Dunn, V. et al.

7. What is critical appraisal? • Weighing up evidence to see how useful it is in decision making • Balanced assessment of benefits and strengths of research against its flaws and weaknesses • Assess research process and results • Skill that needs to be practiced by all health professionals as part of their work

8. What critical appraisal is NOT • Negative dismissal of any piece of research • Assessment of results alone • Based entirely on statistical analysis • Only to be undertaken by researchers/ statisticians

9. Why do we need to critically appraise? • “It usually comes as a surprise to students to learn that some (the purists would say 99% of) published articles belong in the bin and should not be used to inform practice” (Greenhalgh 2001) • Find that 1% - save time and avoid information overload

10. How do I appraise? • Mostly common sense. • You don’t have to be a statistical expert! • Checklists help you focus on the most important aspects of the article. • Different checklists for different types of research. • Will help you decide if research is valid and relevant.

11. Quantitative Uses numbers to describe and analyse Useful for finding precise answers to defined questions Qualitative Uses words to describe and analyse Useful for finding detailed information about people’s perceptions and attitudes Research methods

12. Levels of quantitative evidence. (In order of decreasing scientific validity.) • Systematic reviews. • Randomized controlled trials. • Prospective studies (cohort studies). • Retrospective studies (case control). • Case series and reports • Opinions of respected authorities.

13. Systematic Reviews. • Thorough search of literature carried out. • All RCTs (or other studies) on a similar subject synthesised and summarised. • Meta-analysis to combine statistical findings of similar studies.

14. Randomised Controlled Trials (RCTs) • Normal treatment/placebo versus new treatment. • Participants are randomised. • If possible should be double-blinded. • Intention to treat analysis

15. Cohort studies • prospective • groups (cohorts) • exposure to a risk factor • followed over a period of time • compare rates of development of an outcome of interest • Confounding factors and bias

16. Case control studies • Retrospective • Subjects confirmed with a disease (cases) are compared with non-diseased subjects (controls) in relation to possible past exposure to a risk factor. • Confounding factors and bias

17. Appraising original research Are the results valid? • Is the research question focused? • Was the method appropriate? • How was it conducted, e.g. randomisation, blinding, recruitment and follow up? What are the results? • How was data collected and analysed? • Are they significant? Will the results help my work with patients?

18. Appraising systematic reviews. In addition to the above: • Was a thorough literature search carried out ? • Publication bias - papers with more ‘interesting’ results are more likely to be: • Submitted for publication • Accepted for publication • Published in a major journal • Published in the English language

19. Reviews in general medical journals • 50 reviews in 4 major journals 1985-6 • No statement of methods • Summary inappropriate • “Current systematic reviews do not routinely use scientific methods to identify, assess and synthesise information” (Mulrow, 1987)

20. Is the research question focused? • Patient (e.g. child) • Intervention (e.g. MMR vaccine) • Comparison (e.g. single vaccines) • Outcome (e.g. autism)

21. Are results significant? • How was data collected? • Which statistical analyses were used? • How precise are the results? • How are the results presented?

22. Intention to treat analyses • Analysing people, at the end of the trial, in the groups to which they were randomised, even if they did not receive the intended intervention

23. Statistical analyses Odds ratios, absolute and relative risks/benefits • The likelihood of something happening vs the likelihood of something not happening Numbers needed to treat (NNT) • The number of people you would need to treat to see one additional occurrence of a specific beneficial outcome

24. Odds Ratio Diagrams. (Blobbograms or Forest Plots.)

25. Odds Ratio Diagrams • Line of no effect – no difference between treatment and control group • Result (blob) to the Left of the line of no effect = Less of the outcome in the treatment group. • Result to the Right of the line = More of the outcome. • BUT - Is the outcome good or bad?

26. Cardiac deaths – Less = good

27. Smoking cessation – More = good

28. Confidence Intervals. • Longer confidence interval = less confident of results – wider range. • Shorter confidence interval = more confident – narrower range. • Crosses line of no effect/no significance =Inconclusive results.

29. Confidence intervals

30. P Values. • P stands for probability - how likely is the result to have occurred by chance? • P value of less than 0.05 means likelihood of results being due to chance is less than 1 in 20 = “statistically significant”. • P values and confidence intervals should be consistent

31. Number Needed to Treat • The number of people you would need to treat to see one additional occurrence of a specific beneficial outcome. • The number of patients that need to be treated to prevent one bad outcome. • The NNT can be calculated by finding the Absolute Risk Reduction (ARR)

32. Events or outcomes are used for reporting results. The event rate is the proportion of patients in a group in whom the event is observed

33. CER and EER • Control Event Rate (CER) is the proportion of patients in the control group in whom an event is observed. CER = c/(c+d) • Experimental Event Rate (EER) is the proportion of patients in the experimental in whom an event is observed. EER = a/(a+b)

34. AAR & NNT • Absolute Risk Reduction is the difference between the Control Event Rate (CER) and the Experimental Event Rate (EER). ARR = CER – EER • Number needed to treat (NNT) NNT = 1/ARR

35. Outcome event Total Yes No Experimental group 3 7 10 Control group 5 5 10 Total 8 12 20

36. Answers • What is the event ? • Lack of concentration and sleeping • What is the control event rate (CER)? • 5/10 = 0.50 • What is the experimental event rate (EER)? • 3/10 = 0.30 • Calculate the absolute risk reduction (ARR) • 0.50 – 0.30 = 0.20 • What is the number needed to treat (NNT)? • 1.00/0.20 = 5

37. Are results relevant? • Can I apply these results to my own practice? • Is my local setting significantly different? • Are these findings applicable to my patients? • Are findings specific/detailed enough to be applied? • Were all outcomes considered?

38. The good news! • Some resources have already been critically appraised for you. • An increasing number of guidelines and summaries of appraised evidence are available on the internet.

39. Summary. • Search for resources that have already been appraised first, e.g. Guidelines, Cochrane systematic reviews. • Search down through levels of evidence, e.g. systematic reviews, RCTs. • Use checklists to appraise research. • How can these results be put into practice?