Centre for Evidence Based Intervention

1. Centre for Evidence Based Intervention Evidence from Review Groups

2. Objectives Identify policies & procedures on… • Determining publication of empty reviews • Practice of handling excluded studies across Cochrane and other providers • Informing practice based on empty reviews and excluded studies

3. Methods • Survey with closed and open-ended questions • 53 Cochrane Collaboration Review Groups • WHO, JBI, What Works Clearinghouse and SCIE • Coded & analyzed responses

4. Why might a review be empty? • “because it is a new or recently introduced intervention which has yet to gather strength of evidence” • “the intervention does not lend itself to randomised controlled trials” • “poor question?” • ……..

5. Cochrane Groups publishing empty reviews? N=53

6. Rationale for publishing Empty Reviews N=46 % of responses

7. Other providers • WHO - doesn't have a policy on empty reviews. In a way it isn't relevant - they would search for the best available evidence, but would need to make a recommendation regardless of what they find. • SCIE - As the evidence base in social care is very limited, we may often find that a review with inclusion criteria specifying high quality controlled (and/or randomised) studies of specific interventions would be ‘empty’ – hence we would not commission a review. • WWCH - review focus is changed depending on the evidence base, thus empty reviews are not generally produced.

8. Views about including empty reviews • Include to instigate further research • “Our aim with empty reviews is to raise the profile of the question and hopefully generate research leading to high quality evidence which we can then report quickly in an updated review.” • Include if good case made • “There may be benefit in amending the TRF form to include the line: ‘If you think this review might be empty, please give your reasons why you feel it is important to do the review.’”

9. Reasons for not publishing empty reviews (Some groups have minimum # of studies stipulation - e.g., 2-3) • Not worth time and resources • “a colossal waste of everyone’s time and energy” • Influence on group’s impact factor • May be misleading - Empty reviews (< 3 studies) can put positive supportive evidence from initial, low quality studies behind treatment • BUT, question can still matter • “may be exceptions where review questions appear of great public health relevance”

10. Handling excluded studies Standard policy? Permitted in review?

11. Conditions for providing evidence based on excluded studies N=31 % of responses

12. Further considerations for whether to allow excluded studies • Excluded based on intervention or outcomes? • RCTs feasible/ethical for topic area? • If not, lower grade studies may be acceptable • Guidelines may be needed for application of evidence from excluded studies • “We dissuade authors from providing evidence from excluded studies because of inconsistency, which would lead to provision of evidence that is arbitrary and unsystematic.”

13. Guidance for ‘Implications for practice’ section For ‘Yes’ responses: (n=7) • Extra guidance but on case-by-case basis (3) • General advice (2) • Refer to other empty reviews, EPICOT, & extra advice (1) • Group Methodologists give guidance (1) N=53

14. Aim to update or withdraw empty reviews faster than others? For ‘Yes’ responses: (n=4) • No reason given (2) • If there are recently published RCTs (2) N=53

15. Views about priority on updating empty reviews • No, topic-based priority only • “Priority is only given on the basis of importance of the review topic, not on whether there are included studies or not.” • Only if new study emerges • “Aim to update/review them around the same time as other reviews , but if we became aware of a study relevant to an empty review, it would probably prompt a more rapid update.”

16. Thank you! paul.montgomery@spi.ox.ac.uk