Searching & screening e fficiency : emerging techniques for systematic review

1. Searching & screening efficiency: emerging techniques for systematic review Alison Weightman Co-convenor Cochrane Information Retrieval Methods Group

2. This presentation • Searching for studies • Minimise the number of databases searched? • Maximise the value of search terms → text mining? • Screening studies • Boost relevant records to the top of the list → text mining? • Use two screens Emphasis on text mining for search development …. and on free software

3. Acknowledgement:Selected text analysis tools Julie Glanville Associate Director, York Health Economics Consortium, University of York, UK

4. Stages of a systematic review Step 1: Develop a protocol Step 2: Search for the evidence Step 3: Screen and select studies Step 4: Assess the evidence Step 5: Extract the data Step 6: Synthesise the evidence Step 7: Present findings

5. Step 2: Search for the evidence – traditional steps Manual search of……. • Multiple databases with lots of search terms many 000s hits Plus supplementary searching • Web sites • Reference lists • Citation tracking • Expert contact

6. Step 2: Search for the evidence – new approaches • 2.1 Minimise the number of databases.... analyse previous reviews? • 2.2 Maximise the value of search terms.... use text mining?

7. 2.1 Minimise the number of databases? Morgan EBLIP 2015….. Searches for public health intervention reviews should include Medline/PubMed,reference list checking and citation tracking along with a small number of specially selected additional databases and expert contact/specialist web site searching

8. 2.2 Maximise the value of search terms? ..consider text mining Deriving high-quality (useful) information from unstructured text. The text words in relevant publications are grouped or structured to find word combinations that can be used to locate other similar materials

9. 2.2 Text mining to maximise the value of search terms O’Mara-Eves 2014: ‘We highly recommend that reviewers use automatic term extraction to identify additional search terms, particularly where the topic is broad or multidisciplinary’

10. 2.2 Text mining for search development: A worked example A systematic review of musculoskeletal care pathways for adults with hip and knee pain at the interface between primary and secondary care Three main concepts for search strategy: Musculoskeletal + pathway + primary-secondary interface

11. Some text analysis tools • Most common text words • PubMed PubReMiner, GoPubMed • Endnote • Most common phrases • Termine • Most common indexer-defined subject headings (MeSH) • MeSHonDemand • GoPubMed • Yale MeSH analyser

12. PubMed PubReMiner • http://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi • Example topic • Musculoskeletal primary care referral: Musculoskeletal + care pathway + primary-secondary12 relevant papers. PubMed identifiers23561693, 25442485, 25242531, 21056164, 25410703, 15576743, 10848384, 24337981, 25342909 , 22716771, 23750266, 22711378

13. Musculoskeletal, care (text) Musculoskeletal diseases, referral and consultation (MeSH)

15. Terminecombined with visual scan of abstracts • http://www.nactem.ac.uk/software/termine/ • Example topic Musculoskeletal primary care referral: Musculoskeletal + care pathway + primary-secondary12 relevant papers. Abstracts Enter all 12 abstracts in fullBACKGROUND: Musculoskeletal condition assessment and management is increasingly delivered at the primary to secondary care interface, by inter-disciplinary triage and treat services.OBJECTIVES: This review aimed to describe Intermediate Care pathways, evaluate effectiveness, describe outcomes and identify gaps in the evidence……………

18. Yale MeSH analyser • http://mesh.med.yale.edu/ • Example topic Musculoskeletal primary care referral: Musculoskeletal + care pathway + primary-secondary12 relevant papers. PubMed identifiers23561693, 25442485, 25242531, 21056164, 25410703, 15576743, 10848384, 24337981, 25342909 , 22716771, 23750266, 22711378

19. Critical pathways (MeSH) Musculoskeletal pain (MeSH)

21. OR AND OR AND OR

22. Search efficiency • 256 hits in Ovid Medline • Circa 1/5 hits potentially relevant (select for full text) • Finds >90 % relevant papers (development set) • Finds >70% papers (test set) • 50% of test set were not in Medline • Additional databases (including physio) and extensive supplementary searching required • Then further test for sensitivity

23. Stages of a systematic review Step 1: Develop a protocol Step 2: Search for the evidence Step 3: Screen and select studies Step 4: Assess the evidence Step 5: Extract the data Step 6: Synthesise the evidence Step 7: Present findings

24. Step 3: Screen studies – traditional steps Manual human process involving… • Knowledge of inclusion criteria • Identified in protocol • Selection • Title and abstract • Full text • Screen and select independently in duplicate • Reduction of error and bias

25. Step 3: Screen and select studies – new approaches • 3.1 Boost relevant records to the top of the list…. Text mining? • 3.2 Reduce screening time ….. Two screens? http://www.cs.manchester.ac.uk/our-research/research-groups/text-mining/

26. 3.1 Faster screening: EPPIReviewer …Free soon to Cochrane review authors Weightman 2014…if the reviewer is willing to accept that some titles/abstracts may not be picked up but that all full text includes will be found, benefits may be huge and could reduce the screening burden by up to 70%. An enormous efficiency saving where a large number of results need to be screened.

27. 3.1 Faster screening: Abstrackr http://abstrackr.cebm.brown.edu/ Rathbone 2015….Abstrackr has the potential to save time and reduce research waste. [Of four reviews one relevant citation was missed in each of two reviews]

28. 3.2 Reduce screening time ….. Two screens? http://northerncomputer.blogspot.co.uk/2011/11/many-people-spend-good-part-of-their.html Wang 2014…..Using dual monitors when conducting SRs is associated with a significant reduction of time spent on data extraction (~24 minutes per article). No significant difference was observed in time spent on abstract screening or full text screening

29. Summary: • Choice of databases: Carefully chosen databases – and emphasise the additional searching techniques (reference lists, citation tracking, expert contact…) • Text mining for searching and screening: Wide choice of free software • Two screens: No need for screening but yes for data extraction….. I use for both (ed!)

30. Any questions or suggestions WeightmanAL@cardiff.ac.uk

31. References • Morgan HE et al. Systematic reviews of public health interventions to support practice & policy: Where should you look? [Poster] Eight International Evidence Based Library and Information Practice Conference (EBLIP8), Brisbane Australia, July 2015 • O’Mara-Eves A et al. Techniques for identifying cross-disciplinary and ‘hard-to-detect’ evidence for systematic review. Research Synthesis Methods 2014; 5(1): 50-59 • Weightman A et al. Text mining for screening efficiency? Testing within a Cochrane public health review. [Poster] 23rd Cochrane Colloquium, Hyderabad India, September 2014 • Rathbone J et al. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Systematic Reviews 2015; 4: 80 • Wang Z et al. Dual computer monitors to increase efficiency of conducting systematic reviews. Journal of Clinical Epidemiology 2014; 67(12): 1353-1357