1 / 82

So… You Want to Produce a Literature Review ?

Daniel Amyot damyot@uottawa.ca January 2019. Systematic. So… You Want to Produce a Literature Review ?. Bits of Wisdom. If I have seen a little farther than others, it is because I have stood on the shoulders of giants. Sir Isaac Newton

lynelle
Télécharger la présentation

So… You Want to Produce a Literature Review ?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Daniel Amyotdamyot@uottawa.ca January 2019 Systematic So… You Want to Produce a Literature Review?

  2. Bits of Wisdom If I have seen a little farther than others, it is because I have stood on the shoulders of giants. Sir Isaac Newton Science is supposed to be cumulative, not almost endless duplication of the same kind of things. Richard Hamming, 1968 Turing Award So… You Want to Produce a Literature Review p.2

  3. Why Do a Literature Review? The literature Mandatory part of the program… My supervisor wants me to… Part of the thesis template I got… I want to graduate!!! So… You Want to Produce a Literature Review p.3

  4. Are There Better Reasons? http://www.phdcomics.com/comics/archive.php?comicid=789 So… You Want to Produce a Literature Review p.4

  5. Why Do This Systematically? Better Reasons… Understand the domain and the state of the art Use a recognized methodology (not ad hoc) Avoid missing important (or sometimes better) related work Be evidence-based and avoid bias Synthesis of work and relationships Find gaps/trends/agreements/disagreements in existing work, or realize there are no gaps! Framework to compare/position your own work Support repeatability (rigor) and evolution So… You Want to Produce a Literature Review p.5

  6. Another (More Selfish) Good Reason • Get a publication! • Reviews are usually well cited. Make yours serious enough to be published. • There are just too many papers being published… Help people save time with a good review, with useful insights. • EBC survey publications So… You Want to Produce a Literature Review p.6

  7. Why Systematic? Evidence-Based! • Evidence-based medicine has changed research practices (Cochrane 1972) • Failure to organise existing medical research cost lives • Clinical judgement of experts worse than systematic reviews • Evidence-based paradigm adopted by other disciplines providing service to public • Social policy, Education, Psychiatry… • Sloooowwwlllly getting there in Management, IS, CS, and Software Engineering! • Work of Kitchenham and others So… You Want to Produce a Literature Review p.7

  8. Systematic Reviews 1/2 [Kitchenham] • A systematic literature review (SLR) is • An overview of research studies that uses explicit and reproducible methods • Systematic reviews aim to synthesize existing research • Fairly (without bias) • Rigorously (according to a defined procedure or protocol) • Openly (ensuring that the review procedure is visible to and auditable by other researchers) So… You Want to Produce a Literature Review p.8

  9. Systematic Reviews 2/2 [Kitchenham] • Support the evidence-based paradigm • Start from a well-defined question • Define a repeatable strategy for searching the literature • Critically assess relevant literature • Synthesize answers to research questions • With thanks to Cochrane, since 1972 http://en.wikipedia.org/wiki/Archie_Cochrane So… You Want to Produce a Literature Review p.9

  10. What is So Systematic about This? http://dilbert.com/ So… You Want to Produce a Literature Review p.10

  11. What is So Systematic about This? Key search words and queries Searching method and databases Criteria for including/excluding references Clear structure in appraising the evidence Explicit discussion of limitations (threats to validity) Many guidelines, but there is still room for adaptation to your context and domain So… You Want to Produce a Literature Review p.11

  12. SLR Process AND PROTOCOL So… You Want to Produce a Literature Review p.12

  13. A Systematic Review Process [Kitchenham, 2007] So… You Want to Produce a Literature Review p.13

  14. Research Questions… • In Software Engineering: • Assessing the effect of a technology (causality?) • Assessing the frequency or rate of a project development factor • Identifying cost and risk factors • Identifying the impact of technology on reliability, performance, cost… • Pragmatic, well-focused questions • More general questions for other purposes • Review of research in domain X? So… You Want to Produce a Literature Review p.14

  15. A Good Research Question… • Must: • Be meaningful and important to practitioners as well as researchers • Lead either to changes in current practice/understanding or to increased confidence in current practice/understanding • Identify discrepancies between commonly held beliefs and reality • Important! • See the advice of Robert Feldt • See section 5.3 of Kitchenham’s guide So… You Want to Produce a Literature Review p.15

  16. Question Formats in Health Science • PICO for evidence-based clinical practice (Cochrane Library Tutorial, 2005): • Patient problem or Population, • Intervention, • Comparison (if there is one), and • Outcome(s). • Example: In adults with recurrent furunculosis (skin boils), do prophylactic antibiotics, compared to no treatment, reduce the recurrence rate? So… You Want to Produce a Literature Review p.16

  17. Other Question Formats in Health Science • PEO for qualitative research questions • Population, Exposure, Outcome(s) • SPIDER for qualitative and mixed-methods research • Sample, Phenomenon of Interest, Design, Evaluation, and Research type. • COSMIN for reviews of measurement properties • COnsensus-based Standards for the selection of health status Measurement INstruments) • Source: https://guides.library.vcu.edu/health-sciences-lit-review/question • More info: https://journals.library.ualberta.ca/jchla/index.php/jchla/article/view/22636 So… You Want to Produce a Literature Review p.17

  18. SLR Protocol • Methods to be used for a systematic review • Predefined protocol • Reduces bias… Too easy and tempting to change the research questions to fit search results! • Improves reproducibility; chance for early feedback • Publishable in certain areas. In medicine, a protocol can be evaluated and published, which sometimes guarantees the publication of the research results coming from executing this protocol. • https://www.researchprotocols.org/ • In practice however, for a first review, chances are you will be “cheating”… Iterative approach! So… You Want to Produce a Literature Review p.18

  19. SLR Protocol Content • Content • Background • Research question(s) • Search strategy • Selection (inclusion/exclusion) • Quality assessment criteria • Data extraction • Synthesis • Threats/limitations • Reporting • Schedule/plan • Sample SLR protocol template • Sample protocol So… You Want to Produce a Literature Review p.19

  20. Executing the Review Protocol… So… You Want to Produce a Literature Review p.20

  21. IDENTIFICATION OF Relevant RESEARCH So… You Want to Produce a Literature Review p.21

  22. Pearl Growing • Using 4-5 key paper(s) to find more • Make use of paper’s subject headings (and other index details) in key databases • Also make use of its terminology (for your search strings), the databases in which it is indexed (for your databases), the journal it comes from (for browsing), and so on • Make sure the initial 4-5 papers are covered! So… You Want to Produce a Literature Review p.22

  23. Some Potential Search Databases Scopus (http://www.scopus.com/) Web of Science (https://webofknowledge.com/) PubMed (http://www.ncbi.nlm.nih.gov/pubmed/) Medline (Ovid) (http://www.ovid.com/site/catalog/databases/901.jsp ) IEEE Xplore (http://ieeexplore.ieee.org/) ACM Digital Library (http://dl.acm.org/) SpringerLink (http://www.springerlink.com/) Google Scholar (http://scholar.google.ca/) Even plain Google/Bing, sometimes (especially for commercial products) Many more on Wikipedia and at the uOttawa library In your protocol, choose those that make sense in your context. So… You Want to Produce a Literature Review p.23

  24. Non-Academic Databases • Canadian Periodical Index Quarterly (CPIQ) • National Bureau of Economic Research (NBER) • Organisation for Economic Co-operation and Development (OECD) • … • But in practice: • Make sure selected academic and non-academic (grey) sources are reliable and have sufficient search options to handle the complexity of your queries • See also: Garousi, V. et al. (2019) Guidelines for including the grey literature and conducting multivocal literature reviews in software engineering. Information & Software Technology, 106: 101-121 So… You Want to Produce a Literature Review p.24

  25. Some Important Advice • Be ready to use the Advanced Search options, always • If you are an employee, get your VPN accessif you want to work from home, else use the off-campus library access • For technologies, do not underestimate patents (Google Scholar) • Your university librarian can help! • http://biblio.uottawa.ca/en/research-help/research-guides-and-librarians • See also research guides and relevant databases at the same URL. So… You Want to Produce a Literature Review p.25

  26. Keywords and Queries • Quickly test your queries, especially for scope • Not too broad (scope creep, unfeasible) • Not too narrow (empty net, nothing to learn) • Think of all possible synonyms for a same concept • “literature review” vs “literature survey” • Think of possible spellings • modelling/modeling, behaviour/behavior • Check already retrieved papers for possible keywords and their combinations • Make sure to follow the search engines rules. All engines have different limitations… So… You Want to Produce a Literature Review p.26

  27. Search Operators • Boolean operators: AND, OR, NOT… • Grouping: () • Exact phrase: " " • Truncation: * • Proximity operators: NEAR, FAR, WITHIN, W/n… • Some ensure sequencing: Pre/n, FOLLOWED BY… • Search fields: title, subject, abstract, keywords, full-text • Limiters: date range, publication type (e.g., journal vs book review), general area (e.g., business vs medicine), language, peer-reviewed, study design… So… You Want to Produce a Literature Review p.27

  28. Play a Little, for Scoping and for Understanding Engine Limitations • Interested in a review of "Computer Security"? • Google Scholar returns 541,000 results! (Jan. 2019) • When do you want to finish your thesis?  • "Computer Security" Cloud? • 53,100 results • "Computer Security" Cloud Amazon (since 2018) • 1,050 results • "Computer Security" Cloud Amazon Prolog (since 2018) • 12 results… and many irrelevant. Too restrictive? • "Computer Security" Cloud Amazon (Prolog OR ".NET") (since 2018) • 472 results… but much garbage (net instead of .NET; dot removed!)… • … So… You Want to Produce a Literature Review p.28

  29. Even Advanced Queries Have Limitations (Google Scholar) So… You Want to Produce a Literature Review p.29

  30. Exercise on Google Scholar How would you search this abstract query? (UCM OR "Use Case Map") AND (IP OR "Internet Protocol") Need to split it into multiple queries, and merge results? UCM AND (IP OR "Internet Protocol") "Use Case Map" AND (IP OR "Internet Protocol") Still unreliable! Expand to 4 queries (all combinations), and check with (or use right away) the Advanced Search… Careful… Google Scholar considers AND a term to search! So… You Want to Produce a Literature Review p.30

  31. More Specialized, But Still Limited(SpringerLink) So… You Want to Produce a Literature Review p.31

  32. Getting There (PubMed) So… You Want to Produce a Literature Review p.32

  33. Scopus So… You Want to Produce a Literature Review p.33

  34. Just How Complex Can Queries Get? • Consent/ or "consent (research)"/ • "confidentiality (research)"/ or "Privacy and Confidentiality"/ • HIPAA.mp. • "Health Insurance Portability and Accountability Act"/ • (Opt-in or opt in or opt-out or opt out or non-consent or no consent or full consent or explicit consent).tw. • (waive$ adj3 consent).tw. • (waiver or consent status).tw. • (consent adj3 model$).tw. • (identifiable adj3 (data or information)).tw. • Data Protection Act.tw. • (Health and Social Care Act).tw. • Human Rights Act.tw. • (Caldicott or PIPEDA or Personal Data Protection Directive).tw. • (privacy adj3 act$).tw. • De-identif$.tw. • (Personal information protection and electronic documents act).tw. • (double-cod$ or double cod$ or single-cod$ or single cod$ or Re-identif$ or reidentif$ or deidentif$ or anonymiz$ or anonymis$ or pseudonymiz$ or pseudonymis$ or reconsent$ or anonymity or identifiability).tw. • ((express$ or knowledgeable) adj3 consent).tw. • (data adj3 unlink$).tw. • ((strip$ or remov$ or delet$) adj3 identifier$).tw. • ((linked or linkable or coded) adj3 (information or data)).tw. • ((unidentif$ or non-identif$ or nonidentif$) adj3 (data or information)).tw. • or/1-22 • "bias (research)"/ or nonresponse bias/ or sampling bias/ or selection bias/ or research subject recruitment/ • patient selection/ • sample size/ • "costs and cost analysis"/ or cost-benefit analysis/ • Time Factors/ • ((consent or response or recruitment or participation or refusal$) adj3 rate$).tw. • (bias or biases).mp. • (survey$ adj3 (response$ or participation)).tw. • (Predictor$ adj3 consent).tw. • accrual.tw. • or/24-33 • (audit$ or registr$ or observational or epidemiolog$).tw. • ((health service$ or medical record$) adj3 (research or study or studies)).tw. • research ethics/ or research methodology/ or data collection/ or data collection methods/ or data collection, computer assisted/ or data mining/ or exp observational methods/ or "record review"/ • "quality of health care"/ or "outcomes (health care)"/ or outcome assessment/ or quality assessment/ or nursing audit/ or "process assessment (health care)"/ or program evaluation/ • nutrition assessment/ or exp Medical Records/ or sampling methods/ • CROSS SECTIONAL STUDIES/ or PROSPECTIVE STUDIES/ • epidemiological research/ or exp health services research/ • registries, disease/ or registries, implant/ or registries, organ/ or registries, trauma/ or surveys/ or exp vital statistics/ • "QUALITY OF CARE RESEARCH"/ or Quality Assurance/ or Audit/ or databases/ • or/35-43 • and/23,34,44 • limit 45 to english • limit 46 to yr="1990 - 2007" So… You Want to Produce a Literature Review p.34

  35. On Getting (Only/All) Relevant Items Precision: how many retrieved items are relevant? Recall: how many relevant items were retrieved? See: https://en.wikipedia.org/wiki/Precision_and_recall#Definition_.28classification_context.29 So… You Want to Produce a Literature Review p.35

  36. How to Handle Complex Queries? • Queries might have to be split into multiple sub-queries, and results (list of papers) merged manually • Lists from multiple engines require merging too • This is where a citation manager (e.g., Zotero, Mendeley…) can help • Downloading the papers themselves is also painful…! • Hopefully Covidence can help. So… You Want to Produce a Literature Review p.36

  37. Document your Queries • List your exact queries in your thesis! • Annex or online. Allows repeatability by others • Allows YOU to repeat the queries before submitting your thesis… • As your thesis work spans many years, it is good to refresh your review before submission, just in case! • Different engines have different syntaxes and limitations • Document your abstractqueries in your thesis, as if you were not limited by the intricacies of the concrete search engines • You might want to keep concrete queries for yourself, to run them again later So… You Want to Produce a Literature Review p.37

  38. Beware of Common Mistakes Business Process Business Process Management "Business Process" OR "Business Process Management" So… You Want to Produce a Literature Review p.38

  39. Is This Query Uselessly Complex? ("Business process compliance" OR "Business Process Compliance Management" OR "Regulatory Compliance" OR "Regulatory Compliance Management" OR "Legal Compliance") AND ("Systematic review" OR "Systematic Survey" OR "Literature Review" OR "Literature survey" OR "State-of-the-art") AND ("Assess*" OR "Measure*" OR "Monitor*") So… You Want to Produce a Literature Review p.39

  40. Is This Query Uselessly Complex? ("Business process compliance" OR "Business Process ComplianceManagement" OR "Regulatory Compliance" OR "Regulatory Compliance Management" OR "Legal Compliance") AND ("Systematic review" OR "Systematic Survey" OR "Literature Review" OR "Literature survey" OR "State-of-the-art") AND ("Assess*" OR "Measure*" OR "Monitor*") So… You Want to Produce a Literature Review p.40

  41. Simpler and Better Query How many concrete Google Scholar queries do we need to run here? ("Business process compliance" OR "Regulatory compliance" OR "Legal compliance") AND("Systematic review" OR "Systematic survey" OR "Literature review" OR "Literature survey" OR "State-of-the-art") AND ("Assess*" OR "Measur*" OR "Monitor*") So… You Want to Produce a Literature Review p.41

  42. What Years Should Be Covered? • This depends on the novelty of the subject • In general, do not impose time limits unless there is a very, very good reason to do so • If no previous reviews, go by when the first relevant study was published • No need to go before 1995 for the Java programming language, unless you want papers about that island or about coffee! • Can also be systematic for a recent period, and ad hoc before (to include seminal papers) So… You Want to Produce a Literature Review p.42

  43. Keep a Table of Returned Papers • This is your raw data! • You may take note of different characteristics (columns) • Year published, type (conference/journal/thesis/ patent), country, query that returned it if many, number of Google Scholar and/or Scopus references… • Can include other columns indicating or encoding why each paper was selected/rejected • Can be used for statistics and trends • Can form an (online) Appendix in your thesis or paper • Some tools (Scopus) can export such a table! • Example: https://goo.gl/ZFNfhc So… You Want to Produce a Literature Review p.43

  44. What To Do Once You Have 300 Papers to Read? (1/2) Use specific and explicit inclusion/exclusion criteria Might want to focus on the abstract and perhaps on the introduction/conclusion sections first (filtering) A paper is of low quality or irrelevant or in a language you don’t understand? Exclude! A paper cites another interesting paper not returned by your initial queries? Include! So… You Want to Produce a Literature Review p.44

  45. What To Do Once You Have 300 Papers to Read? (2/2) • Reduce to a manageable number of papers • About 25-40 for a systematic review, but can be a lot more for a systematic map • Document how many (and which) papers got filtered out (or added) at each step • Involve a second person to do the same exercise in parallel and compare (hard and costly!), or minimally to review your decisions (cheaper), in order to avoid bias So… You Want to Produce a Literature Review p.45

  46. Citation Chaining as Inclusion Strategy • Backwards chaining: reference list checking (this paper is referencing older papers) • Forward chaining: citation checking (this paper is referenced by newer papers) • Web of Science • Scopus • Google Scholar

  47. Do Not Limit Yourself to Databases! Brings et al. (2018) reported on three literature review on the same topic using: 1) databases, 2) snowballing, 3) manual approach. The relevant papers were very different and complementary! Snowballing was most effective (number of papers) but database was most efficient (time). Brings, et al. (2018) On Different Search Methods for Systematic Literature Reviews and Maps: Experiences from a Literature Search on Validation and Verification of Emergent Behavior. In 22nd Int. Conf. on Evaluation and Assessment in Software Engineering (EASE'18). ACM, USA, 35-45. DOI:10.1145/3210459.3210463 So… You Want to Produce a Literature Review p.47

  48. When to Stop? • How much time have we got left? Be practical! • Saturation! The search was exhaustive. • Discussion on literature searching (example): • Conventional subject search (30 of 41) • Citation searching (3 + 2 serendipitous) • Reference list checking (4) • Contact with experts (2) • Pearl growing (abandoned)

  49. Summary of Selection Results Many use PRISMA 2009, for reporting on the selection process/results. Flow Diagram (Word version) So… You Want to Produce a Literature Review p.49

  50. ASSESSMENT AND reporting So… You Want to Produce a Literature Review p.50

More Related