Affan Yasin Muhammad Ijlal Hasnain Supervisor : Dr. Richard Torkar Master Thesis Presentation

1. On the Quality of Grey literature and its use in information synthesis during systematic literature reviews Affan Yasin Muhammad Ijlal Hasnain Supervisor: Dr. Richard Torkar Master Thesis Presentation Thesis #: MSE-2012_97

2. Flow of Presentation • Research Questions • Research Methodology (RM) • Review protocol • Results and Analysis • Total Grey Evidence • Google Scholar findings • Strategies to categorize grey Literature • Quality Assessment Checklist for grey literature • Conclusion/Contribution

3. Research Questions • Research Question 1: What is the extent of usage of grey literature in systematic reviews (in SE)? • Research Question 2 : What are the strategies that can be used to categorize grey literature (non-peer reviewed)? • Research Question 3 :How can we Assess the Quality of Grey Literature ? • Research Question 4 : To what extent does Google Scholar bring the same results as we get from commercial research databases such as ACM, IEEE, SpringerLink, and ScienceDirect ?

4. RM: Systematic Literature Review with Meta Analysis • Systematic Literature Review based on Kitchenham Guidelines. • Cohen Kappa Calculation

5. 138 SLRs

6. Results & Analysis Total Studies Analyzed Total SLRs: 138 Total Primary studies: 6307

7. Results & Analysis – IEEE Total Studies Analyzed 1. Total SLRs: 48 2. Total Primary studies: 2018 3. Grey Evidence: 161 (7.98%)

8. Results & Analysis – ACM Total Studies Analyzed 1. Total SLRs: 9 2. Total Primary studies: 240 3. Grey Evidence: 27 (11.25%)

9. Results & Analysis – Science Direct Total Studies Analyzed 1. Total SLRs: 67 2. Total Primary studies: 3573 3. Grey Evidence: 371 (10.38%)

10. Results & Analysis – Springer Link Total Studies Analyzed 1. Total SLRs: 14 2. Total Primary studies: 476 3. Grey Evidence: 23 (4.83%)

11. Total Grey Evidence Total primary studies : 6307 Grey literature studies : 582 (9.22%)

12. Various Indicators - In Thesis • Frequency of GL use. • Frequency of GL Citing. • Intensity of GL Use. • Forms of GL Cited. • Origin of Documents. • Publication Year of GL Documents. Reference of Indicators: The Evaluation of Grey Literature Using Bibliometric Indicators. Rosa Di Cesare, Roberta Ruggieri CNR – IstitutodiRicerchesullaPopolazione e le PoliticheSociali.

13. Google Scholar Results Summary 1. Total Primary studies: 6307 2. Found By GS: 6026 (95.5%) 3. (GS + google combination) > 95.5%

14. Categorization of Grey Literature

15. Application of Strategies

16. Quality Assessment of Grey Literature • Assessment checklist • Assign scores. Yes = 1, Partially = 0.5, No = 0

17. Conclusion/Contribution • Grey evidence in SE SLRs found to be 9.22% • Google Scholar results. ~96 % hit. • Conference proceedings and technical reports are most produced grey literature documents (68%). • Universities, International organizations and research centers are the foremost producers of grey literature. • The grey literature included is concentrated in recent past. ~48% in last 5 years. • Categorization techniques for grey literature. • Quality assessment checklist for grey literature.

18. Future Work • Verify Quality Assessment Checklist. (Controlled Experiment). • Comparison of Quality Assessed Grey studies with commercially published studies. (Difference in Results; if any). • Future Challenges – Its Solutions. Etc

19. Thank you.Questions?