Combining Bibliometric and Knowledge Elicitation Techniques to Map a Knowledge Domain

1. Combining Bibliometric and Knowledge Elicitation Techniques to Map a Knowledge Domain Katherine W. McCain*, June M. Verner, Gregory W. Hislop, William Evanco, & Vera Cole. College of Information Science & Technology Drexel University

2. KATE'S PHILADELPHIA BRAND BIBLIOMETRICS

3. PHILADELPHIA brand Bibliometrics • Organizations • ISI: Gene Garfield, Henry Small • Drexel: Belver Griffith, Howard White, Chaomei Chen, Xia Lin, Carl Drott, Jackie Mancall, and a host of grad students • Center for Research Planning: Dick Klavans, Len Simon • Major themes: • citation analysis/core literatures; • aging of scholarly literatures; • single period and longitudinal studies of scholarly literatures and fields; • real-time, on-the-fly mapping of literatures, fields, paradigm shifts, vocabulary structures, etc.; • bibliometric applications in collection management, competitive intelligence, institutional evaluation, etc.

4. AGENDA • Introduction: Domain analysis & software engineering • Mapping methods: • Author Cocitation Analysis • Knowledge Elicitation – card sorting • Results • ACA clusters & map • PFNet author network • Card sorting clusters & map • Comparisons of ACA and KE results • Conclusions

5. DOMAIN ANALYSIS SYSTEMS ANALYSIS: the task of identifying the operations and objects needed to specify information processing in a particular application domain INFORMATION SCIENCE: the study of the field (knowledge domain) as a thought or discourse community. It focuses on such topics as knowledge organization, structure, cooperation patterns, language and communication forms, information systems, and relevance criteria as a way of understanding these communities(Hjørland, B., & Albrechtsen, H. (1995)

6. An Aside On DISCOURSE COMMUNITY A group (likely to be geographically dispersed) who share: • a common public goal or goals • a body of specialized knowledge • mechanisms of intercommunication and participation • a genre (e.g. scholarly journal) • a specialized vocabulary Adapted from John Swales, Genre Analysis (1990 Cambridge)

7. SOFTWARE ENGINEERING • The establishment and use of sound engineering principles in order to obtain economically software that is reliable and works efficiently on real machines. • the technological and managerial discipline concerned with systematic production and maintenance of software products that are developed and modified on time and within cost estimates

8. DOMAIN ANALYSIS OF SOFTWARE ENGINEERING • a study of the journal literature of software engineering, based on both author referencing patterns and index term assignments • a study of the factors that affect the “visibility” of software engineering authors • an INSPEC-based co-descriptor mapping of software engineering • a conjoint study of the intellectual and cognitive structure of software engineering • Citation content analysis of Brooks’ Mythical Man-Month

9. TWO APPROACHES TO MAPPING SE • BIBLIOMETRICS: Cocited author mapping uses the patterns of co-occurrence of authors’ names in reference lists to examine the intellectual structure of scholarly literatures and, by extension, the fields that produce those literatures • KNOWLEDGE ELICITATION: the process of collecting from a human source of knowledge, information that is thought to be relevant to that knowledge. [Cooke] • Card sorting: structural analysis of mental models elicited via sorting named cards into piles

10. AUTHOR COCITATION ANALYSIS • AUTHOR SELECTION: authors highly cited in texts and in the core SE literature = 60 authors selected for study • COCITATION DATA GATHERED: cocitation counts retrieved from SCISEARCH, 1990 – 1997 • ANALYSIS: • Raw cocitation counts -- PFNets • Correlation matrix – cluster analysis & multidimensional scaling

11. 60 AUTHORS Abdel-Hamid, Tarek K. Albrecht, Allan J. Basili, Victor R. Beizer, Boris Biggerstaff, Ted J. Boehm, Barry W. Booch, Grady Brooks, Frederick P., Jr. Card, David N. Clarke, Lori A. Coad, Peter Curtis, Bill David, Allan M. DeMarco, Tom Dijkstra, Edsger W. Kaiser, G. E. Kemerer, C. F. Kernighan, Brian W. Kitchenham, Barbara A. Lehnman, M. M. McCabe, Thomas J. Meyer, Bertrand Mills, Harlan D. Musa, John D. Myers, Glenford J. Parnas, David L. Pfleeger, Shari L. Pressman, Roger S. Prieto-Diaz, R. Ramamoorthy, C. V. Rombach, H. D. Rumbaugh, James Selby, R. W. Shaw, Mary Shepperd, M. Shneiderman, Ben Sommerville, Ian Tichy, W. F. Tracz, Will Wasserman, A. I. Weiser, M. Weyuker, Elaine J. Wing, Jeanette, M. Yourdon, Edward Zave, Pamela Fagan, M. E. Fenton, Norman E. Garlan, David Ghezzi, Carlo Gilb, Tom Glass, Robert L. Goldberg, Adele Gomaa, Hassan Grady, Robert B. Harrison, W. Hoare, C.A.R Humphrey, Watts S. Jackson, Michael A. Jacobson, Ivar Jones, T. Capers

12. Data Gathering for ACA

13. Analytical Tools for Raw Cocitation counts

14. Analytical Tools for Proximity Matrix

15. ACA ANALYSES • Raw Cocitation Matrix • PFNet: links nodes (authors) based on their single highest co-occurrence counts. The result is generally a network structure with some authors appearing as major foci (many links to others) representing specialties • Correlation Matrix • Hierarchical cluster analysis: 8 cluster solution identifies major subject clusters • Multidimensional scaling: 2 dimensional map shows overall structure and major themes

16. Knowledge Elicitation Methods • Interviews and observation • Process tracing (e.g. protocol analysis) • Conceptual techniques Card sorting is a conceptual technique that can be done alone or combined with semi-structured interviews.

17. Card Sorting • Software engineers contacted via e-mail, invited to participate in study • Task: sort cards bearing authors’ names into piles, label piles, complete short questionnaire • As many piles as desired • Piles with single authors • Pile of “don’t know” or “aren’t software engineers • 46 respondents participated in postal mail study (a few interviews)

18. Card Sorting Procedure

19. CARD SORTING ANALYSES(correlation matrix) • Hierarchical cluster analysis—8 cluster level • Multidimensional scaling – 2 dimensional map

23. Comparisons: ACA and KE • Cluster similarity – most authors in similar clusters in terms of membership. Some differences in labeling • There are differences between the way authors’ works are cited and the way the authors are perceived in terms of labels (known for textbook writing, cited for specific textbook content)

27. Comparisons: ACA and KE • Map similarity – similar distribution of authors and clusters along X-axis (r=0.73) but not along Y-axis (r=-0.08) • The most important structural theme in Software Engineering, the “micro  macro” dimension, exists in both citation patterns and in perceptions of the field by citing authors. Along the Y-axis, citing patterns focus on the content of authors’ work while general perceptions include more aspects of the authors’ personae.

28. Conclusions • Boehm, Basili, Booch, and Hoare are central figures in the Software Engineering R&D literature; we can identify other authors as probable linkers between research specialties. • The main organizing principle in SE is a continuum of activities related to the process of software design, development, and evaluation. • Key specialties in Software Engineering (in the decade of the 1990s) included Object-Oriented Programming, Analysis & Design, Formal Methods, Software Reuse, Software Testing & Reliability, Software Process Management, and Software Metrics.

29. Conclusions • ACA (mapping, PFNets) and KE (cardsorting) provide complementary views of software engineering. KE methods increase our understanding of the domain by capturing subjects’ mental models of the domain and providing additional information about mapped entities • ACA and KE provide useful cross-validation. The structure of the literature as seen through networks of author indebtedness (citation of previous work) is a good reflection of their mental models of the field, the place of the (cited) authors, and the relationships among their contributions