Semi-Automated Design Guidance Enhancer (SADGE)

1. Semi-Automated Design Guidance Enhancer (SADGE) Olaf Zimmermann University of Applied Sciences of Eastern Switzerland Rapperswil, Switzerland ozimmerm@hsr.ch Mohsen Anvaari Norwegian University of Science and Technology Trondheim, Norway mohsena@idi.ntnu.no A Framework for Architectural Guidance Development

2. Agenda • Introduction • An Example Scenario • SADGE Framework Development • SADGE Framework Operation and Maintenance • Framework Evaluation – Setup • Framework Evaluation – Results • Discussion • Related Work • Conclusions • Future Work • Questions Semi-Automated Design Guidance Enhancer (SADGE)

3. Introduction • Architectural decision making is a non-trivial task for architects • Researchers have developed several concepts, methods and tools to assist practitioners • One of the promising approaches is a decision identification technique • It enhances architectural guidance (decisions required) from decisions made previously • Practitioners still are reluctant to apply this technique • SADGE automaticallyextracts architectural issues from architectural related documents by applying natural language processing first • Then, it manuallypost processes and fine-tunes the extracted knowledge entities Semi-Automated Design Guidance Enhancer (SADGE)

4. An Example Scenario • The book is a 500 pages text, and architect “A” has spent 50 hours to manually annotate 200 sentences that contain architectural issues • SADGE assists the architects to automatically annotate the sentences that contain architectural issues

5. SADGE Framework Development Automated Annotator: We used ANNIE and JAPE from an open source tool called GATE (general architecture for text engineering) Semi-Automated Design Guidance Enhancer (SADGE)

6. Annotating Rules and Catalog of Terms High Priority Terms agree on, choose Low Priority Terms approach, articulate, class, component, construct, concern, define, design determine, different, employ, establish, evaluate, exchange, facilitate framework, investigate, limitation, make, philosophy, principle, profile provide, protocol, recommend, refactor, require, schema, select, service several, strategy, support, topology, transaction management, type, various Semi-Automated Design Guidance Enhancer (SADGE)

7. SADGE Framework Operation and Maintenance

8. Framework Evaluation • Purpose: Evaluate SADGE by comparing the three quality attributes (Precision, Recall, Effort) of manual, automatic and semi-automatic approaches of decision identification • Recall: A∩B/B • Precision: A∩B/A

9. Framework Evaluation • Participants: 19 students of a bachelor’s program in information technology • randomly selected, familiar with the software architecture, but not familiar with the concept of architectural knowledge • We divided them into two groups of ten and nine students • Stages: In the first stage, students were supposed to annotate the sentences they think are architectural issues • In the second stage, they were asked to reject the sentences they disagreed with automated annotator to fine-tune the results Semi-Automated Design Guidance Enhancer (SADGE)

10. Framework Evaluation - Material • The document we selected for the experiment was the one that had the smallest deviation from the mean of precision and recall of annotating all of the tested documents and therefore can be considered as a representative of the tested documents • The right figure shows how the output of the experiment can look like Semi-Automated Design Guidance Enhancer (SADGE)

11. Framework Evaluation - Results Semi-Automated Design Guidance Enhancer (SADGE)

12. Discussion • The effort reduction results are in correspondence to our expectation • The recall and precision of student annotations are very low (38 and 25 respectively) • In the industrial projects we expect that both precision and recall should be higher • The results for automated annotator are relatively high (86 and 57) • These show that if the people in charge of enhancing architectural guidance are not expert enough, automated approach will perform more accurate and more sensitive by spending much less time • The other part that is not the same as our expectation is the recall rate of semi-automatic approach. The reason may lie in the expertise level of the participants Semi-Automated Design Guidance Enhancer (SADGE)

13. Related Work • Using NLP for knowledge extraction is not novel in software engineering • Existing tools and methods for generating object oriented models from natural language texts by applying NLP (Perez-Gonzalez, 2002)(Harmain and Gaizauskas, 2003)(Bajwa et al., 2009)(Soeken et al., 2012) • The majority of work has been done to extract the object oriented data from the documents • Extraction of architectural knowledge (specifically architectural decisions) is not mainly in focus • (Figueiredoet al., 2012) approach and TREx (López et al., 2012) are two works applying NLP in AK • The catalog of terms and annotating rules are not presented in the papers nor publicly accessible • It is not possible to replicate the approaches and as a result the comparison is not feasible Semi-Automated Design Guidance Enhancer (SADGE)

14. Conclusions • Applying natural language processing (NLP) techniques, SADGE automatically annotates (highlight) the architectural issues to reduce the knowledge engineering effort • Next, a knowledge engineer manually post processes the outcome of the automated annotation process to increase the accuracy of the results • Our preliminary evaluation shows that when the practitioners are not experienced, the automatic approach has the highest sensitivity and the lowest processing time while the semi-automatic approach has the highest accuracy • SADGE can assist practitioners to identify required architectural decisions more confidently Semi-Automated Design Guidance Enhancer (SADGE)

15. Future Work • Improving the sensitivity and accuracy of automated annotator by applying machine learning algorithms • Increasing the training data by receiving more architectural related documents from real projects in the industry • Evaluating the framework by conducting case studies that involve expert architects and also include project documents Semi-Automated Design Guidance Enhancer (SADGE)

16. References • Perez-Gonzalez, H. G.: Automatically Generating Object Models from Natural Language Analysis, 17th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, ACM New York, USA, pp: 86 – 87 (2002) • Bajwa, Imran Sarwar, Ali Samad, and ShahzadMumtaz. "Object oriented software modeling using NLP based knowledge extraction." European Journal of Scientific Research 35.01: 22-33 (2009) • Harmain, H. M. and Gaizauskas, R.: CM-Builder: A Natural Language-based CASE Tool, Journal of Automated Software Engineering, 10, 2003, pp. 157-181 (2003) • Soeken, M., Wille, R., and Drechsler, R.: Assisted behavior driven development using natural language processing. In Objects, Models, Components, Patterns (pp. 269-287). Springer Berlin Heidelberg (2012) • López, C., Codocedo, V., Astudillo, H., & Cysneiros, L. M. (2012). Bridging the Gap between Software Architecture Rationale Formalisms and Actual Architecture Documents: An Ontology-Driven Approach. Science of Computer Programming, 77(1), 66-80 (2012) • Figueiredo, A. M., dos Reis, J. C., and Rodrigues, M. A.: Improving Access to Software Architecture Knowledge: An Ontology-based Search Approach. International Journal Multimedia and Image Processing (IJMIP), 2(1/2) (2012)

17. Thanks!Questions? Semi-Automated Design Guidance Enhancer (SADGE)