Technical and Empirical Software Engineering: Bridging the Chasm

1. Joost Schalken Johan Hoorn Henk Koning Hans van Vliet Technical and Empirical Software Engineering:Bridging the Chasm IMSE meeting

2. Outline • Problem background • Why the chasm between technicians and empiricists? • Towards a common goal • Towards a common vocabulary • Conclusions

3. Outline • Problem background • Why the chasm between technicians and empiricists? • Towards a common goal • Towards a common vocabulary • Conclusions

4. Problem backgroundIntroduction At typical IMSE meetings a presentation… • explains details of a single research project • no relations with other research are discussed • no clarification is give how the project helps advance computer science or software engineering (practice) as a whole Within Information Management and Software Engineering: • very diverse research topics (see next slide…) • not a lot of cross-disciplinary research

5. Software asset management Software architecture Software metrics Program transformation Interaction Design Requirements Engineering Object-oriented software development Mental models of users Distributed logic programming Intelligent multimedia Logical disks Problem backgroundIMSE Staff Research Statements Information Management and Software Engineering Section: Scientific Staffurl: http://www.cs.vu.nl/imse/staff-en.html (accessed 9th June, 2004)

6. Problem backgroundWouldn’t it be great if we could… • get inspiration for research from our colleagues? • better understand the research from colleagues? • see how all research projects combined help advance science and help IT practitioners? • prevent reinventing the wheel twice, but instead rely on the existing knowledge of our colleagues? • perform research together with our colleagues, finding new niches on the borders of our disciplines?

7. Problem backgroundPotential improvements Potential for improvement: • more co-operation • more idea sharing • more communication between sub-disciplines • bridge the chasm between engineering oriented andempirically oriented researchers

8. Problem backgroundExamples of Joined Research • Does program transformation help increase the productivity of software maintenance? • How does interface design affect software architecture? • How can we best reverse engineer the architecture of a system to improve human understanding? • What is the impact of object orientation on program transformation?

9. Outline • Problem background • Why the chasm between technicians and empiricists? • Towards a common goal • Towards a common vocabulary • Conclusions

10. My hypothesisLanguage barriers cause other barriers The chasm between technologists & empiricists caused by: • different vocabularies/languages • different research methods • lack of clarity of how we work towards a common goal

11. My hypothesisImproved co-operation by better understanding We hope to bridge the chasm between the researchers by: • identifying our common goals • mutual understanding of our research methods • explaining the differences between technological andempirical research

12. Outline • Problem background • Why the chasm between technicians and empiricists? • Towards a common goal • Towards a common vocabulary • Conclusions

13. Towards common goalsDifference between Engineering and Science • “Scientists extend our knowledge of the laws of nature while engineers apply those laws of nature to build useful artefacts, under a number of constraints.” A. Abran & J.W. Moore (eds.), SWEBOK: Guide to the Software Engineering Body of Knowledge.IEEE Press, 2004. • Scientists want to understand the world. • Engineers want to understand how to improve the world.

14. Towards common goals What sets up apart from Computer Science Software Engineering vs. Computer Science: • [For software engineering] “the content is markedly different from Computer Science.Just as electrical engineering is based upon the science of physics, software engineering should be based, among others, upon computer science.In both cases, though, the emphasis is necessarily different.” A. Abran & J.W. Moore (eds.), SWEBOK: Guide to the Software Engineering Body of Knowledge.IEEE Press, 2004.

15. Towards common goalsDefinition of Software Engineering (I) • The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software;that is, the application of engineering to software. Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries(IEEE Std 610-1990). IEEE Press, 1990.

16. Towards common goalsDefinition of Software Engineering (II) • “Software engineering is often used to describe the activities, methods and tools of large scale software development.” C. Lewerentz & H. Rust, Are software engineers true engineers?.In: Annals of Software Engineering, vol. 10, 2000, pp. 311-328.

17. Towards common goalsDefinition of Software Engineering Research Discover, improve and evaluate… • methods, and • tools … to help practitioners develop large scale software. Software Engineering Research entails both • discovery of new/enhanced methods and tools • evaluation of methods and tools

18. Outline • Problem background • Why the chasm between technicians and empiricists? • Towards a common goal • Towards a common vocabulary • Conclusions

19. Towards a common vocabularyEngineering and Empirical Research Questions • Engineers improve solutions to problems in the real worldTheir core (design) problem is an action problem. • knowledge is needed as far as it contributes to solving problems and understanding the solutions. • Empirical scientists advance understanding of real world phenomena.Their core problem is a knowledge problem. • all knowledge is sought for, regardless of its applicability.

20. Towards a common vocabularyTypical Action and Knowledge Problems • Engineering research questions (action problems) • Can we construct a new tool U? • Can we invent a new method X? • Can we improve method Y? • Can we improve tool Z? • Empirical research questions (knowledge problems) • How much does the efficiency increase when tool X instead of tool Y is used? • Why does method Y help reduce requirements ambiguity? • Under what circumstances does design X improve modifiability?

21. Towards a common vocabularyDifferent Questions different Paradigms • Type of problem (action/knowledge) has influence on the research methods used. • Empirical scientists typically put more emphasis on research methodology up front. • Engineers typically explain their design or solution using a framework or design methodology afterwards. D.L. Parnas & P.C. Clements, A Rational Design Process, How and Why to Fake It.In: IEEE Transactions on Software Engineering, vol 12(2) feb. 1986.

22. action plan torealize a goal action change in situation: effect reason causes control Towards a common vocabularyEngineering Methodology (I): the Action Cycle C. Mitcham, Thinking Through Technology: The Path Between Engineering and Philosophy. University of Chicago Press, 1994.

23. implementation pilot study design realisation/construction test, evaluation, revision implementation Towards a common vocabularyEngineering Methodology (II): OKT T.J. Plomp, Onderwijskundige technologie: Enige Verkenningen.Enschede: Universiteit Twente, 1982. Educational technological model(in Dutch: OnderwijsKundig Technologisch model)

24. evaluation observation induction testing deduction Towards a common vocabularyEmpirical Methodology: The empirical cycle A.D. de Groot,Methodologie:Grondslagen van onderzoek en denken in de gedragswetenschappen.‘s-Gravenhage: Mouton, 1961.

25. Towards a common vocabularyEmpirical Methodology: Some remarks (I) With empirical research attention is paid to: • Exact statement of research goals / hypothesis • Design of the experiments • which subjects receive which treatments? • Operationalization • how do you measure the attributes • Data collection

26. Towards a common vocabularyEmpirical Methodology: Some remarks (II) With empirical research attention is paid to (continued): • Analysis of data • quantitative/statistical methods • qualitative methods • Validity of the experiment • internal validity • construct validity • external validity • statistical validity T.D. Cook & D.T. Campbell, Quasi-Experimentation: Design & Analysis Issues for Field Settings.Chicago: Rand McNally College Publishing Company, July 1979.

27. Outline • Problem background • Why the chasm between technicians and empiricists? • Towards a common goal • Towards a common vocabulary • Conclusions

28. Working together makes sense! • Without new methods and tools from engineering research no empirical software engineering research! • What could be investigated? • Without empirical software engineering research the usefulness of new methods and tools is unclear! • Why would practitioners trust your tool or method that has only been validated on trivial problems?

29. Conclusions • Co-operation can be beneficial • interesting research areas at the borders of sub-disciplines • certain types of research follow others types of research • Co-operation only possible through mutual understanding

30. And now its your turn… • Clarifications • Questions • Comments • Discussion • …