Using Experiments to Teach Software Engineering

1. Using Experiments to Teach Software Engineering Dieter Rombach University of Kaiserslautern Dept. of Computer Science Software Engineering Group Kaiserslautern, Germany (wwwagse.informatik.uni-kl.de) Fraunhofer Institute for Experimental Software Engineering (IESE) Kaiserslautern, Germany (www.iese.fhg.de)

2. Contents • Challenges (teaching software engineering) • Idea(s) • Software Engineering Course Design • Example Experiments • Benefits • Lessons Learned

3. Challenges (teaching software engineering) • Software development is an engineering activity; benefits of engineering principles need to be experienced actively! • Most students in the past did not see relevance of engineering principles to their practical work; after course exam they went back to development business as usual! • Students with prior work experience appreciated software engineering principles better

4. Idea(s) • Include empirical studiesin regular software engineering courses (since 1997) - partial replications of research exp’s! • Focus on basic principles such as • information hiding (to improve modifiability) • tractable documentation (to improve modifiability) • reading of programs (to improve effectiveness of defect finding as compared to testing) • reading of hl documents (to improve early defect detection) • perform empirical (self experience) studies after initial teaching of techniques/methods - max. 3 x 3h sessions per exp! • self-evaluate data and derive ‘good practice’ guidelines

5. Software Engineering Course Design • Two software engineering courses • Course SE 1: Systematic software development (& project) • Course SE 2: Software project planning & management (&..) • SE 1 • overview • foundations of se • project planning & management • software application development • software system development • software component development • software maintenance & evolution Experiment 1 Experiment 2 Experiment 3

6. Example Experiments • Experiment-1: Effectiveness/Efficiency of different requirements reading techniques • goal: to understand that defects can be detected effectively/efficiently, even in informal requirements documents • techniques: ad-hoc, check-lists, pbr • design: each student applies ad-hoc &(check-list or pbr) on 2 different requirements documents (from building automation domain) - about 50 pages each • data collected: number/type of defects found, effort spent • results: more (at least equal # of) defects found (effectiveness increases; efficiency increases) - cap of 3h???

7. Example Experiments • Experiment-2: Understandability/Modifiability of differently structured/documented OO-Systems (guidelines!) • goal: to understand the impact of ‘good’ documentation and structuring guidelines on understandability/modification • techniques: vertically/horizontally consistent documentation vs … & high degree of information hiding/low coupling/… vs ... • design: half the students use ‘good’, half ‘bad’ system (= requirements, design, code) • step 1: answer questions about system • step 2: perform changes • data collected: # correctly answered questions; # correctly made changes; effort involved • results: more questions answered/changes made correctly (better understandability/modifiability, efficiency?)

8. Example Experiments • Experiment-3: Effectiveness/efficiency of different code defect detection techniques • goal: to understand that defects can be found more effectively/efficiently via code reading than testing • techniques: PBR code reading versus structural/functional testing • design: fractional factorial design (see: Selby); each student uses PBR and one testing technique • data collected: number/types of defects found, effort spent • results: more (at least equal # of) defects found (code reading is always more (or at least as) ‘effective’ than (as) other techniques; same is true for ‘efficiency’)

9. Benefits • Students UNDERSTAND software engineering principles (‘they derived them themselves!’) • Students (in their majority) own principles and continue to apply them naturally • Industry appreciates knowledge about key se principles • Students appreciate empirical studies as a means of learning • Students live ‘empirically-based’ improvement in subsequent project courses (Optimization of PBR across courses leads to high defect detection effectiveness - >80%) • It is fun for the teacher (No more ‘trust me, this is important’)

10. Lessons Learned • Participation should be voluntary (typically > 80%, i.e. 40 - 60 graduate students) • Certificate of participation appreciated by students (& industry) • Students expect experiments (!) • Personal data must be owned by individuals • Interpretation should be done (on aggregate level) in class feedback sessions (later complemented by other experimental results!) • Materials should be packaged in reusable form (lab packages require lots of effort!) • Learning effect must over-write research interests (or we pay!) • Participation in publication of results could be offered • ISERN could host database of pre-packaged experiments