1. Teaching and Training Developer-Testing Techniques and Tool Support • Tao Xie (North Carolina State University) • Peli de Halleux , Nikolai Tillmann, Wolfram Schulte (Microsoft Research)

2. The Recipe of Unit Testing • Three essential ingredients (e.g., testing List.Add(int)): • Data • Method Sequence • Assertions void TestAdd() { int item = 3; var list = new List(); list.Add(item); Assert.AreEqual(1, list.Count); }

3. Parameterized Unit Testing [Tillmann&Schulte ESEC/FSE 05] • Parameterized Unit Test = Unit Test with Parameters • Separation of concerns • Data is generated by a tool (e.g., Pex) • Developer can focus on functional (algebraic) spec void TestAdd(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

4. Test Generalization void TestAdd() { int item = 3; var list = new List(); list.Add(item); Assert.AreEqual(1, list.Count); } Required skill set Abstraction Generalization, Experimentation, Comprehension, … void TestAdd(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }

5. Teaching in University Settings • Graduate software testing course (CSC 712) at North Carolina State University • 2008, 2009, 2010 Fall semesters (20, 18, and 27 students) • Lectures: two 75-minute lectures per week • Mons: foundation/coverage criteria instructed based on “Introduction to Software Testing” by Ammann and Offutt • Weds: practical/hands-on testing techniques/tools, with Pex, Code Contracts, and Nmodel • Assignments/projects • Test generalization + writing new PUTs http://research.csc.ncsu.edu/ase/courses/csc712/

6. Tools/Technologies used in Project Using industrial-strength tools and technologies (e.g., Pex) • reduce the “debugging” overhead imposed on both the students and teaching staff • give students experiences that they could immediately benefit from when they took on their industrial jobs Over using academic research prototypes • lack support for dealing with various types of real-world code features • encounter many faults in prototypes • lack timely technical support or fixing of reported faults

7. Training in Industrial Settings • One/half-day tutorials (10-25 practitioners) • inside MS, e.g., internal training of MS developers • outside MS, e.g., invited tutorials at .NET user groups • Inside Microsoft (in a training lab at Microsoft) • Attendees using lab computers installed w/ Pex • Slides + demos + intensive hands-on exercises • Outside Microsoft (in a meeting room) • Attendees often not having computers installed w/ Pex • Slides + demos http://research.microsoft.com/pex/documentation.aspx

8. Lessons Learned from Training in Industrial Settings • Set realistic expectations right away • Tools/technologies have limitations, not “panacea” • Need teach developers about limitations • Don’t try to change deeply ingrained beliefs • … at least not all at once • Example: Believers in TDD (Test-Driven-Development) • Emphasize how new techniques relates to existing ones, how they complement

9. Comparison btw University and Industrial Settings • Trainees with different incentives of learning (inducing different emphasis on technology adoption) • Univ: earn good course grades, learning various skills • Industry: “come and watch”, learning what is going on • Training with different durations • Univ: one semester • Industry: full/half day

10. Pex for Fun (Web-based Learning) http://pexforfun.com/ http://pexforfun.com/TheSocialClassroom

11. Pex for Fun – Takeaway • Accessibility • Visual Studio vs. Browser • Active learning • Doing with feedback • Contributing back • Social aspects • Ranking • Livefeed • Competitive environment

12. PexVisual Studio Power Tool • http://msdn.microsoft.com/en-us/vstudio/bb980963.aspx

13. Thank you http://research.microsoft.com/pex http://codeplex.com/Pex https://sites.google.com/site/asergrp/

14. Thank you http://research.microsoft.com/pex http://codeplex.com/Pex https://sites.google.com/site/asergrp/

15. Desirable Project Characteristics • Testing real-world code (not just “toy” code examples) • Different projects for different teams • Allow learning from each other in real-world complications • Effort focused on testing, not code development • Fully exploit limited course period time • Student team selects a real open-source project to test • Equipped with existing conventional unit tests • Project milestones • Midterm: test generalization; final: new PUT writing • Drawback • Not “real” develop testing!