Isolated Parameterized Unit Testing with Pex and Moles

1. Isolated Parameterized Unit Testingwith Pex and Moles • Nikolai Tillmann, Jonathan “Peli” de Halleux • Microsoft Research

2. Learning objectives After I attend this class I will be able to... • Write Unit Tests • Coverage, assertions, isolation • Use Moles to Isolate Unit Tests • Test legacy code • Write Pex Parameterized Unit Tests • Achieve high code coverage

3. Preparation • We will use Pex for all exercises • Pex includes Moles • Visual Studio 2010 Power Tools • http://research.microsoft.com/Pex • http://www.pexforfun.com • Works with .NET 2, 3.5, 4, x86 and x64 • Visual Studio 2008, 2010, Command line • (Alpha) Silverlight support

4. Preparation • Install latest publicversion pex.powertool.x86.msi

5. Unit Testing

6. Quiz: Unit testing • What is a unit test?

7. Unit Testing • A unit test is a small program with assertions • Test a single (small) unit of code void AddAndCount() { // Arrange int item = 3; // Act var list = new List(); list.Add(item); // Assert Assert.AreEqual(1, list.Count); }

8. static string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach(var line in lines){ intindex = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index); return value; } } return null; } t:\myapp.ini A=B Foo=C C=D

9. Quiz: Code Coverage • How much block coverage do we need? • 50% • 80% • 100% • Block coverage alone is not enough

10. Quiz: Coverage • How much block coverage do we need? • 50% • 80% • 100% • Block coverage alone is not enough • Research: no correlation between high code coverage and quality

11. Quiz: White box testing [TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue();} • Do we need more tests to get 100% cov.? [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini", new string[0]); Reader.ReadFooValue();}

12. Quiz: Assertions • Why write Assertions (or not)? • Documentation • Double check your program • Please your manager • Prevent future bugs • Validate user inputs • Catch errors early

13. Quiz: Assertions • Why write Assertions (or not)? • Documentation • Double check your program • Please your manager • Prevent future bugs • Validate user inputs • Catch errors early

14. Quiz: Assertions • Which Assertions should you write? • Assert.IsTrue(value == “b”); • Assert.IsTrue(value == null); • Assert.IsTrue(String.IsNullOrEmpty(value)) • Assert.IsTrue(true); • No assertions [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

15. Quiz: Assertions • Which Assertions should you write? • Assert.IsTrue(value == “b”); • Assert.IsTrue(value == null); • Assert.IsTrue(String.IsNullOrEmpty(value)) • Assert.IsTrue(true); • No assertions [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

16. Quiz: Coverage +Assertions • What gives you confidence in the code? • High coverage, few assertions • Low coverage, many assertions • High coverage, many assertions • Low coverage, no assertions • I wrote it

17. Quiz: Coverage +Assertions • What gives you confidence in the code? • High coverage, few assertions • Low coverage, many assertions • High coverage, many assertions • Research: Experienced developers write good assertions, junior developers write ‘debugging’ assertions • Low coverage, no assertions • I wrote it

18. string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach(var line in lines){ intindex = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; } t:\myapp.ini A=B Foo=C C=D

19. Quiz: Isolation string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... • In the example, what are the external dependencies? • Network Share • Local Disk • No file system, all in memory

20. Quiz: Isolation string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... • In the example, what are the external dependencies? • Network Share • Local Disk • No file system, all in memory

21. Quiz: Isolation • What is the problem with a Local Disk? • Mapping already exists • Cannot run tests concurrently • Disk full • Access rights

22. Quiz: Isolation • What is the problem with a Local Disk? • Mapping already exists • Cannot run tests concurrently • Disk full • Access rights

23. Unit TestingExercise • Map local directory:> mkdir c:\foo> net use t: \\[machinename]\c$\foo • Create C# class library, copy in Reader snippet • Create test project, write unit tests • Run unit tests • Optional: Measure code coverage

24. Definition of Unit Test • What is a good Unit Test? • A Unit Test is a program that runs fast the code under test, without environment dependencies, withassertions • What is a good Unit Test Suite? • A set of Unit Tests which achieves high code coverage

25. 10 Minutes Break

26. Isolation withMoles

27. Dependency hell • Code under test should not depend on hard-coded environment dependencies: • How do you mitigate the Local Disk issues? • Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration • Refactoring: use Streams • Refactoring: introduce IFileSystem • Refactoring: pass the lines as parameter • Change implementation of File.ReadAllLines var lines = File.ReadAllLines(@"t:\myapp.ini");

28. Dependency hell • Code under test should not depend on hard-coded environment dependencies: • How do you mitigate the Local Disk issues? • Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration • Refactoring: use Streams • Refactoring: introduce IFileSystem • Refactoring: pass the lines as parameter • Change implementation of File.ReadAllLines var lines = File.ReadAllLines(@"t:\myapp.ini"); Reality check Refactoring not always an option

29. Dependency hell • Code under test should not depend on hard-coded environment dependencies: • How do you mitigate the Local Disk issues? • Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration • Refactoring: use Streams • Refactoring: introduce IFileSystem • Refactoring: pass the lines as parameter • Change implementation of File.ReadAllLines var lines = File.ReadAllLines(@"t:\myapp.ini");

30. Dependency hell • Code under test should not depend on hard-coded environment dependencies: • How do you changeFile.ReadAllLines? • Override static method • Changing the CLR (and recompiling it) • Rewrite application in JScript • Code instrumentation var lines = File.ReadAllLines(@"t:\myapp.ini");

31. Dependency hell • Code under test should not depend on hard-coded environment dependencies: • How do you changeFile.ReadAllLines? • Override static method • Changing the CLR (and recompiling it) • Rewrite application in JScript • Code instrumentation – the Moles framework var lines = File.ReadAllLines(@"t:\myapp.ini");

32. Motivation for Moles • Why another isolation framework? • Specifically designed to enable Pex • Simple, Well-defined semantics • “Replace any .NET method” • Type safe

33. Moles = Replace any .NET with a delegate var lines = File.ReadAllLines(@"t:\myapp.ini"); What if we could replace File.ReadAllLines? File.ReadAllLines = delegate(string fn) MFile.ReadAllLinesString= delegate(string fn) { return new string[0]; }; Moles

34. Mole Types Code Generation // System.IO public static class File{ public static string[] ReadAllLines(string fn);} // System.IO.Moles public class MFile{ public static Func<string, string[]> ReadAllLinesString { set; } } // delegate R Func<T, R>(T t);

35. Injecting Detours at Runtime // System.IO public static class File { public static string[] ReadAllLines(string fn) { if (MFile.ReadAllLinesString != null) return MFile.ReadAllLines(fn); … original code } } Automatically injected at runtime

36. Demo

37. Quiz: Func<T> • Match the delegates with the methods? • Func<string> • Action • Action<string> • Func<bool,string> • Func<string, bool> • Action<int> • Action<List<T>, int> • Func<string,string[]> • boolFile.Exists(string) • Console.WriteLine(string) • void Flush() • String.Empty {get;} • List<T>.Capacity {set;} • string[] File.ReadAllLines(string)

38. Quiz: Func<T> • Match the delegates with the methods? • Func<string> • Action • Action<string> • Func<bool,string> • Func<string, bool> • Action<int> • Action<List<T>, int> • Func<string,string[]> • boolFile.Exists(string) • Console.WriteLine(string) • void Flush() • String.Empty {get;} • List<T>.Capacity {set;} • string[] File.ReadAllLines(string)

39. C# 3.0 Lambdas MFile.ReadAllLinesString = delegate(string fileName) { return new string[]{“a=b”}; }

40. C# 3.0 Lambdas MFile.ReadAllLinesString = (fileName) => { return new string[]{“a=b”}; }

41. C# 3.0 Lambdas MFile.ReadAllLinesString = (fileName) => new string[]{“a=b”};

42. C# 3.0 Lambdas MFile.ReadAllLinesString = fileName => new string[]{“a=b”};

43. Quiz: Lambdas • Match the Lambdas with the methods • () => “” • () => {} • s => {} • (s) => “” • (s) => false • boolFile.Exists(string) • Console.WriteLine(string) • void Flush(); • String.Empty {get;} • string ToString();

44. Quiz: Lambdas • Match the Lambdas with the methods • () => “” • () => {} • s => {} • (s) => “” • (s) => false • boolFile.Exists(string) • Console.WriteLine(string) • void Flush(); • String.Empty {get;} • string ToString();

45. MolesExercise I • Add moles for mscorlibto the test project • Add new Item • Moles and Stubs for Testing • mscorlib.moles • Write test using moles • Run test • Debug test

46. Exercise II [TestMethod, ...] void ReadFooValueTest() { MFile.BehavedAsNotImplemented(); ... static string ReadFooValue() { if (!File.Exists(@"t:\myapp.ini")) return null; string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ... Exercise II

47. Constructors and Instance methods static string ReadFooValue() { var reader = new StreamReader(“t:\myapp.ini”); string content = reader.ReadToEnd(); void ReadFooValueTest(string content) { MStreamReader.ConstructorString = delegate(StreamReader me, string file) => { var mole = new MStreamReader(me); mole.ReadToEnd= () => content; };

48. Exercise II static string ReadFooValue() { var reader = new StreamReader(@"t:\myapp.ini"))var lines = reader.ReadToEnd().Split(‘\n’); foreach(var line in lines){ intindex = line.IndexOf('='); string name = line.Substring(0, index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null; }} Exercise III

49. Quiz: Moles Usage • When should you use Moles (and not)? • Always use Moles to solve isolation issues • With Moles, one does not need to use interfaces anymore • Moles only should be used for 3rd party API, use interfaces for isolation in your APIs • Moles can be used in production code • Moles lets you get away with untestable APIs • Moles make test cases more robust • With Moles, you do not need integration tests anymore • Moles make tests easier to understand • Moles is for poor programmers, real programmers rely on interfaces

50. Quiz: Moles Usage • When should you use Moles (and not)? • Always use Moles to solve isolation issues • With Moles, one does not need to use interfaces anymore • Moles only should be used for 3rd party API, use interfaces for isolation in your APIs • Moles can be used in production code • Moles lets you get away with untestable APIs • Moles make test cases more robust • With Moles, you do not need integration tests anymore • Moles make tests easier to understand • Moles is for poor programmers, real programmers rely on interfaces