1 / 27

Lessons Learnt While BUILDING Pex

Peli de Halleux Senior Research Software Design Engineer Microsoft Research. Lessons Learnt While BUILDING Pex. What is Pex?. Pex is…. A Unit Test Generation Tool for .NET That Uses Dynamic Symbolic Exception Uses Constraint Solving. Pex is…. Developed at Microsoft Research

nirav
Télécharger la présentation

Lessons Learnt While BUILDING Pex

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Peli de Halleux Senior Research Software Design Engineer Microsoft Research Lessons Learnt While BUILDING Pex

  2. What is Pex?

  3. Pex is… • A Unit Test Generation Tool for .NET • That Uses Dynamic Symbolic Exception • Uses Constraint Solving

  4. Pex is… • Developed at Microsoft Research • 5 years x 2 developers Nikolai Tillmann Peli de Halleux • Z3 Constraint Solver - 5 years x 2 developers Nikolaj Bjorner Leonardo de Moura • 40000 downloads last year, 3000 forum posts, …

  5. Challenges Faced BY PEX(Or ANY OTHER TEST GENERATION TOOL)

  6. DISCLAIMER GUT FEELING AHEAD...

  7. Universal Low of Constraint Solving • “If you cannot collect constraints,you cannot solve them”Famous test generation guru

  8. Closed with Friends Private!!! Easy challenge internal void Test(int i){ if (i == 123) throw new BugException();}

  9. Assertions Deficit Syndrome • “Your tool only finds null references” • “Do you have any assertion?” • “Assertion???”Conversation between a developer and a test generation tool writer

  10. Hidden Complexity • “Why do something in a couple instructions, when you can do it in millions”CPU Spoiled Developer

  11. String Arithmetics Hidden Complexity Hidden Complexity void Sum(int[] numbers) { string sum = “0”; foreach(int number in numbers) { sum = (int.Parse(sum) + number) .ToString() } if (sum == “123”) throw new BugException(); }

  12. XML “Mon Amour” Hidden Complexity Hidden Complexity void DoSomething(string xml) { var doc = new XmlDocument(xml); var node = doc.Select(“//foo[@bar = 5]”); if(node.Value == 123) throw new BugException();}

  13. Technology Soup • “XML, XAML, Linq, Javascript, Aspx, Silverlight, .NET, COM, C/C++, Java, PHP, …….” Pex

  14. The life of a generated test • “What do we do with the generated tests?” A developer in front of 100 generated tests

  15. Test Poetry • “Your tool generated Foo001. I don’t like it.” • “What did you expect?” • “Foo_Should_Fail_When_The_Bar_Is_Negative.” Conversation between a developer and a test generation tool writer

  16. Test Poetry II • “Your tool generated “\0”” • “What did you expect?” • “”Marc”.” Conversation between a developer and a test generation tool writer

  17. Dangers of a Great Demo • “Your tool is Magic!” A developer wowed by a cool demo

  18. Toxic Code Challenge • “New Testing Tools are always evaluated against toxic code”

  19. Last Week’s Bug Challenge • “New Testing Tools are evaluated against previously found bugs”

  20. Unit Testing Utopia • “I do not need test generation, I already practice Unit Testing (and/or TDD)” A Unit Testing Enthusiast

  21. Test Driven Development Dogma • “Test Generation does not fit into the TDD process” TDD Convert

  22. Dr Unit vsMr Integration • “Most Unit Tests are Integration Tests in disguise” Testing tool writer gut feeling

  23. Isolate First Development • “Test Generation cannot work without Isolation”Peli’s gut feeling

  24. Welded Coding Hard coded dependency void ReadFile(string fileName) { if(!File.Exists(fileName)) throw new FileNotFoundException(); ... ... ... }

  25. The Change Function Find a project with lots of pain • Change = Pip Coburn Make it as easy as possible to get started

  26. The Chicken and The Egg • “Who is using your tool?” • “Do you want to be the first?” • “I love your tool but no.”

  27. Conclusions • Test Generation one piece of the puzzle • Isolation is key • Testing is an “integrated experience”

More Related