Software Testing & Test-Driven Development

1. Software Testing & Test-Driven Development JAMS Workshop Makerere University September 2010

2. Agenda • Intro to Software Testing • Test-Driven Development • Writing a Test Plan • Test Frameworks • JUnit • Visual Studio

3. Software Testing Software testing is the process of verifying that a software program works as expected. • Without testing, there’s no proof that the software does what it is intended to do • Testing == Quality • Testing should be incorporated into the development process from the beginning • The need to test your software will impact how you develop it • The earlier you find a problem, the easier it is to fix Cost of fixing a defect depending on the stage it was found

4. Who is Involved with Testing?

5. Testing Methods • The “box” analogy – describes the point of view the engineer takes when testing White Box Black Box Grey Box

6. Many Levels of Testing

7. Test-Driven Development • A development technique that relies on the concept of writing the test cases before the product code • Validates that spec and requirements are well-understood • Test cases will initially fail • Developer writes code to make them pass • The test is the proof that the code works • Developer must clearly understand user requirements in order to write tests • There should be no functionality in product code that isn’t tested • Encourages simple designs and inspires confidence • “Clean code that works”

8. TDD Workflow Repeat 1. (Re)write a test Test succeeds 2. Run all tests & see if the new one fails 3. Write some code Test(s) fail Test fails 4. Run the automated tests & see them pass • Always start by writing a test • Test must fail because the feature isn’t implemented (if it succeeds, the need for the feature is obviated) • To write a test, you must fully understand the feature’s specification and requirements • Write minimal product code to make the test compile and run • Validates that test harness works • Tests the test itself: make sure it doesn’t pass without new code • Test should fail for the expected reason – ensure that it is testing the right thing • Write some code to make the test pass • May be inelegant or suboptimal; we will improve it later • Make the test pass only, no ‘extra’ untested functionality • If all test cases pass, all tested requirements are met • If they fail, keep iterating… • Assuming the tests are comprehensive, move on to the final stage • Clean up the code as necessary to achieve production quality • Focus on removing duplication, including duplication between test & product code • Re-run the test cases to ensure that refactoring isn’t breaking any functionality 5. Refactor code All tests succeed

9. Writing a Test Plan • Like a functional specification for your test code • Test Plan Template • Test Plan Objectives • Scope • Features to be tested • Features NOT to be tested • Test Strategy • Test Cases • Open Issues

10. Test Plan Objectives & Scope • Scope • Features to be tested • Basic game play • The game alternates between players • Players can place moves • Only legal moves are allowed • Ending the game • The game can declare a winner • The game can declare a tie • Player statistics table (if implemented) • Timer (if implemented) • Machine player (if implemented) • Features NOT to be tested • User interface rendering • Networked play • Similar to functional spec • What are we trying to accomplish with our test cases? • What is in and out of scope for testing? • Test Plan Objectives • [P1] Test functional correctness of the game’s underlying methods using white-box unit testing • [P1] Test that only valid moves are allowed • [P1] Validate that the game can correctly switch between players and declare the end of the game • [P2] Test advanced features, like the statistics table, play timer, and computer player, if implemented • [P3] Performance testing

11. Test Strategy • Describe the testing methodology you plan to use • E.g. white box, black box, grey box, or a combination • Explain which testing frameworks you’ll use, if any • E.g. JUnit, Visual Studio • Will your tests require any sophisticated infrastructure, setup, or tools? • E.g. mock objects, load simulation, test bridges • Test Strategy • We plan to employ mostly white-box unit tests in order to test the application, using Visual Studio’s built-in unit-testing framework. Each of the following major classes will have at least one unit test: • Game • GameBoard • Player • GameStatistics

12. Test Case Detail Feature 2: Validate Basic Moves/Allow Legal Moves Only

13. Testing Frameworks • JUnit is a unit-testing framework for Java • Developed by the same people who pioneered TDD • Uses source code annotations to decorate special methods to be run by the test harness • Integrated with Java IDEs like Eclipse and JCreator • Visual Studio has a test framework for any .NET language • Based on the same ideas as JUnit • Supports unit tests, database unit tests, generic tests, manual tests, load tests, web tests

14. Test Methods import org.junit.Test; public class AdditionTest { private int x = 1; private int y = 1; @Test public void testAdd() { int z = this.x + this.y; assertEquals(2, z); } } • JUnit • Annotate test case methods with @Test • Use methods from org.junit.Assert to check your test conditions or fail the test case • Visual Studio • Annotate test case methods with <TestMethod()> • Use methods from the Assert class to check your test conditions or fail the test case Imports Microsoft.VisualStudio.TestTools.UnitTesting <TestClass()> Public Class AdditionTest Private x As Integer = 1 Private y As Integer = 1 <TestMethod()> Public Sub TestAdd () Dim z As Integer = Me.x + Me.y Assert.AreEqual(2, z) End Sub End Class

15. Initialization & Cleanup Methods @Before protected void setUp() { this.x = 1; this.y = 1; } @After protected void tearDown() { this.x = 0; this.y = 0; } • Common code that runs before/after each test case • … also known as ‘Fixture’ methods • Add a field for each part of the fixture • Annotate a method with Before/TestInitialize and initialize the variables in that method • Annotate a method with After/TestCleanup to clean up before the next test <TestInitialize()> Public Sub SetUp() Me.x = 1 Me.y = 1 End Sub <TestCleanup()> Public Sub TearDown() Me.x = 0 Me.y = 0 End Sub

16. Running Tests in the IDE • Visual Studio & Eclipse both enable running tests inside the IDE • Select specific tests to run or run the whole suite • IDE reports which cases passed/failed • Run selected tests again • Easily switch between test code and product code • Change each as needed and rerun tests • Easy to debug a test case, set breakpoints in test or product code

17. Further Reading • TDD • Test-Driven Development: By Example (Google Books) • testdriven.com • JUnit • JUnit Cookbook • JUnit Javadoc • An early look at JUnit 4

18. appendix

19. Suite Initialization Methods • Similar to setUp and tearDown, but they run before & after the entire test suite • Useful for expensive config operations that don’t need to be run for each unit test • E.g. setting up a DB or network connection, redirecting System.err when testing 3rd-party libraries • Be careful that your unit tests don’t make changes to static state that will impact other unit tests later in the suite // This class tests a lot of error conditions, which Xalanannoyingly logs // to System.err. This hides System.err before the run, restores it after. private PrintStreamsystemErr; @BeforeClass protected void redirectStderr() { systemErr = System.err; // Hold on to the original value System.setErr(new PrintStream(new ByteArrayOutputStream())); } @AfterClass protected void tearDown() { // restore the original value System.setErr(systemErr); }

20. Testing Exceptions • It is easy to test for expected exceptions • Annotate your test with the expected exception • If the exception isn’t thrown (or a different one is), the test will fail • Limitation: if you need to test the exception’s message or other properties, use a different structure: @Test(expected=ArithmeticException.class) public void divideByZero() { int n = 2 / 0; } @Test public void divideByZero() { try { int n = 2 / 0; fail("Divided by zero"); } catch (ArithmeticException(success) { assertNotNull(success.GetMessage()); } }

21. Timed Tests • Simple performance bench-marking: • Network testing: @Test(timeout=500) public void retrieveAllElementsInDocument() { doc.query("//*"); } @Test(timeout=2000) public void remoteBaseRelativeResolutionWithDirectory() throws IOException, ParsingException { builder.build("http://www.ibiblio.org/xml"); }

22. Creating JUnit Tests with Eclipse • Using JUnit with Eclipse is easy • Create new JUnit tests using File -> New -> JUnit • Specify JUnit 4 • Select location, class you want to test, method stubs to create • Then select which methods you want to test, Eclipse will generate test stubs