Testing

1. Testing Test Plans and Regression Testing

2. Programs need testing! • Writing a program involves more than knowing the syntax and semantics of a language • You need to verify that your program runs correctly, i.e. does not crash unexpectedly and gives the correct answers/outputs. • A simple program may be obviously right or wrong, but once it gets past more than 10 lines, a program needs to be tested systematically. • Putting in some data picked at random will do a little good, but to be confident that your program works, you need to have a plan.

3. A Test Case • A typical test case has four items: • A description of the case – what are you testing for here? • Input data which causes the case to happen • The expected output/outcome from that input • The actual output/outcome from that input • In a software company, the authors of the software would write up test cases with the first three items, then hand off the job to the testers. • The testers would run the tests, record the actual outputs and return the results to the authors, who would fix the errors found.

4. A Test Plan • A test plan is a table which contains a number of test cases. • The quality of the test cases is more important than the quantity. • It is important to give test cases that do not overlap if possible. If your program has two paths through it and all your test cases run through just one of the two possible paths, then you have not thoroughly tested your program. • Making good test plans requires thought and knowledge of the specifications of the problem. Think about normal cases, boundary cases, special cases, error cases.

5. Test Plans for CS 115 • Our test plans will only have three major columns instead of four. The last one, the Actual Output, is assumed to be unnecessary. If you find a case where the expected output and the actual output do not agree, you have found a bug and we assume you will fix it. • I say “major columns” because as the programs get more complex, there may be several different inputs for one run of the program. These would all be under “Input” but would have their own columns

6. Sample Test Plan • Suppose the problem was to input 3 numbers and output ‘yes’ if they were in ascending order and ‘no’ otherwise. “Ascending” means strictly each number is greater than the ones coming before it.

7. “Off by one” errors • If your problem involves ranges of values (numbers from 1 to 10, letters from C to R, etc.) then you should be careful to test the boundary cases. Humans commonly make the error of going one time too far or stopping one time too soon. • In the case of numbers from 1 to 10 (assuming integers), test cases should be done for 0, 1 and 2, and 9, 10, and 11. • This is an example of checking boundary conditions. • Some problems have cases which will not have correct answers or will have unusual behaviors (crashing). These cases should be investigated also, as far as possible.

8. Regression Testing • This is a particular principle of testing that will save you some points! • If you are testing your program with your test plan, you will run through the cases in the order in the table, most likely. • Say you find that Case 1 works, Case 2 works, but Case 3 does NOT. • You naturally will fix the bug found by Case 3. • BUT the natural thing to do after that would be to start with Case 4, assuming that Cases 1 and 2 are still valid from previous runs. • Regression testing says to start OVER with the FIRST test case and run all the test cases again. The reason is that you may have introduced a NEW bug when you fixed the one in Case 3, or you may have uncovered a bug which was already there but hidden by the bug you have now fixed.

9. Regression Testing continued • How does this save you points? • Suppose the situation described does happen • You fix Case 3, but there is a new bug which will only be seen if you run Case 1 again. • You don’t do regression testing, so that bug is not found. • When you turn your program in, the TA runs their test cases on your program. Bang!! The first case does not run correctly!! You lose points! • So you want to do regression testing!