Testing & Software Quality

1. Testing & Software Quality Seminar on software quality 13.5.2005 Karipekka Kaunisto

2. Contents • Role of testing in quality assurance • Challenges of software testing • What is test automation? • Test automation: Possible benefits • Common pitfalls of test automation • Conclusions • References

3. Role of testing in quality assurance • Quality controlling • Final product meets it’s requirements • Find potential errors and flaws • Enforce standards and good design principles • Regression testing • Improving quality • Preventive testing • Find cause of an error not just symtoms

4. Role of testing... • Testing as supportive action • Data collected during testing can be used to develop various quality metrics • These can be used to some extend when evaluating system quality and maturity • However, numbers alone don’t solely assure good quality!

5. Examples of poor testing • A major U.S. retailer was hit with a large govermental fine in October of 2003 due to web site errors that enabled customers to view one anothers' online orders • In early 1999 a major computer game company recalled all copies of a popular new product due to software problems. The company made a public apology for releasing a product before it was ready • A retail store chain filed suit in August of 1997 against a transaction processing system vendor (not a credit card company) due to the software's inability to handle credit cards with year 2000 expiration dates

6. Challenges of software testing • Complexity of testing • Even in a seemingly simple program there can be potentially infinite number of possible input and output permutations to test • What about large software systems with complex control logic and numerous dependencies on other modules and entire systems?

7. Complexity of testing • => It is not feasible to get even close to testing all combinations and thus finding all possible errors! • Tester needs to carefully create test set in a way that minimises risk of fatal errors in final product • Related problem: How do you know when to stop testing? • Acceptable risk level

8. Managing large test sets • Various general techniques have been introduced for managing test sets • Partitioning to smaller subsets • Testing special cases (boundaries, special values etc.) • Testing most important functions only (focus testing) • Invalid inputs and data • Program flow and code coverage testing

9. Are we ready to ship? • Even with all the techniques available it will require tester’s personal expertise and domain knowledge to create test plan and make the final decision to approve the product • Business issues may also affect on this: Risk of errors vs. risk of delay • Plan and test effort correlate quite well to quality controlling role of testing

10. Other challenges • Testing activities require significant amount of time and resources of the project => Delays, hasty testing • Testing is often regarded as dull, monotonous and laborous part of software development => Poor effort • System architecture is often quite complex, which require special testing effort => Reliability suffers, all tests not even possible manually

11. What is test Automation? • ”The management and performance of test activities to include the development and execution of test scripts so as to verify test requirements, using an automated test tool”. – Dustin, Rashka & Paul • ”Testing supported by software tool”. – Faught, Bach

12. Automation in practice • Tester describes the test cases for tool by using special scripting language designed by tool developers • Some tools may also include graphical interface and recording options but in practise scripting has to be used • Script should also specify how tool is supposed to interpret the correct results of any given test case • Tool then takes care of executing specified tests and examining the results

13. Automation in practise (cont.) • Result validation includes text outputs, elapsed time, screen captures etc. • Can be very challenging part to do automatically and may require some human intervention in some cases! • Evaluation results are presented in clear test reports that can be used to examine results of test round • Produced reports can also be used to gather data for various quality metrics

14. Areas of test automation • Automation suits mainly on testing that requires repeated effort of similar tests cases • Regression testing • Portability testing • Performance and stress testing • Configuration testing • Smoke testing • ...

15. Possible benefits • More reliable system • Improved requirements definition • Improved performance (load & stress) testing • Better co-operation with developers • Quality metrics & Test optimisation • Enchanced system development life cycle

16. Benefits (2) • More effective testing process • Improved effort in various sub-areas like regression, smoke, configuration and multi-platform compatibility testing • Ability to reproduce errors • Dull routine tests can be executed without human intervention • ”After-hours testing”

17. More effective... • Execution of tests that are not possible manually • Better focus on more advanced testing issues • Enchanced business expertise

18. Benefits (3) • Reduced test effort and schedule • Initial costs of automation are usually very high • Payback comes later on (possibly quite much later) when team has adopted the process and use of tools

19. Pitfalls of test automation • Automatic test planning and design • There are no tools that can generate these automatically! • Requires human expertise and domain knowledge • Tool just does what it is scripted to do and nothing else

20. Pitfalls (2) • Immeadiate cost and time savings • On the contrary introduction of automation and new tools will increase the need of resources! • Automation process must be planned, test architecture created, tools evaluated, people trained, scripts programmed... • = Lot’s of work

21. Immediate... • Potential savings will be archieved (much) later on when organisation has ’learned’ the process and created needed infrastructure for it • If automation is introduced poorly, savings will never be gained at all! • In the worst case automation can just mess things up

22. Pitfalls (3) • One tool does it all • Wide array of operating systems, hardware and programming languages • Very different systems and architectures are often used • Testing requirements differ depending on system and project • Result analysis differ (graphical, text, time etc.)

23. Pitfalls (4) • Automation requires no technical skill • Tools rely solely on scripts when executing tests • Maintainable and reusable script building requires good programming skills and knowledge of the tool • Testers may have to be able to use several different tools with different scripting technologies!

24. Pitfalls (5) • 100% test automation • Even if automation succeeds it cannot completely replace manual testing • Some tests must be conducted manually and others require at least some human intervention • Automation is really useful only with test cases that are executed repeatedly over time (regression)

25. Other related tools • Code analyzers • Coverage analyzers • Memory analyzers (purifiers) • Web test tools • Test management tools

26. Conclusions • Testing has significant role in software quality assurance • Automation, when implemented properly can further improve test effort and thus lead to improved quality • However many automation attempts have failed because of unrealistic expectations and inproper introduction of automation tools

27. References • Dustin E., Rashka J., Paul J.: Automated Software Testing: Introduction, Management and Performance. Adison Wesley, 1999 • Craig R. and Jaskiel S.: Systematic Software Testing, Artech House Publishing, 2002 • Pettichord Bret, Presentations and Publications. http://www.io.com/~wazmo/papers/