Software testing techniques Automated software testing

1. Software testing techniquesAutomated software testing Presentation on the seminar Kaunas University of Technology

2. Topics 2 Automated software testing. What? Why? Automated vs. Manual Common types of automated testing methods Automated software testing framework Summary

3. What is automated software testing? 3 A process which covers: • the use of software to control the execution of tests • the comparison of actual outcomes to predicted outcomes • the setting up of test preconditions • other test control and test reporting functions.

4. Why testing is critical? 4

5. Why automated testing? 5 A time savings translates directly into cost savings Improves testing productivity Improves accuracy Increases test coverage Does what manual testing cannot do

6. Effort and Cost 6 • Let’s assume 6 test cases • Effort required to run all 6 manually => 10 min • Effort required to write unit tests for all 6 cases => 10 min • Effort required to run unit tests for all 6 cases => < 1 min • Number of testing iterations => 5 • Total manual testing time => 50 min • Total unit testing time => 10 min

7. Automated vs. Manual testing 7 Not all tests can be automated and most times is difficult to decide what to automate and what to manually test.

8. Advantages 8

9. Disadvantages & other factors 9

10. Common types of automated testing methods 10 Monkey testing Capture / playback Code-based (Unit) testing Intelligent test automation

11. Monkey testing 11 Randomly selecting inputs from a large range of values and monitoring if exceptions are thrown. For example, a monkey test can enter random strings into text boxes to ensure handling of all possible user input. It applies not only for GUI or WEB testing, but also for Unit testing.

12. Tools example 12 Monkey – Android testing tools. The simplest way to use the monkey is with the following command, which will launch the application and send 500 pseudo-random events to it. $ adb shell monkey -v -p your.package.name 500

13. Capture / playback 13 • It’s a set of software programs that capture user inputs and stores it into a script for later replay. • Repeated testing can be performed quickly. • Does not require programming skills. • When the GUI changes, input sequences previously recorded may no longer be valid. • Difficult to determine location of bugs.

14. Tools example 14 Autohotkey - Free keyboard macro program. Supports hotkeys for keyboard, mouse.

15. Code-based (Unit) testing 15 • Individual units of source code are tested to determine if they fit for use. • Ideally, each test case is independent from the others • Bugs can be found in early development stage • Easy to test boundary cases • Not effective for the integrated system testing

16. Unit test-case generation method 16 Random Genetic Other…

17. Random 17 Input values generated randomly Run with generated inputs User checks output values (pass/fail) and saves them for later automatic checking (Oracle).

18. Genetic 18 This methods approach is to combine JML (JAVA model language) and genetic algorithms. JUnit is used as a test execution platform.

19. Tools examples TestGen is a collection of open-source tools that automatically generates unit test cases. 19 Parasoft Jtestis an automated Java testing and static code analysis product

20. Intelligent test automation 20 • A model-based method for generating tests from a description of an application’s behavior (UML) • Models are simple to modify, so model-based tests are economical to maintain over the life of a project. • Most of these methods are researched only theoretically.

21. OCL + UML Constraints Inputs Inputs generator Tested software UML + OCL Oracle Outputs Constraints pass / fail Report 21

22. Automated testing framework 22 A framework is an integrated system that sets the rules of automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. The framework provides the basis of test automation and simplifies the automation effort.

23. Common types of frameworks 23 Data-driven testing Keyword-driven testing Hybrid testing

24. Data-driven testing 24 Data-focused automation User defines just data sets to run tests with Data is defined in external data source and de-coupled from script Good fit for a single application testing, featuring multi-environment, big datasets, and rarely changing Test Cases

25. Example Data set Test script 25

26. Keyword-driven testing 26 Good fit for a multi-application testing featuring multi-environment, big datasets, and a big number of short and straight Test Cases Users de-compose test cases into reusable action keywords Non-coders are able to author automated test cases with action keywords

27. Example 27

28. Hybrid testing 28 • Combines the best of data-driven and keyword-driven testing: • User defines data sets to run tests with • User also defines flow control through action keywords • Data is separated from script and stored in external data source with action keywords

29. Framework components 29 Test generation (data and script generators).Tools might create specialized data such as randomized email messages, or populate databases etc. System configuration. Tools might preserve or reproduce system parameters. Test execution. Tools might operate the software itself, either simulating a user working through the GUI and using an alternative testable interface. Oracles. An oracle is any mechanism by which we detect failure or success. Tools might automatically detect certain kinds of error conditions in a product. Activity recording & coverage analysis. Tools might watch testing as it happens and retrospectively report what was and was not tested. Test management. Tools might record test results, organize test ideas or metrics.

30. General framework structure 30

31. Common functional automated test tools 31

32. Tools examples 32

33. Summary 33 Automated testing improves productivity, accuracy, increases test coverage, saves time and cost Need to decide what to automate and what to manually test The framework provides the basis of test automation and simplifies the automation effort.

34. More information 34 • http://www.slideshare.net/lfingerman/test-automation-best-prcatices-with-soa-test-approach • http://opuntia.cs.utep.edu/utjml/utest.html • http://www.slideshare.net/yogindernath/understanding-of-automation-framework-presentation • http://www.ibm.com/developerworks/rational/library/591.html • http://www.satisfice.com/presentations/agileauto.pdf • http://automation-beyond.com/2009/05/30/test-automation-practices-recordplayback/ • http://en.wikipedia.org/wiki/Test_automation • http://www.slideshare.net/directi/automated-testing-vs-manual-testing-391121

35. Thank you! 35

36. Question ??? 36 What is the benefit of automated testing? What are the common types of automated testing methods? What are the common types of frameworks? What components come to framework? What are the examples of automated testing tools?