AUTOMATED TEST GENERATION Muhammed İ. KALKAN

1. AUTOMATED TEST GENERATION Muhammed İ. KALKAN

2. Automatic Test Generation (ATG) • Index • What is ATG? • ATG Types • Why do we need ATG? • What are the main benefits of ATG • ATG Methods • Testing Methodologies • Automatic Test Pattern Generation • Relevant Examples

3. What is ATG? • Testing an engineered system is the most important part of the product revealed. • Engineers are working on and putting alot of effort into this issue. • Since testing process is required for all obtained engineering products, engineers decided to develop an automated test mechanism instead of generating test cases manually.

4. What is ATG? • Automated Test Generation includes test cases , scenarios and the evalution of the obtained result such as passed and failed. • Given inputs and obtained outputs are evaluated automatically. • The Main Idea : • TEST COMPLETELY AUTOMATICALLY

5. ATG or ATPG? • ATG is an engineering subject which can be applied to both Computer Sciences and other engineering specialities like Electronics or hardware circuits. • ATG is a term which is generally used for Computer Sciences especially software testing. • On the other hand for hardware testing Automatic Test Generation is usually referred as Automatic Test Pattern Generation.

6. ATG or ATPG? • ATPG is defined as follows: • ATPG is an electronic pattern automation method/technology used to find an input (or test) sequence • When applied to a digital circuit, it enables testers to distinguish between the correct circuit behavior and the faulty circuit behavior caused by defects.

7. ATG…….But WHY?? • As years have passed,computer hardwares and its capabilities grow bigger and bigger. • Result 1: Software types and their capabilities are varied. • Result 2: New software demand on market for jobs. • Result 3: Bigger income ,more firms to compete for market ration. • Result 4: Requirement for strong,bug free and developed in small amount of time softwares

8. ATG…….But WHY?? • Result 5 : Software Engineering and Testing studies.

9. ATG…….But WHY?? • Software testing is important • Software errors cost the U.S. economy about $59.5 billion each year • Improving testing infrastructure could save 1/3 cost • Software testing is costly • Account for even half the total cost of software development

10. Benefits • Reduces Testing Time • Reduces Testing Cost • Reduces human errors while testing

11. ATG Methods • Data-Driven ATG • xUnit Frameworks for ATG • Examples : JUnit and NUnit • Model-Based ATG • Keyword-Driven ATG

12. xUnit • Various code-driven testing frameworks have come to be known collectively as xUnit. • Based on a design by Kent Beck, originally implemented for Smalltalk as SUnit, but are now available for many programming languages and development platforms.

13. xUnit • The overall design of xUnit frameworks depends on several components. • Test Fixtures • A test fixture is the set of preconditions or state needed for a test to run. Also known as a test context. • Test Suites • A test suite is a set of tests that all share the same fixture. • Test Execution • The execution of an individual unit test proceeds as follows:

14. xUnit • The setup() and teardown() methods serve to initialize and clean up test fixtures. • Assertions • An assertion is a function or macro that verifies the behavior of the unit under test. Failure of an assertion typically throws an exception, aborting the execution of the current test.

15. Data-Driven ATG • Data-driven testing is a methodology used in Test automation • Test scripts are executed and verified based on the data values stored in one or more central data sources or databases. • Databases can range from datapools, ODBC sources, csv files, Excel files, DAO objects, ADO objects, etc. • Data-driven testing is the establishment of several interacting test scripts together with their related data results in a framework used for the methodology.

16. Data-Driven ATG • Variables are used for both input values and output verification values: navigation through the program, reading of the data sources, and logging of test status and information are all coded in the test script. Thus, the logic executed in the script is also dependant on the data values. • An example of xUnit Framework Automatic Testing by using Data-Driven Test Methodology is as follows:

17. Example • Put the data that varies from test to test into the Data-Driven Test file that the interpreter reads to execute the tests • For each test it does the same sequence of actions to implement the Four-Phase Test

18. Example (FPT) • Set-up the test fixture • Try the test case • Evaluate the result • Release the fixture

19. Keyword-Driven ATG • It is also known as table-driven testing, action-word testing • It can be used for manuel testing as well as automated testing • The advantages for automated tests are the reusability and therefore ease of maintenance of tests which have been created at a high level of abstraction. • The Keyword-Driven ATG methodology divides test creation into two stages. • Planning Phase • Implementing Phase

20. Keyword-Driven ATG Planning Phase • The application (or the requirements for the application) is analyzed to determine which operations and objects will need to be tested.

21. Example A web-questionnaire application will require a large amount of text entries. By identifying which operations should be encapsulated into keywords, the efficiency and maintainability of the tests are maximized.

22. Keyword-Driven ATG Implementation Phase • The implementation stage differs depending on the tool or framework used. • Often, automation engineers implement a framework that provides keywords like “check” and “enter”. • Testers or test designers (who don’t have to know how to program) write test cases based on the keywords defined in the planning stage, which have been implemented by the engineers. • The test is executed using a driver which reads the keywords and executes the corresponding code.

23. Keyword-Driven ATG • Other methodologies use an all-in-one implementation stage. • Instead of separating the tasks of test design and test engineering, the test design is the test automation. • Keywords, such as “enter” or “check” are created using tools in which the necessary code has already been written. • This removes the necessity for extra engineers in the test process, because the implementation for the keywords is already a part of the tool.

24. Keyword-Driven ATG • Benefits • This methodology requires more planning and a longer initial time-investment than going directly to the test creation stage and recording a test. • It does make the test creation and test maintenance stages more efficient and keeps the structure of individual tests more readable and easier to modify. • The more abstract keywords are, the more reusable they are, and therefore the easier a test is to maintain. • As well as reducing the cost and time spent maintaining and updating tests, the modular structure of Keyword-Driven ATG means that new tests can easily be created from pre-existing modules.

25. Keyword-Driven ATG • Another advantage is the reduction in technical know-how required for the test automation process. • In the first approach, technical know-how is only required by the engineers that implement the keywords. • In the second approach, even this is not required, which means that the test team is capable of entirely automating tests, even without programming knowledge.

26. Model Based ATG • Model-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test process • The model is usually an abstract, partial presentation of the system under test's desired behavior. • The test cases derived from this model are functional tests on the same level of abstraction as the model • Can not run abstracted test cases.Need executable ones.

27. Model Based ATG • Diagram description : Model-Based Test Systems

28. Model Based ATG • Many ways to produce test cases. • No best method for producing tests. • Use your test interest to define your test cases. • Design decisions is often known as "test requirements", "test purpose" or even "use case"

29. Model Based ATG • Use model and test requirements to derive abstract test cases • Implementation Extra Information to abstract test suit to produce executable test cases

30. Model Based ATG • Execute the test • Evaluate the reports

31. Model Based ATG • There are a few ways to deploy the Model-Based Testing • Online Testing: A model-based testing tool connects “directly” to a system under test and tests it dynamically. • Offline Generation of Executable Tests: A model-based testing tool generates test cases as a computer-readable asset that can be later deployed automatically. • A collection of Python classes that embodies the generated testing logic

32. Model Based ATG • Offline Generation of Manually Deployable Tests: A model-based testing tool generates test cases as a human-readable asset that can be later deployed manually. • A PDF document in English that describes the generated test steps • Another Step : • Test Generation

33. Model Based ATG • Model-Based Testing has a good potential in aspect of automation. • Test case generation is one of the issues that ATG is subject to • To find test cases, Model-Based structure is interpreted as a kind of Finite State Machine to see different states that the system might be in • There are a few technics for generating test cases.Most knowns are as follows:

34. Model Based ATG • Test case generation by theorem proving • System model is defined as predicates,kinds of logical expressions • These expression are processed and proved afterwards • Test case generation by constraint logic programming • Selecting test cases satisfying specific constraints by solving a set of constraints over a set of variables

35. Model Based ATG • Test case generation by model checking • We provide a model of the system under test and a property we want to test to the model checker • Test case generation by symbolic execution • Searching for execution traces in an abstract model. In principle the program execution is simulated using symbols for variables rather than actual values • Test case generation by using an event-flow model • With a graphical user-interface (GUI) front-end is called the event-flow model that represents events and event interactions

36. Example • TestMaster™ automates the generation of tests for call processing features developed for the 5ESS®-2000 Switch • This test uses Model-Based Test Generation • Results obtained here as follows. • A test generation productivityimprovement of just over 90%

37. Automatic Test Pattern Generation • ATPG is an electronic design automation method/technology used to find an input (or test) sequence that, when applied to a digital circuit, enables testers to distinguish between the correct circuit behavior and the faulty circuit behavior caused by defects • The generated patterns are used to test semiconductor devices after manufacture, • In some cases to assist with determining the cause of failure

38. Automatic Test Pattern Generation • The effectiveness of ATPG is measured by the amount of modeled defects, or fault models, that are detected and the number of generated patterns. • We can estimate test quality by effectiveness of ATGP

39. Automatic Test Pattern Generation • A defect is an error introduced into a device during the manufacturing process. • A fault model is a mathematical description of how a defect alters design behavior. • A fault is said to be detected by a test pattern if, when applying the pattern to the design, any logic value observed at one or more of the circuit's primary outputs differs between the original design and the design with the fault. • There are two steps that ATPG should take to detect fault.

40. Automatic Test Pattern Generation • Fault activation: Establishes a signal value at the fault model site that is opposite of the value produced by the fault model • Fault propagation: Moves the resulting signal value, or fault effect, forward by sensitizing a path from the fault site to a primary output.

41. Sequential ATPG • Searches for a sequence of vectors to detect a particular fault through the space of all possible vector sequences • Vector Sequence: A sequence of values for circuit input • In Sequential Circuits due to the presence of memory elements, the controllability and observability of the internal signals (in general)are much more difficult than those in a combinational circuit • Complexity of sequential ATPG much higher than that of combinational ATPG.

42. Algorithmic Methods • Testing very-large-scale integrated circuits with a high fault coverage is a difficult task because of complexity. • Many different ATPG methods have been developed to address combinatorial and sequential circuits. • Some examples: • Pseudorandom test generation is the simplest method of creating tests. It uses a pseudorandom number generator to generate test vectors

43. Algorithmic Methods • The D Algorithm was the first practical test generation algorithm in terms of memory requirements. The D Algorithm introduced D Notation which continues to be used in most ATPG algorithms. • Fan-Out Oriented Algorithm: It limits the ATPG search space to reduce computation time and accelerates backtracing.

44. Automatic Test Pattern Generation • ATPG can fail to find a test for a particular fault in at least two cases. • The fault may be intrinsically undetectable • it is possible that a pattern(s) exist, but the algorithm cannot find it

45. References • Automated Test Generation, (From a Behavioral Model), James M. Clarke, Lucent Technologies • Automatic Test Pattern Generation , http://en.wikipedia.org/wiki/Automatic_test_pattern_generation • Data Driven Tests, Gerrard Meszaros, http://xunitpatterns.com/Data-Driven%20Test.html, 2007 • Data-Driven Testing, http://en.wikipedia.org/wiki/Data-driven_testing

46. References • Test Automation, http://en.wikipedia.org/wiki/Test_automation • Test Automation Framework, http://en.wikipedia.org/wiki/Test_automation_framework • A Survay on Automatic Test Case Generation, M.Prasanna, S.N. Sivanandam, R.Venkatesan, R.Sundarrajan, Department of Computer Science and Engineering, PSG College Of Technology, http://www.acadjournal.com/2005/v15/part6/p4/ • Keyword-Driven Testing , http://en.wikipedia.org/wiki/Keyword-driven_testing

47. References • Model – Based Testing, http://en.wikipedia.org/wiki/Model-based_testing • Improving Automation in Developer Testing: Achievements and Challenges, Tao Xie, Department of Computer Science, North Carolina State University • xUnit , http://en.wikipedia.org/wiki/XUnit

48. The End Thank You Q & A