Testing Design Techniques in Embedded Systems

1. Politehnica University of TimisoaraMobile Computing, Sensors Network and Embedded Systems LaboratoryEmbedded systems testingTesting design techniques instructor: Razvan BOGDAN

2. Outlines • The test development process • Categories of Test Design Techniques • Specification-based or Black-box Techniques • Structure-based or White-box Techniques • Experience-based Techniques • Choosing Test Techniques

3. Test scenarios test case 6 test case 6 test case 5 test case 3 test case 5 test case 4 THE TEST DEVELOPMENT PROCESS test case 2 test object and requirements on the test object test case 4 test case 3 delivering test requirements and test criteria test case 1 test case 2 test case 1 • Deriving test cases from requirements • Designing test cases must be a controlled process. • Test cases can be crated in a formal way or in an informal way, depending on the project delimitations and on the maturity of the process that is in use.

4. Test scenarios test case 6 test case 6 test case 5 test case 3 test case 5 test case 4 THE TEST DEVELOPMENT PROCESS test case 2 test object and requirements on the test object test case 4 test case 3 delivering test requirements and test criteria test case 1 test case 2 test case 1 • Traceability • Tests should be traceable: which test case was included in the test portfolio based on which requirement? • The consequences of changes in the requirements on the tests to be made can be identified directly. • Traceability also helps to determine requirement coverage.

5. THE TEST DEVELOPMENT PROCESS • Definitions • Test object: The subject to be examined: a document or a piece of software in the software development process. • Test condition: An item or an event: a function, a transaction, a quality criterion or an element in the system. • Test criteria: The test object has to confirm the test criteria in order to pass the test. • Test execution schedule: A scheme for the execution of the test procedures. The test procedures are included in the test execution schedule in their context and in their order in which they are to be executed.

6. THE TEST DEVELOPMENT PROCESS • Test case description according to IEEE 829 • Preconditions: situation previous to test execution or characteristics of the test object before conducting the test case. • Input values: description of the input data on the test object. • Expected results: output data that the test object is expected to produce. • Post conditions: characteristics of the test object after test execution, description of its situation after the test. • Dependencies: order of execution of the test cases, reason for dependencies. • Distinct identification: Id or key in order to link, for example, an error report to the test case where it appeared. • Requirements: characteristics of the test object that the test case will examine.

7. THE TEST DEVELOPMENT PROCESS • Combining test cases • Test cases may be combined to test suites and test scenarios. • A test procedure specification: defines the sequence of actions for the execution of a single test case or a test suite. It is a script or screenplay for the test describing the steps, handlings and/or activities required for test execution. • With the use of adequate tools, test suites can be coded to be executed automatically. • The (dynamic) test plan states the sequence of tests planned, who is to do them and when. Constraints that have to be considered are priorities, availability of resources, test infrastructure etc. • The test execution schedule defines the order of the execution of test procedures and automation test scripts using for prioritization, regression tests etc.

8. THE TEST DEVELOPMENT PROCESS • Summary • Test cases and test suites are derived from the test object requirements or characteristics • Components of a test case description are: • key/Id • input values • pre-conditions • expected results • post-conditions • dependencies • requirements out of which the test case was derived

9. Outlines • The test development process • Categories of Test Design Techniques • Specification-based or Black-box Techniques • Structure-based or White-box Techniques • Experience-based Techniques • Choosing Test Techniques

10. Black-Box White-Box Equivalence partitioning Boundary value analysis State transition testing Decision tables Use case based testing CATEGORIES OF TEST DESIGN TECHNIQUES black box dynamic experience-based techniques • Black box and white box testing • Dynamic testing is divided in two categories/groups • The grouping is done on the bases of the method to derive test cases • Every group has its own set of methods for designing test cases Statement coverage Branch coverage Condition coverage Path coverage Analytical QA white box ✔ Reviews / walkthroughs Control flow analysis Data flow analysis Compiler metrics/analyzer static

11. Test case design Test cases on the basis of specifications Internal program structure irrelevant Test of all selected resp chosen combinations of input/output data Test object CATEGORIES OF TEST DESIGN TECHNIQUES Black-Box • Specification-based or Black-box Techniques • The tester looks at the test object as a black box • Internal structure of the test object is irrelevant or unknown • Test cases are derived/selected based on specification analysis (functional and non-functional) of a component system • Testing of input/output behaviour • Functionality is the focus of attention! • Black-box technique is also called functional testing or specification oriented testing

12. CATEGORIES OF TEST DESIGN TECHNIQUES • Structure-bases or White-box Techniques • The tester knows the internal structure of the program/code • i.e. component hierarchy, control flow, data flow, etc. • Test cases are selected on the basis of internal program code/program structure • During testing, interference with the test execution is possible • Structure of the program is the focus of attention! • White-box technique is also called structure-based testing or control flow based testing Test case design Test cases on the basis of program structure Test process controlled externally Analyze control flow within the test object during test execution

13. CATEGORIES OF TEST DESIGN TECHNIQUES • Categories of test design methods – overview • Specification-based methods • Test objects have been selected in accordance with the functional software model. • The coverage of specification can be measured (e.g. which percentage of specification covered by test cases). • Structure-based methods • The internal structure of the test object is usedto design test cases (code/statements, menus, calls, etc.) • The coverage percentage is measured and used as a bases for creating additional test cases • Experience-bases methods • Knowledge and experience about the test object and its environment are the bases for designing test cases • Knowledge and experience about possible weak spots, probable errors and former errors are used to determine test cases.

14. CATEGORIES OF TEST DESIGN TECHNIQUES • Summary • Test cases can be designed using different methods • If specification functionality is the focus of testing, the methods used are called specification oriented methods or black-box methods. • If the internal structure of an object is investigated, the methods used are called structure oriented methods or white-box methods. • Experience basedmethods use knowledge and skills of the personnel involved in test case design.

15. Outlines • The test development process • Categories of Test Design Techniques • Specification-based or Black-box Techniques • Structure-based or White-box Techniques • Experience-based Techniques • Choosing Test Techniques

16. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Overview • The following black-box methods will be explained in detail: • Equivalence partitioning • Boundary value analysis • Decision table testing and cause-effect graphing • State transition testing • Use case testing • This accounts for the most important and popular methods • Other black box methods are for instance • Statistical testing • Pairwise testing • Smoke testing

17. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • General • Functional testing is targeted at verifying the correctness and the completeness of a function • Are all specified function available within the module? • Do executed functions give correct results? • The execution of the test cases should be done without high redundancy, but nevertheless comprehensive • Test as little as possible but • Test as much as necessary

18. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning • Equivalence class partitioning is what most testers do intuitively: they divide the possible values into classes. Hereby they look at • Input values of a program (usual use of EC-method) • Output values of a program (rarely used EC-method) • The range of defined values is grouped into equivalence classes, for which the following rules apply: • All values, for which a common behaviour of the program is expected, are groupedtogether in one equivalence class • Equivalence classes may not overlap and may not contain any gaps • Equivalence classes may contain a range of values: (e.g. 0<x<10) or a single value (e.g. x = “Yes”)

19. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – valid and invalid • The equivalence classes of each variable (element) may be divided further • Valid EC: all values within the definition range are combined into one equivalence class, if they are handles identically by the test object • Invalid EC: we distinguish two cases for values outside of the definition range: • Values with a correct format but outside of the value range can be combined into one or more equivalence classes • Values with a wrong format generally build a separate EC • Tests are performed using a single representative from each EC • For every other value from the EC the same behaviour as for the chosen value is expected

20. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – example • Equivalence classes are chosen for valid and invalid inputs • If a value x is defined as 0 ≤ x ≤ 100, then we can initially identify three equivalence classes: • x<0 (invalid input values) • 0 ≤ x ≤ 100 (valid input values) • X > 100 (invalid input values) • Further invalid EC can be defined, containing, but not limited to: • Non-numerical inputs • Numbers too big or too small • Non-supported format for numbers

21. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – input variables • All input values (elements) of the test objective were identified e.g. • Fields of a GUI (e.g. system test) • Parameters of a function (e.g. component test) • A range for each input value is defined • this range defines the sum of all valid Equivalence Classes (vEC) • Invalid equivalence classes (iEC) result from the values outside of this range • values expected to be handled differently (known or suspected) are assigned to a different separate equivalence class.

22. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – example • A program expects a percentage value according to the following requirements: • Only integer values are allowed • 0 is the valid lower boundary of the range • 100 is the valid upper boundary of the range • Valid are all the numbers from 0 to 100, invalid are all the negative numbers, all numbers grater than 100, all decimal numbers and all non numerical values (e.g. “fred”) • one valid equivalence class: 0 ≤ x ≤ 100 • 1st invalid equivalence class: 0 < x • 2nd invalid equivalence class: x > 100 • 3rd invalid equivalence class: x = no integer • 4th invalid equivalence class: x = non numeric (n.n.)

23. <0 0-15 16-50 51-85 86-100 >100 SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – example • The percentage value will now be displayed in a bar chart. The following additional requirements apply (both values included): • Values between 0 and 15: grey bar • Values between 16 and 50: green bar • Values between 51 and 85: yellow bar • Values between 86 and 100: red bar • Now there are four instead of one valid equivalence classes: • 1st valid equivalence class: 0 ≤ x ≤ 15 • 2nd valid equivalence class: 16 ≤ x ≤ 50 • 3rd valid equivalence class: 51 ≤ x ≤ 85 • 4th valid equivalence class: 86 ≤ x ≤ 100

24. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – picking representatives • On the last step, one representative of each EC is determined as well as the result to be expected for it

25. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Exercise: Equivalence Partitioning • From a given specification (online shop) please extract: • All input values • Equivalence classes for each of the input values • Valid equivalence classes • Invalid equivalence classes

26. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – example 2/1 • Analysing the specification • A piece of code computes the price of a product based on its value, a discount in % and shipping costs (6, 9 or 12 EURO, depending on shipping mode) • Assumptions: • Value of goods is give as a positive number with 2 decimal places • Discount is a percentage value without decimal places between 0% and 100%. • Shipping costs can only be 6, 9 or 12

27. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – example 2/2 • Test cases for valid EC: • Valid equivalence classes provide the following combinations or test cases: T01, T02 and T03

28. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – example 2/3 • Test cases for invalid EC: • The following test cases were created using the invalid EC, each in combination with valid ECs of other elements:

29. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – example 2/4 • Test cases for invalid EC: • 10 test cases are derived: 3 positive (valid values) and 7 negative (invalid values) test cases:

30. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – output based • Equivalence classes can also be built, based on the expected output values • The method is used in analogy, applied to the output values • The variable (element) is thus the output (for example, a field values on the GUI) • Equivalence classes are built for all possible outputs defined • A representative is determined for each equivalence class of output values • The input leading to the representative value is than acquired • A higher costs and effort because the input values have to be found for the given output recursively

31. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – in general /1 • Partitioning • The quality of the test depends on precisely segmented variables/elements in equivalence classes • EC that were not identified hold the risk of overlooking possible defects, since the representatives used did not cover all possibilities • Test cases • Equivalence class method provides test cases for which a representative still has to be chosen • Test data combinations are selected by defining the representative or representatives of each equivalence class

32. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – in general /2 • Choosing representatives • Any value within EC can be a representative. Optimal are: • Typical values (used often) • Problem values (suspected failures) • Boundary values (on the edge of the EC) • Representatives of valid EC may be combined • Representatives of invalid EC may not be combined • Representatives of invalid EC may only be combined with valid representatives of other EC • For test cases, representatives of invalid EC should be combined with always the same values of other valid EC (standard combinations) • Choosing representatives implies that the function within the program uses compare operations

33. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – coverage • Equivalence class coverage can be used as exit criteria to end testing activities • EC – Coverage = Number of EC tested / Number of EC defined * 100

34. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – transition • The transition from the specification or definition of functionality to the creation of equivalence classes • Often a difficult task due to the lack of precise and complete documentation • Boundaries that are not defined of missing descriptions make it difficult to define equivalence classes • Often, contact with the customer is needed to complete information

35. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Equivalence Partitioning – benefits • Benefits • Systematic test case design method, i.e. with a minimumamount of test case a defined coverage can be expected • Partitioning value ranges in equivalence classes on the basis of specifications covers the functional requirements • Prioritizing equivalence classes can be used to prioritize test cases (Inputs that are used rarely are to be tested last) • Test of known exceptions is covered by test cases on the basis of negative equivalence classes • Equivalence partitioning is applicable at all levels of testing • Can be used to achieve input and output coverage goals • Can be applied to human input or via interfaces to a system or interface parameters in integration testing

36. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Boundary analysis /1 • Boundary analysis extends equivalence class partitioning by introducing a rule for the choice of representatives • The edge values of the equivalence class are to be tested intensively • Why put more attention to the edges? • Often, the boundaries of value ranges are not well defined or lead to different interpretations • Checking if the boundaries were programmed correctly • Please note: • Experience shows that errors occur very frequently on the boundaries of values ranges! • Boundary value analysis can be applied at all test levels. It is easy to apply and its defect finding capability is high on base of detailed specifications.

37. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Boundary analysis /2 • Boundary analysis assume that: • The equivalence class is composed of a continuous value range (not a single value or a group of discrete values ) • That boundaries can be defined for the value range • As an extension to equivalence class partitioning, boundary value analysis is a method suggesting the choice of representatives • Equivalence class partitioning: • Examines one (typical)value of the equivalence class • Boundary analysis • Examines the boundaries and the neighbouring values • Uses the following scheme:

38. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Defining boundary values • Basic scheme can only be applied when the value range was defined accordingly • In this case additional testing is needed for a valued in the middle of the value range • If the EC is defined as single numerical value, for example x=5, the neighbouring value will be used as well • The representatives (of the class and its neighbouring values) are: 4, 5 and 6

39. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Boundary analysis for invalid EC • Boundary value make little sense for invalid equivalence classes • Representatives of the invalid EC at the boundary to a valid EC are already covered through the basic schema • For value ranges defined as a set of values, no boundaries can be created generally • For example: single, married, divorced, widowed

40. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Boundary analysis example 3a • Example 3a: • Value range for a discount in %: 0.00 ≤ x≤100.00 • Definition of EC • 3 classes: • EC: x <0 • EC: 0.00 ≤ x≤100.00 • EC: x > 100 • Boundary analysis • Extends the representative to: • 2.EC: -0.01; 0.00; 0.01; 99.99; 100.00; 100.01 • Please note • Instead of one representative for the valid EC, there are now six representatives (four valid and two invalid)

41. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Boundary analysis example 3b • Basic scheme: choose three values to be tested – the exact boundary and two neighbouring values (within and outside the EC) • Alternative point of view: since the boundary value belongs to the EC, only two values are needed for testing: one within and one outside the EC • Example 3b • value range for a discount in %: 0.00≤ x ≤ 100.00 • valid EC: 0.00≤ x ≤ 100.00 • Boundary analysis additional representatives are: -0.01; 0.00; 100.00; 100.01 • 0.01 – same behaviour as 0.00 • 99.99 – same behaviour as 100.00 • A programming error caused by a wrong comparison operator will be found with the two boundary values

42. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Decision table testing /1 • Equivalence class partitioning and boundary analysis deal with isolated input conditions • However, an input condition may have an effect only in combination with other input conditions • All previously described methods do not take into account the effects of dependencies and combinations • Using the full set of combination of all input equivalence classes usually leads to a very high number of test cases (test case explosion) • With the help of cause-and-effect graphs and the decision tables derived from them, the amount of possible combinations can be reduced systematically to a subset

43. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Decision table testing /2 • A cause-and-effect diagram use a formal language • A cause-and-effect diagram is created by translating the (mostly informal) specification of a test object into a formal language • The test object is subject to a certain amount of effect that are traced back to their respective causes • Elements 7 symbols: • Assertion (if cause A – then effect E) • Negation (if cause A – then no effect E) • Or (if cause A or B – then effect E) • And (if cause A and B – then effect E)

44. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Decision table testing /3 • Other elements of a cause-and-effect diagram: • Exclusive (Either cause A or cause B) • Inclusive (At least one of two causes: A or B) • One and only one (One and exactly one of two causes: A or B) • Required (If cause A then also cause B)

45. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Decision table testing /4 • Example 5: Online-Banking • The user has identified himself via account number and PIN. If having sufficient coverage, he is able to set a transferal. To do this, he must input the correct details of the recipient and a valid TAN.

46. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Decision table testing /5 • Example 5: Online-Banking • Each table column represents a test case • Creating a decision table: • Choose an effect • Trace back along the diagram to identify the cause • Each combination of causes represent a column of the decision table (a test case) • Identical combinations of causes, leading to a different effects, may merged, to form a single test cases.

47. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Decision table testing /6 • Practical use • The specification is divided into manageable parts, thus leading to decision tables of a practical size • It is difficult to deduce boundary values out of a cause-and-effect diagram or decision table • It is recommended to combine test cases derived from decision tables with values derived from boundary analysis • The number of causes and effects that are examined will determine the complexity of the cause-and-effect diagram: for n preconditions that may be true or false, 2n test cases can be created • On systems of larger size, this method is only manageable with tool support

48. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • Decision table testing /7 • Benefits • Systematical identificationof input combinations (combined causes) that might not be found using other methods • Test cases are easily derived from the decision table • Easy to determine sufficient test case coverage, e.g. at least one test case created for each column of the decision table • The number of test cases can be reduced by systematicallymerging columns of the decision table • Drawbacks • Setting up a large number of causes leads to complex and extensive results • Thus, many errors can occur when applying this method • This makes it necessary to use a tool

49. SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES • State Transition Testing /1 • Many methods only take into account the system behaviour in terms of input data and output data • Different states that a test object might take on are not taken into account • For example, results of actions that happened in the past- actions that caused the test object to be in a certain internal state • The different states that a test object can take on are modelled using state transition diagrams • State transition analysis is used to define state transition based test cases • State transition testing is much used within embedded software industry and technical automation in general

50. dead dead SPECIFICATION-BASED OR BLACK-BOX TECHNIQUES married married “to die” “to die” “to marry” “to marry” widowed divorced “d.o.p” • State Transition Testing /2 • To determine test cases using a state transition diagram, a transition tree is constructed: • The initial state is the root of the tree • For every state that may be reached from the initial state, a node is created which is connected to the root by a branch • This operation is repeated and comes to an end if • The state of the node is an end state (a leaf of the tree) Or • The same node with the same state is already part of the tree dead married “to die” single dead “to die” “be single” unborn “to marry” Event “d.o.p.”: death of partner