Black Box Testing Techniques: Partitioning and Model-Based Testing

Black Box Testing(A&O Ch. 4)(2nd Ed. Ch. 6) Course Software Testing & Verification 2018/19 WishnuPrasetya & Gabriele Keller

Plan • Partitioning based testing (A&O Ch 4/2nd Ed Ch. 6). • Model based testing Note: black box testing is an important concept. For example, system level testing is often done in a black box setup. In this lecture we will discuss two important techniques that are commonly used in such a setup. The first one, partition based testing, is discussed in length in A&O. However, the second one, model based testing (MBT), is only lightly touched in A&O. This lecture will introduce you to some basic concepts of MBT. We will borrow some key concepts from Hoare’s “Communicating of Sequential Processes” (CSP). For a more practical exposition on MBT you can look at e.g. Practical Model-Based Testing by Utting - Lageard (UU students can access it for free from Scienc Direct, https://www.sciencedirect.com/science/book/9780123725011)

White box testing Testing is “white box testing” if you have knowledge of the source code of the target program to help you designing the tests, and your tests have in principle access to all the program’s variables so that they can inspect them if they are in the correct state. • Compare this with Def 1.26 A&O (2nd Ed. See p 26). • White box setup is common at the unit-testing level.

Black box testing Testing is ”black box testing” if you do not assume full knowledge of the inner working of the target program. Usually this also entails that your tests have only limited access to the program state to inspect its correctness. • Compare with Def 1.25 in A&O (2nd Ed. See p26). • Note that although we may have full access to the source code, at the system level we may choose not to use this knowledge because it becomes to complicated to comprehend. Black box testing is common at the system-testing level. We will discuss two approaches : (1) Partition-based testing, Ch 4 (2nd ed. Ch. 6) and (2) Model-based testing.

Partition-based testing • Premise: The input space of a program can be partitioned into “equivalence classes”, such that inputs from the same partition/equivalence-class trigger the “same kind” of behavior. • It then makes sense to require that every partition should be tested at least once. • Even without source code, we can often propose a reasonable partitioning.

Recall this example... Let’s now try to come up with test cases for this function from the black box perspective. EnumTriType { Isoleces, Equilateral, Scalene } TryType triangle(Float a, Float b, Float c) { ... } If a, b, c represent the sides of a triangle, this methods determines the type of the triangle.

Partitioning triangle()’s domain ISOLECES SCALENE EQUIDISTANCE NON TRIANGLE input domain of triangle This suggests 4 test-cases, one for every partition.

Boundary value test Errors often lurk in the “boundaries” between partitions  test values on and around the boundaries. Examples: triangle(1,2,2.999) ISOLECES SCALENE EQUIDISTANCE NON TRIANGLE triangle(1,2,3)

Example 2 intincomeTax(intincome, intage, intchildren) • Fragments from its informal specification: the method calculates income tax: • Income below 10K is not taxed. • A person under 18y or above 70y is not taxed, as long as the income is below 50K. • Tax reduction applies, linear to the number of children the person has. This method has a more complex input space than the “triangle” example.

Consider the following partitioning The input domain of incomeTax(..) • As before, we can try to come up with test cases that would cover every partition. • Note that just 3 test cases can cover all partitions!  does not feel very strong. This is because the approach ignores that different parameters may ”interact”. E.g. normally when the age<18 the person will not be taxed. However, the combination of income≥50k and age<18 will trigger its own behavior, namely that the person will be taxed anyway. <10k <18 0 X X [18..70] [10k..50k) >0 >70 ≥50k age #children income

Combinatoric testing incomeTax(income,age,#children) • In this example the input domain is spanned by 3 characteristics, and there are in total 8 blocks. • A block combination over the characteristics, e.g. (<10k,<18,0) abstractly specifies a test case. • There are in total 3x3x2 = 18 such combinations of blocks. <10k <18 0 X X [18..70] blocks [10k..50k) >0 >70 ≥50k age #children income characteristics

Combinatoric testing Let’s name the blocks for convenience: • Stats: 8 blocks, 18 combinations. • (C4.24/2nd Ed. C6.2, EACH CHOICE coverage) Each block must be tested. |T| = (maxi: 0i<k: Bi) ; usually too weak. • (C4.23/2nd Ed. C6.1, ALL coverage) All combinations must be tested. |T| = ( i: 0i<k: Bi) ; does not scale up. A A A X X B B B C C income age #children

How about to just cover all pairs? • Premise: most of the times, only t out of N parameters actually influence the behavior. E.g: • when the income is less than 10k, there will be no tax, regardless the age and #children. • When the income is <50k and the age is <18, there will be no tax, regardless #children. • If we speculate on t=2, this leads to “pair-wise” testing: to cover all block-pairs over different characteristics. A A A X X B B B C C income age #children

Pair-wise and t-wise testing • (C4.25/2nd Ed. C6.3, pair-wise coverage). Each pair of blocks (from different characteristics) must be tested. • There are 9+6+6 = 21 pairs to cover. But we can cover them with just 9 test cases. • Pair-wise coverage is stronger than EACH CHOICE, and still scalable. • (C4.26/2nd Ed. C6.4, t-wise coverage). Generalization of pair-wise. A A A X X B B B C C income age #children

Example: a test set with full pair-wise coverage A A A X X B B B C C income age #children

PWC, example 2 • Four characteristics, each with two blocks: • Minimal test set that gives PWC: A B A B A B A B

Consider again this test set <18 <10k A A 0 A • Despite giving full pwc these tests miss some important cases: • We want to fully test (B,B,x)  missing (B,B,B). • We want to fully test (C,x,x)  missing e.g. (C,A,B) and (C,C,B) • Solution: add constraints, but... (see next slide). X X [18..70] [10k..50k) B B B >0 ≥50k C C >70 income #children age

Pair-wise and t-wise testing • Consider a program P with N characteristics [1..N]. For simplicity, suppose each characteristic is split into B number of blocks. • Total number of pairs = B2*V2 where V2 is the number of subsets of size 2 out of N characteristics. So, V2 = = • To cover all pairs you need at least B2 number of test cases. Expect that it can be more (as in Example 2). • In general, finding a minimum size test set that gives full t-wise coverage is not trivial. • However, as pointed out before, k-wise testing ignores the “semantic” (that some combinations should be included or excluded (because they are not sensical)). We can add constraints, though this makes the problem of calculating the minimum test set even harder.

Adding a bit of semantic (C4.27/2nd Ed. C6.5, Base Choice Coverage, BCC) Decide a single base test t0. Make more tests by each time removing one block from t0, and forming combinations with all remaining blocks (of the same characteristics). Example: t0 = (B,Q,X), generates these additional test requirements : (B,Q,Y) (A,Q,X) (C,Q,X) (B,P,X) (B,R,X) A B C P Q R X Y |T| = 1 + (i : 0i<k : Bi - 1)

Consider again Base test t0 = (B,Q,X), and these additional test, giving full BCC: (B,Q,Y) (A,Q,X) (C,Q,X) (B,P,X) (B,R,X) A B C P Q R X Y • What if we also need to insist on testing out all combinations of (C,P,-) ? • Proposal: use multiple “base tests”

MBCC (C4.28/2nd Ed. C6.6, Multiple Base Choices coverage). For each characteristic we decide at least one base block. Then decide a set of base tests; each only include base blocks. For each base test, generate more tests by each time removing one base block, and forming combinations with remainingnon-base blocks. Decide the “base blocks”: (red) Choose one or more best tests. They can only use base blocks. E.g.: base test t0 = (B,Q,X) base test t1 = (C,P,X) A B C P Q R X Y |T| at most M + M*(i : 0i<k : Bi - mi)

Example MBCC We need to add these tests to get full MBCC: • Varying t0 over non-base blocks: (A,Q,X), (B,R,X), (B,Q,Y) • Varying t1 over non-base blocks: (A,P,X), (C,R,X), (C,P,Y) • Remove duplicates (there are none above) The base blocks are marked red. The base tests: base test t0 = (B,Q,X) base test t1 = (C,P,X) A B C P Q R X Y

Example-2, MBCC A BC P QR X Y Z Red : base blocksChosen base tests = (A,P,X), (A,Q,Y) These produce these additional test requirements: (B,P,X)(C,P,X)(B,Q,Y)(C,Q,Y) (A,R,X) (A,R,Y) (A,P,Z) (A,Q,Z) • Some properties to note: • base-blocks are not cross-combined except as in the base tests. • non-base blocks are not cross-combined with each other. • BCC and MBCC will also cover every pair of (base-block,non-base-block).

Overview of partition-based coverage ALL Multiple Base Choice Coverage t-Wise (if the base tests cover all combinations of base blocks) Base Choice Coverage Pair-Wise EACH CHOICE

Models are useful a b • A model helps us in understanding the program it models. • It can be used as a specification that defines the correctness of the program. • It provides guidance on how to systematically test the program (e.g. if the model is an FSM, we can try to cover all its prime paths, rather than just trying out random paths). A simple FSM modelling some program: 𝜏 a b

Model based testing (MBT) • MBT is a popular testing technique. • Applications: • Testing communication protocols • Testing embedded systems • Testing GUI frontends • Typically used in a blackbox setup. • I will explain MBT using a simplified setup, to focus on illustrating MBT’s main concepts such as model “conformance”.

Example: synchronously interactive systems • Consider an interactive system that interacts with its environment by exchanging messages. We’ll ignore the direction (to or from the system). Such an exchange is atomic and synchronous (both the system and environment first agree to do the exchange and then both will wait until the exchange is completed). • The visible behavior of such a system can be modelled by an FSM. Example: a • Note: • a,b represents observable messages. • 𝜏 represents an internal action (not visible from the environment). b 𝜏 a b

MBT, basic idea a b 2 1 a model M : 0 a 3 b System S The tester (a human or algorithm) takes the role of the environment. He/she tests S by interacting with it to determine if S conforms M. interact

Black Box MBT • MBT is often done in a black box setup where the tester cannot see S’s actual state. This complicates testing. a b System S 2 1 model M 0 a 3 interact b

Trace • A trace is a sequence of messages exchanged by an implementation or a model. • We can reasonably assume that traces are observable. • traces(M) = the set of all traces that are allowed by the model M.

Conformance • When do we consider an implementation S to conform a given model M? Keep in mind that in our blackbox setup we can’t peek into I’s actual state. • Observational-based definition (Hoare, Communicating Sequential Processes, http://www.usingcsp.com/cspbook.pdf, p39) : • We can observe traces. S conforms M = if every possible observation of the behavior of S is described by M

Conformance • Let ∑ be the set of possible messages that a system S can do (so-called “alphabet” of S), and also the set of messages that a model M can do. • Let ∑* be the set of all possible finite traces over ∑. Let’s define: • Note: this is a quite conservative definition. A system S that does nothing will conform any model (rationale: an S that does nothing will also not do anything dangerous). A more advanced definition takes e.g. “refusals” into account (Hoare CSP p102); this will be sketched later. S conforms M, iff every trace of S is also a trace of of M.

Testing Conformance a b • Given the previous def. of “S conforms M”, we can test it with a finite-sized test suite T ⊆ ∑*. For each σ∊T that S can do, we check if σ is also allowed by M. • Since ∑* is infinite in size, we can’t test all its traces. But we can select T e.g. such that we cover all transitions, or all prime paths in M. System S 2 1 model M 0 a 3 interact b

Limitation • Pure trace-based conformance has its limitation though. E.g. a system S that behaves as in the left box above would be judges as conforming the model M on the right, despite that S may refuse to do anything further after an “ab” (and the model won’t). • To remedy this, we would need more “observation”. a System S b 2 1 a b model M 0 2 1 a 3 𝜏 0 interact a 3 4 b b

Note: extending observation with refusals a b • Let V⊆ ∑. (𝜎,V) is a so-called failure-trace (CSP) of M if M can do 𝜎 (so, 𝜎∈traces(M)), and afterwhich M may refuse to do any of the action in V. • For the above M, with ∑ = {a,b}, its failure traces: • (𝜖,∅), (b,{b}) , (ba, {a,b}) • (abk, {a}) , with k≥0 • Also: if after the trace 𝜎 M can refuse V, it can also refuse any subset of V • An improved definition of “conform”: S conforms M iff every failure-trace of S is also a failure-trace of M. System S 2 1 model M 0 a 3 interact b

Failure-trace-based conformance • Under trace-based conformance: S conforms M • S may refuse {b} after abk, M cannot refuse it. So, under failure-trace-based conformance: S does not conform M. a System S b 2 1 a b model M 0 2 1 a 3 𝜏 0 interact a 3 4 b b

Non-determinism • Some system can be non-deterministic, e.g. due to internal concurrency, use of random generator, probabilistic AI. a b a 2 1 a 𝜏 2 1 0 a b 3 4 0 a 3 4 b After doing a, this system can be in either state 1 or 3 (the tester has no control on the choice) This system can non-deterministically proceed from 3 to 4.

Non-determinism a b • Non-determinism complicates testing. Consider the above model. • Decreasing observability: suppose S can do ab. By our trace-based definition of conformance, this conforms M, because ab∊traces(M). However, if after a the system actually moves to state 4, it would be an error to subsequently do b. Unless there is a way to obtain more information about S’ state, the possibility is indistinguishable. • Decreasing control: it might be hard to try to get S to do ab (it might keep taking the 𝜏 to go to state 4). System S 2 1 𝜏 model M 0 a 3 4 interact b

Other MBT setups • We considered where a is a synchronous action of data exchange. We ignored the direction. • Outside our scope: different systems may require different types of model, for example: • You may want to distinguish between actions which are inputs for the system and actions that are outputs of the system: • Actions may be parameterized. • Actions may be asynchronous. • Different types of models lead to a different concept of “conformance” and therefore also a different way to test it. !b ?b The tester cannotcontrol which output the system will do on state 0. 1 1 The tester can control which input action to do on state 0. 0 0 3 !a 3 ?a

Black Box Testing Techniques: Partitioning and Model-Based Testing