Acceptance Testing

1. Acceptance Testing

2. V Lifecycle Model

3. Waterfall Model

4. Definition • Acceptance testing consists of comparing a software system to its initial requirements and to the current needs of its end-users or, in the case of a contracted program, to the original contract [Meyers 1979].

5. Purpose • Other Testing: • intent is principally to reveal errors • Acceptance Testing: • a crucial step that decides the fate of a software system. • Its outcome provides quality indication for customers to determine whether to accept or reject the software product.

6. Goal & Guide • GOAL: • verify the man-machine interactions • validate the required functionality of the system • verify that the system operates within the specified constraints • check the system’s external interfaces • The major guide : the system requirements document • The primary focus : on usability and reliability [Hetzel 1984].

7. Categories of AT Four categories emerged from the survey: • Traceability method • Formal method • Prototyping method • Other method

8. Traceability methods • Traceability methods work by creating a cross reference between requirements and acceptance test cases, fabricating relationships between test cases and requirements. • A straightforward way to match the test cases to the requirements: • allow testers to easily check the requirements not yet tested. • show the customer which requirement is being exercised by which test case(s). • However, each cross reference is created by the tester subjectively.

9. Traceability Methods – cont. • Make software requirement specification (SRS) traceable by requirements tagging: • Number every paragraph and include only one requirement per paragraph • Reference each requirement by paragraph and sentence no. • Use a database to automate traceability task • A matrix or checklist ensures there is a test covering each requirement • Keep track of when tests completed/failed.

10. Figure of Traceability Methods

11. Traceability Methods – samples • TEM • Test Evaluation Matrix [Krause & Diamant 1978] • The TEM shows the traceability from requirements to design to test procedures allocated to verify the requirements. • Allocate requirements to 3 test levels. • Select test scenario that satisfy test objects and exercise requirements in that level. • Generate test cases from test scenarios by selecting target configuration.

12. Traceability Methods – samples • REQUEST • Requirements QUEry and Specification Tool [Wardle 1991] • The REQUEST tool automates much of the routine information handling involved in providing traceability. • A database tool consisting CRD, TRD and VCRI. • Ensure a test covering each requirement.

13. Traceability Methods – samples • SVD • System Verification Diagram [Deutsch 1982] • The SVD tool connects test procedures with requirements. • SVD contains threads • threads stimulus elements • elements represent function associated with a requirement. • The cumulation of threads forms the acceptance test.

14. Formal methods • Apply mathematical modeling to test case generation. They include the application of formal languages, finite state machines, various graphing methods, and others to facilitate the generation of test cases for acceptance testing.

15. Example • A Requirements Language Processor (RLP). It produces a finite state machine (FSM) model whose external behavior matches that of the specified system. Tests generated from the FSM are then used to test the actual system.

16. RLP & FSM example method

17. Prototyping methods • Prototyping methods begin by developing a prototype of the target system. This prototype is then evaluated to see if it satisfies the requirements. Once the prototype does satisfy the requirements, the outputs from the prototype become the expected outputs from the final system. • Therefore, prototypes are used as test oracles for acceptance testing.

18. Prototype Methodology

19. Prototype Methodolgy • Advantage: less function-mismatching errors found in the final test stage and thus saving lots of cost and time. • The difficulty in generating test data for system acceptance test can be greatly reduced.

20. Other Methods • Empirical Data: is an acceptance testing method based on empirical data about the behavior of internal states of program • Define test-bits and combine them into a pattern which replects the internal state. • Collect empirical data by an oracle which stores value of test-bit patterns in the form of a distribution. • Compare pattern observed to data stored in distribution.

21. Other Methods – cond. • Structured analysis (SA) methods involve the use of SA techniques (entity-relationship diagrams, data flow diagrams, state transition diagrams, etc.) in aiding the generation of test plans.

22. Other Methods – cond. • Simulation is more a tool for acceptance testing than a stand-alone method since it is not used in the development of test cases. It is used for testing real-time systems and for systems where “real” tests are not practical.

23. Special characteristics of UAT • Test objective, an identified set of software features or components to be evaluated under specified conditions by comparing the actual behavior with the specified behavior as described in the software requirements. • Test criteria, are used to determine the correctness of the software component under test. • Test strategy, are the methods for generating test cases based on formally or informally defined criteria of test completeness • Test oracle, is any program, process, or body of data that specifies the expected outcome of a set of tests. • Test tool

25. Major problem of Ad hoc testing • test plan generation relies on the understanding of the software system, • lack of formal test models representing the complete external behavior of the system from the user perspective, • no methodology to produce all different usage patterns to exercise the external behavior of the system, • lack of rigorous acceptance criteria, • no techniques for verifying the correctness, consistency, and completeness of the test cases • no methods to check if the acceptance testing plan matches the given requirements.

26. Reference • Software Requirements and Acceptance Testing, Pei Hsia and David Kung, Annals of Software Engineering 3 (1997) 291-317

27. Strategies for UAT • Behavior (Scenario) based acceptance test • Black-box approach for UAT • Operation-based test strategy

28. Definition • A scenario is an concrete system usage example consisting of an ordered sequence of events which accomplishes a given functional requirement of the system desired by the customer/user. • A scenario schema is a template of scenarios consisting of the same ordered sequence of event types and accomplishing a given functional requirement of a system. • A scenario instance is an instantiation of a scenario schema.

29. Scenario-based Acceptance Test • a new acceptance test model - a formal scenario model to capture and represent the external behavior of a software system. • discuss acceptance test error, test criteria, and test generation.

30. Scenario-based AT • Test model consists of three sub-models: • a user view: a set of scenario schema, describe interactive behavior of user with system. • an external system interface: represents behavior of an interface between a system and its interfaced external system. • an external system view: represents external view of the system in terms of user views and external interface view.

31. Scenario-based AT Procedure • elicit and specify scenarios • identify the user’s and external system views according to requirements; • formalize test model • formalize the scenario trees and construct the scenario-based finite state machines (FSMs) • combine FSMs to form a composite FSM (CFSM); • verify test model • verify the generated FSMs and CFSMs against their man–machine and external system interface. • generate test case to achieve a select test criteria • execute test cases

32. Scenario-based AT – cont. • Scenario tree: T(V) = (N, E, L) • For a user view V, consists of finite set of nodes N, a finite set of edges E, a finite set of edge labels L. • Node set consists of a set of states as perceived by the user. • A label l∈L of an edge e∈E between nodes N1, N2∈N, shows the state of the system changed from N1 to N2 due to the occurrence of event type l

33. Benefit • Benefit: its ability to allow users with the requirements analysts to specify and generate the acceptance test cases through a systematic approach.

34. Black-box approach for UAT • Source of TC: functional or external requirements specification of system • Use a functional test matrix to select a minimum set of test case that cover system functions.

35. Black-box approach – cond. Steps to select requirements-based test case: • Start with normal test cases from the specification and an identification of system functions. • Create a minimal set that exercises all inputs/outputs and covers the list of system functions. • Decompose any compound or multiple condition cases into single-condition cases. • Examine the combinations of single-condition cases and add test cases for any plausible functional combinations. • Reduce the list by deleting any test cases dominated by others already in the set.

36. Black-box approach – cond. • Advantage: reduce possibility of having untested procedures • Difficulty: acceptance criteria not clearly defined.

37. Operation-based test • OBT consists of the following components: • Test selection based on operational profiles; • Well-defined acceptance criteria; • Compliant with ISO9001 requirements.

38. Test based on operational profile • An operation is a major task the system performs. • An operational profile (OP) is a set of operations and associated frequencies of occurrence in expected field environment. • Operation profile dependent on users of application. • Most applications have targeted user classes and each user class has their own OP.. • Reliability must be checked against each OP.

39. Operational profiles example

40. Drive testing by using OP • Select test cases according to the occurrence probabilities of the operation. • The amount of testing of an operation is based on its relative frequency of use. • The most recently used operations will receive the most testing and less frequently used operations will receive less testing.

41. Criticality • May classify system operations by their criticality. • Assume criticality of each OP determined separately. • Example, a banking application • teller: the most critical • supervisor: less critical • manager: the least critical

42. Test Selection by Criticality • # of test cases for each OP proportional to its weight criticality: : criticality of OP i : its occurrence probability : total number of OPs

43. Example of test selection based on criticality

44. Acceptance Criteria • The OBT strategy uses the following acceptance criteria: • no critical faults detected, and • the software reliability is at an acceptable level.

45. First acceptance criterion • Let T be a set of acceptance test cases and with the mapping , where siis the input of test ti; and is the output of applying sito program P. • The requirement of no critical fault implies that • where C is the set of incorrect critical output values as defined by the user.

46. Second acceptance criterion (a) reliability of every OP class is acceptable when: where Riis the estimated reliability of the OP i, and is the acceptable reliability of the OP i. (b) Reliability of the whole application is acceptable when: where R0is the estimated reliability of the whole application, and is the acceptable reliability of the whole application.

47. Acceptance decisions • Accept - no defects are detected during UAT, user accept the software deliverable • Provisional acceptance - accepted without repair by concession • Conditional acceptance - accepted with repair by concession • Rework to meet specified requirements - too many faults found for the whole application, but the reliabilities of individual operational profiles are acceptable • Rejected - too many faults found in both the whole application and the individual operational profiles, reject the software deliverable

48. Acceptance decisions

49. OBT strategy disadvantages • It requires more analysis upfront as the user needs to determine the different operational profiles. • Test selection involves more work as the frequency of occurrence becomes a factor in picking the test cases.

50. Comparing UTA strategies