1. IN-Application

2. Granny´sFreephone

3. The IN Service Definition Environment An Instance of the IN-Product Line! Early AMDD

4. Agile Process-Oriented Design Business Process Analysis Orchestration Process Requirements Platform Services Validation, Analysis Process Model Deployment Running Process Testing& Monitoring

5. Testing

6. Call control services, Call screening • Call monitoring and management • Voice and Data Call Association • Intelligent Routing The Scenario • System-level testing of 3rd-party CTI-system Today: VoIP

7. Test Coordinator LAN Switch Application PCs ISDN Network Application Server 3rd Party CTI-system

8. Hicom 150 H OfficePro Medium Size Communication System based on Euro-ISDN technology for digital and analog trunk and user interfaces.

9. TestCoordinator Rational Robot Rational Robot HUSIM Hipermon ISDN Network Intranet/ Internet Concrete Test Setting LAN TAPI CorNet Call Center Clients TAPI CSTA II/III Call Center Server

10. The Problem Improving effectiveness and efficiency of system-level testing • Test design • separate test of functions from test tool specific actions initiating the functions • guide designers towards building plausible test cases • Test execution • automatic • coordinate activities of a set of test tools • guarantee repetition

11. Design Integrated Test Environment Analysis Verification Execution ITE Concept

12. Test design Test execution Management layer Abstraction Coordination Test tool 1 Test tool 2 System Component 2 System Component 1 System Component 3 System-under-test (SUT) The Problem • Objective

13. Concept - Coarse Granular Level • Establishment of a coarse-granular level Hmm ... If I am here and I do this I should end up there this ( "agent login" ) there here

14. Coarse-grained Models Application Building Center (ABC)

15. Design of test cases: • combine basic functionalities • by means of a graphical notation • according to the control flow followLink isCorrectPageVisible? passed failed Concept - Coarse Granular Level

16. The Implementation • Library of test blocks • Definition and classification • Common actions • Switch-specific actions • Call-related actions • Internet-related actions • SUT-component -related actions • Application-specific actions • Specialization • Parameterisation • Specification of branching conditions

17. Concept - Aspect Oriented Level • Establishment of an aspect-oriented level Hmm ... If I want to end up here I must have done thisbefore here ("agent logout") this ("agent login")

18. Local constraints • Global constraints Concept - Aspect Oriented Level • The aspect-oriented level

19. The Concept • Test model • defined as a quadruple (S, Act, , s0) • set of test blocks S • set of possible branching conditions Act • a set of transitions   S  Act  S • initial test block s0 • visualised as a graph (DFA) • basis for design of test cases • basis for checking a test case's consistency

20. The Concept • Formulation and interpretation of global constraints • formulated as SLTL formulasA: atomic propositions over Sc: propositional logical formula over Act • interpreted over set of all possible test sequences • Library of constraints • formulated using a first-order extension of SLTL

21. The Implementation • Legal test cases e.g., non-ambiguity and non-contradiction of verdicts • Telephone service test casescorrect functioning of phones, e.g., turn-on/turn-off cycle of basic or special functions • SUT-specific test casese.g., opening/closing of modal dialogs

22. TestTool TestCoordinator SUTComponent Concept • Test Execution • Control flow driven execution of test blocks • Preparation of an execution protocol

23. TestCoordinator Rational Robot TestTool TestCoordinator SUTComponent Call Center Clients ISDN Network Hipermon Intranet/ Internet Call Center Server Concept • Test Execution • Control flow driven execution of test blocks • Preparation of an execution protocol

24. drastic improvement of the test process (factor ~ 40 for test execution) Results 20% - 80% - • open extensible test framework

25. Unified Messaging Services • Automated Call Distribution (topic experts) • Interactive Voice Response (IVR) - dynamic teleworking • ... HiPath AllServe Call Center: Hipermon 30 ACD-groups 8 Supervisors (PCs) 64 Agents (PCs) LAN/internet LAN/internet CVS Virtual Switch Integrated Testing Environment Test coordinator HPCO Client Rational-Robot LAN HUSIM V24 LAN V24 HPCO Server S0-Link (2 Ports) for Unified Messaging a/b-Link (4 Ports) for announcements Hicom 150 H Office TCP/IP-Link for TAPI 3rd

26. The Evolution Problem: • Testing large evolving systems during reconfiguration • (HiPath Personal Call Manager) • testing internet services (web-based applications) • dealing with a virtual switch • higher benefit of error detection compared to ‘steady state’ test phases • (handle scalability issues)

27. PCM Features Role management • views, roles, rights • inside the application under test • groups of developers/testers User management • actions, rights, reporting • normal mode • at reconfiguration Switch/call management • devices, policies, exceptions • qualified call forwarding • consistency and failover

28. TestCoordinator Rational Robot HTTP HUSIM HUSIM HUSIM HUSIM PCM Application PCs Hipermon Hipermon Hipermon Hipermon CSTA II/III HTTP PCM Application Server Concrete Test Setting : PCM operation

30. TestCoordinator Rational Robot HTTP PCM Application PCs CSTA II/III HUSIM HUSIM HUSIM HUSIM HTTP PCM Application Server Hipermon Hipermon Hipermon Hipermon Test Execution ^ ^ ^

31. Control flow driven execution Behaviour-oriented design Test Coordinator Library-based consistency checking Agent Building Center Workflow Editor Model Checker Workflow Execution Engine Tool Access Adapter The Complete System • Implementation fundamentalsApplication Building Center (ABC)

32. Conclusions • Approach to testing role-based, distributed, • heterogeneous systems with • coordination-oriented model • library-based test design • library-based consistency checking • incremental formalisation • verification-supported testing • global, feasible, open, scalable

33. Many happy users!

34. Model Extrapolation

35. Learning Models Modelling Testing What about Legacy Systems? • Classical learning is very expensive, • Special model properties may drastically reduce the effort (exploiting expert knowledge) to well <=1%% • Adequate Modelsdrastically improve selectivity • `Classical´ validation and analysis methods become available for Legacy-Systems

36. Model Generator Rational Robot Rational Robot HUSIM Hipermon Observe ISDN Network Learn Intranet/ Internet Learning CTI Systems LAN TAPI CorNet Call Center Clients TAPI CSTA II/III Call Center Server

37. Design Analysis Verification Execution ITE Concept Integrated Test Environment

38. Using L* [Angluin 87] L* learns a finite automaton by • actively posing membership queries and equivalence queries to that automaton in order to extract behavioral information, and • refiningsuccessively an own hypothesis automaton based on the answers. Membership query: tests whether a string (a potential run) is contained in the target automaton’s language (its set of runs) Equivalence query: compares the hypothesis automaton with the target automaton for language equivalence.

39. Test Execution Model Extrapolation Model Checking • Testing • Model Verification Model Extrapolation Test Generation ManagementLayer System under Test (SUT)

40. The Principle • initializeL* with the available set of traces • construct a consistent behavioral modelby establishing local consistency with these observations. • Check for global consistency Result: a finite state behavioral model, which is an extrapolation of the given requirement specification: • it comprises all traces of the specification, • but will typically contain many (even infinitely many) more.

41. { {deviceId = A1 hookswitchOnHook, ... }} device A1 display(line 1, ...) LEDs: (1,on) (2,off) ... ... Sketch of the Model Structure Models comprise state changes as well as UPN-and CSTA-Observations. Sys_Info obs_CSTA upnOffHook obs_CSTA obs_CSTA obs_CSTA Sys_Info

42. Conceptual Model Structure:MIOLKTS • Modal:for loose specification • Input: must always be enabled • Output:is system-controlled • Labelled: complex annotations • Kripke: state-oriented • TransitionSystems: action-oriented

43. Lower Hypothesis Automaton Reaching Words Closeness & Consistency Validation Transitions Observation Table Distinguishing Futures OT Unknown System

44.   1 ( is in the language) Membership Queries OT Unknown System {a,b} Abstract States a 1 Not closed! At least 2 states {1, 0} b 0 Transition Relation

45. a b a,b Closure & Consistency  OT Unknown System  1 b 0 1 Closed & Consistent a 1 ba 0 0 bb 0 Finished?

46. a b a,b Equivalence Query  OT Unknown System  1 b 0 a 1 ab 1 a 1 ba 0 bb 0 Counterexample: ab  L

47. a b a,b Counter Example-Based Extension  OT Unknown System  1 b 0 a 1 ab 1 ba 0 bb 0 aa 0 aba 0 abb 1 Closed. Complete?

48. Not consistent: row () = row (a), 1 1 but row (a)  row (aa) 1 0 New Column: a Closure & Consistency  OT Unknown System  1 b 0 a 1 ab 1 ba 0 bb 0 aa 0 aba 0 abb 1

49. a b b a a,b Next Iteration  a OT Unknown System  1 1 b 0 0 a 1 0 10 11 ab 1 0 Closed & Consistent ba 0 0 States {11, 00, 10} bb 0 0 aa 0 0 00 aba 0 0 abb 1 0 Finished!

50. generates huge number of test cases Test Execution Model Extrapolation System Automata Learning System ATE System under Test (SUT)