Integration Testing of Components Guided by Incremental State Machine Learning

1. Keqin Li, CNRS LSR-IMAG Roland Groz, INPG LSR-IMAG Muzammil Shahbaz, France Telecom August 2006 Integration Testing of Components Guided by Incremental StateMachine Learning

2. Content Motivation Background Framework Scenario Testing Unit Testing Integration Testing Conclusion

3. Motivation Software development is more and more based on component integration Unknown Internal structures and insufficient documents Only partial and informal information is available Provide a required system assembling COTS of which he has a limited knowledge Unit testing and integration testing Have to rely on intuitions and little support is available Integrator: Component:

4. Our Approach Infer formal models of components from behaviours observed during tests Perform integration testing based on the formal models Iteratively improve the formal models

5. Background - Angluin’s Algorithm [1987] Infer formal model (conjecture) from observed behavior (observation table) For deterministic finite-state acceptor Teacher answers Membership query Equivalence query

6. Process Add strings to Observation Table No No Closed & Cons. Empty Observation Table Membership Query Yes Equivalence Query Yes Yes? End String Accepted Or Not Yes Or Counterexample Conjecture Teacher

7. Assumptions Black boxes with known interfaces Only external interfaces are controllable Slow environment Test scenarios

8. Framework Scenario Testing Unit Testing Integration Testing Consult Domain Expert Initial Observation Tables Models of Components

9. Models C.E. C.E. I. T. I. T. I. T. U. T.: Unit Testing I. T.: Integration Testing C.E.: Counter- example M1(1) M1(2) M1(3) U. T. U. T. U. T. M2(1) M2(2) M2(3) U. T. U. T. U. T. … … … Mn(1) Mn(2) Mn(3) U. T. U. T. U. T.

10. Scenario Testing A test scenario is an input/output sequence containing external inputs and outputs A test case is constructed for each test scenario Replace problematic component Record observed behaviour of each component in its observation table

11. Unit Testing Each Component is tested individually For each component Complete the initial observation table Using extended learning algorithm to construct closed and consistent observation table Make conjectured model

12. Integration Testing Unit Testing Yes Coverage Criterion Satisfied? Discre- pency? Test Generation Test Execution No Yes No

13. Test Generation based on Composition of FSMs M N [Petrenko 98] An algorithm to compute M N [Lima 97] Assign colour to “global” transitions An algorithm to generate tests covering colours [Kim 02]

14. References [Petrenko 98] A. Petrenko, N. Yevtushenko, “Solving Asynchronous Equations,” Proceedings of FORTE’98, 1998 [Lima 97] L. Lima Jr., A. Cavalli, “A Pragmatic Approach to Generating Test Sequences for Embedded Systems,” Proceedings of IWTCS’97, 1997 [Kim 02] M. Kim, C. Besse, A. Cavalli, and F. Zaïdi, “Two Methods for Interoperability Tests Generation: An Application to the TCP/IP Protocol,” Proceedings of TestCom 2002, 2002.

15. Dealing with Discrepancy Observe the behaviour of each component, and compare with its model Discrepancy Unexpected output sequence New input symbol

16. Refine (relearn) model Counterexample or new input symbol has been found Add corresponding strings into observation table Perform learning algorithm to make the observation table closed and consistent again Make conjectured model

17. Consulting Domain Expert Models of components and test results Domain expert Identify unexpected behaviours Propose additional test scenarios Iteration

18. Results of Integration Testing For each component, we have a model The joint behaviour of these components has been systematically tested Faults might be revealed during testing

19. Conclusion Combines machine learning and model-based testing for the integration of components Reuses Angluin’s algorithm Iteratively refines models

20. The Models Just a “by-product” Our goal is not to build a complete model of the system Just used as basis for the testing process to: Generate integration tests of components Assess the coverage, hence the quality of the integration testing process

21. The Next Steps Move to Parameterized FSM presentation by Muzammil Shahbaz Identify the type of approximation needed Experiment and case study Revisit model-based test generation strategies

22. Thank you

23. Backup

24. Observation Table ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E S S.A

25. Closed ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E tSA, sS, s.t., row(t)=row(s) S SA

26. Consistent ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E s1S, s2S, row(s1)=row(s2), Then, aA, row(s1a)=row(s2a) S SA

27. Conjecture ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E q0 S a b a b b q2 q1 SA a

28. Observation Table a b ε x x a x y b y x aa x y ab y x ba x y bb y x row(ε) a/x b/x b/y row(a) row(b) a/x b/x a/y

29. Example – Component Hotel Reservation OK/CS1 OK/CD1 No/F q3 q5 No/TL T/PD1 Dbl/F q2 Sgl/F T/PS1 No/HL q4 q6 No/TL H1/TL No/F N/HL q0 q1 H2/TL No/TL No/F q8 q10 T/PD2 Dbl/F q7 Sgl/F T/PS2 q9 q11 No/TL OK/CD2 No/F OK/CS2

30. Initial Observation Table N H1 H2 Dbl Sgl T OK No ε HL Ω Ω Ω Ω Ω Ω Ω N Ω TL TL Ω Ω Ω Ω Ω H1 HL Ω Ω Ω Ω Ω Ω Ω H2 HL Ω Ω Ω Ω Ω Ω Ω Dbl HL Ω Ω Ω Ω Ω Ω Ω Sgl HL Ω Ω Ω Ω Ω Ω Ω T HL Ω Ω Ω Ω Ω Ω Ω OK HL Ω Ω Ω Ω Ω Ω Ω No HL Ω Ω Ω Ω Ω Ω Ω

31. Add N to S N H1 H2 Dbl Sgl T OK No ε HL Ω Ω Ω Ω Ω Ω Ω N Ω TL TL Ω Ω Ω Ω Ω H1 HL Ω Ω Ω Ω Ω Ω Ω … … … … … … … … … No HL Ω Ω Ω Ω Ω Ω Ω N/N Ω TL TL Ω Ω Ω Ω Ω N/H1 Ω Ω Ω F F Ω Ω HL N/H2 Ω Ω Ω F F Ω Ω HL N/Dbl Ω TL TL Ω Ω Ω Ω Ω … … … … … … … … … N/No Ω TL TL Ω Ω Ω Ω Ω

32. Conjecture OK/CD1 No/F row(N-H1-D) row(N-H1-D-T) No/TL T/PD1 No/HL D/F row(N-H1) N/HL H1/TL row(ε) row(N) S/F row(N-H1-S-T) T/PS1 row(N-H1-S) H2/TL No/TL No/F OK/CS1

33. Example – Component User Interface UI_Dbl/Dbl UI_Sgl/Sgl UI_No/No UI_T/T UI_No/No PD1/UO_PD1 PS1/UO_PS1 PD2/UO_PD2 PS2/UO_PS2 UI_H1/H1 UI_H2/H2 UI_OK/OK UI_No/No UI_Name/N HL/UO_HL F/UO_F TL/UO_TL q0 q1 q2 q3 q4 TL/UO_TL F/UO_F HL/UO_HL CD1/UO_CD1 CS1/UO_CS1 CD2/UO_CD2 CS2/UO_CS2

34. Integration Testing Environment oMN cM User Interface (M) Hotel Reservation (N) oM oNM

35. Counterexample UI_Name/UI_H2/UI_Dbl/UI_T The real input sequence of Hotel Reservation is: N/H2/Dbl/T The corresponding output sequence is: HL/TL/F/PD2 According to the model, the output should be: HL/TL/F/PD1

36. Comparing with Angluin’s Algorithm Add strings to Observation Table Yes No Closed & Cons. No Counter- example? Initial Observation Table Unit Testing Yes Integration Testing End Test Output Output Test Component Component A Component B Scenario Testing