Complex Test Suites Organization

1. Complex Test Suites Organization Victor Kuliamin kuliamin@ispras.ru ISP RAS, Moscow

2. Modern Software Complexity SystemYearSize(MLOC) Windows 3.119923 Windows NT 3.119936 Windows NT 3.5 1994 10 Windows 951995 15 Windows NT 4.0199616.5 Solaris 7.0 1998 7.5 Red Hat Linux 5.2 1998 12 Debian 2.0 1998 25 Windows 2000 2000 29 RedHat Linux 6.2 2000 17 Sun StarOffice 5.2 2000 7.6 Debian 2.22000 59 RedHat Linux 7.1200130 Windows XP2002 40 Red Hat Linux 8.0 2002 50 Debian 3.0 2002 105 Windows Vista Beta 1 2005 50 Fedora Core 4 2005 76 Debian 3.1 2005 216 Mac OS X 10.4 2006 86 Modern software systems • Are huge and have a lot of functions • Have very complex interfaces • Are developed by hundreds of people SystemYear DevTeam Size Windows NT 3.11993 200 Windows NT 3.5 1994 300 Windows NT 4.01996 800 Debian 1.2 1996 120 * Debian 2.0 1998 400 * Windows 20001999 1400 Debian 2.2 2000 450 * Debian 3.0 2002 1000 * Debian 3.1 2005 1500 *

3. Test Suites Complexity System Test Cases MS Word XP 35000 Oracle 10i 100000 Windows XP >2000000 SystemYear TestTeam Size Windows NT 3.11993 140 Windows NT 3.5 1994 230 Windows NT 4.01996 700 (0.9) Windows 20001999 1700 (1.2) Scalability of testing techniques is strongly related with test suite organization System Size(MLOC) Test suite size(MLOC) Java 1.5 5 13 (JCK)

4. Traditional Test Suite Organization Set of test cases • Suitable for test suite size management • Suitable for failure analysis • Test suites for complex systems became huge and poorly manageable • Many common parts and low reuse • Hard to modify Test suite

5. Full Set of Problems • Test execution • Selection the necessary tests • Test suite configuration • Test result analysis • Failure diagnostics • Test adequacy reporting • Test creation and maintenance • Explicit relations with requirements, testing goals, and components under test • Fight with complexity • Reuse

6. Approaches • Attributing • Configuration parameters • Modularization

7. Attributing Test suite Adding a set of test case attributes • Very simple • Provides multi-aspect tests classification • Provides links to requirements and components under tests • Can help to measure test completeness

8. Configuration Parameters A set of parameters that control test execution and test results assessment • Provide additional flexibility, portability and adaptability of test suite • Help to check optional requirements • May be assigned dynamically • May make test suite too complex

9. Modularization Separation of modules performing specific tasks • Facilitates reuse • Enhances maintainability • Facilitates requirements traceability • May complicate test suite • May complicate failure analysis

10. “Right” Modules • Test run-time support • Scheduler • Timer • Event monitor • Test adapter • Test data generator • Test oracle

11. Test Adapters Make possible reusable and abstract tests Test Adapter SUT

12. Test Data Generators Provide data for tests • Can contain filters or selectors • Can be composed from smaller data generators

13. Test Oracles Provide verdict of SUT behavior correctness ? System under Test Test Data Generator Test Oracle Partial oracles • Event or operation oracle • Data integrity constraint • Enhance reuse • Simplify requirements traceability

14. Oracle Composition • Simple caseAny failure detected by partial oracle is global • Complex data integrity constraints • Behavior of distributed systemsInterleaving semantics ✕       

15. Conclusion • Test suite organization techniques improving testing scalability already exist • The most powerful one is modularization • But usage of these techniques is restricted by existing tools • All the techniques presented need proper support in tools

16. Thank you for attention! Any questions? kuliamin@ispras.ru