LCG-SPI: SW-Testing

1. LCG-SPI: SW-Testing LCG AppArea internal review (20/10/03) Manuel.Gallas@cern.ch LCG/SPI LCG Software Process & Infrastructure LCG-SPI: SW-Testing

2. Outline SPI SW-Testing • SW-Testing Overview • LCG-SPI SW-Testing • Test Frameworks • Test Documents • User support LCG-SPI: SW-Testing

3. Nightly Building System SPI SW-Testing Automated testing Use in Exp. Software testing Acceptance test Examples LHC experiments System test System Tests Sw-testing team Integration test Integration Tests Software developer Work Package Test Unit test CODE SW- testing support SW-testing polices Test FrameWorks SW-testing doc Code Documentation SW-Testing Overview GOAL (from RTAG2) • Software testing should be an integral part of the software development in the LCG App Area. • All level of software testing should be run as part of an automatic process. LCG-SPI: SW-Testing

4. to produce test code suitable to be run in automatic way • to document their tests • to plan their test phase • to run all their test in automatic way • to browse and scan easily the test results and failures Aim:to help developers: to help projects: • Our constraints: • Avoid commercial software and • licensing problems. • Avoid “do it yourself solutions” • Try to adopt commonly used • open-source software. • Our inputs: • Contacts within HEP-community. • What is available as free open source code. LCG-SPI SW-Testing • Requirements from the users : • Work with different languages (C++, Python, …) and in different environments (platforms & compilers) • Allow two ways of testing: (1) “check the output of the test-code” • (2) “check the test results inside the test-code”. • Provide a easy way to integrate existing tests. • Allow the creation of dependencies among tests. • Easily establish a criteria if a test should pass or fail in a given release • Organize the tests by components or packages / integration / system /acceptance . • Regression testing and check respect to the previous sw-release. • Easy integration with the Nightly Building System. • Provide also a graphical interface to run the tests and examine the test results. LCG-SPI: SW-Testing

5. End User Installation checks QA activity Project Release Nightly Building Sw-Testing Top layer QMTest • Uses a GUI for creating and running tests (also in batch). • Can run tests in parallel, supports execution of a single • test or many at once (test-cases & test-suites) • Organizes tests hierarchically • Records dependencies among tests - Integrates different ways to test. - Common environment to run the tests and to access the test results. Oval CppUnit PyUnit QtUnit JUnit Bottom layer Acceptance testing SW Product Examples - Adaptable to the programming language and developer - Prepared to be run in automatic way Old tests Test Scripts Unit testing X-Unit family Test Frameworks: Global picture LCG-SPI: SW-Testing

6. A simple test: • Subclass the TestCase CppUnit or PyUnit class • Override the method runTest(). • When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass inan expresion that is true if the test succeeds CppUnit PyUnit QtUnit JUnit X-Unit family Test Frameworks: X-Unit family • Main Scope: Unit–testing • The same “assertion style” in different languages. • Provides:- The name of the test case that failed. • -The name of the source file that contains the test. • -The line number where the failure occurred • Different platforms/compilers: (Linux/Solaris/Windows) LCG-SPI: SW-Testing

7. Oval build Test Source Code • Oval: • validation and regression • developed & used within • CMS experiment Executable Oval run Ovalfile (configuration) Log file Reference file Oval diff Test Frameworks: Oval • Can be use for Unit-test to Validation tests. • Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”) • It is possible to set different run environments. • Can run external scripts and external binaries. LCG-SPI: SW-Testing

8. QMTest Test Frameworks: QMTest • Uses a graphical interface for creating and running tests • The configuration files are in XML and can be created from the GUI. We provide also script to do it • Runs tests in parallel • Organizes tests hierarchically • Supports execution of a single test or many at once • Records dependencies among tests • Can be run in batch mode -> easy integration with the Nightly-Building systems LCG-SPI: SW-Testing

9. Test Case template collects information about the existing tests and how they cover the expected functionality of the software. Test Documents • For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities. • We use a template LCG-SPI: SW-Testing

10. Policies Test Frameworks HowTo Test doc User support: Web information http://spi.cern.ch/ http://spi.cern.ch/testing LCG-SPI: SW-Testing

11. Follow sw-testing policies Automatic testing requires some standardization 1 3 2 User support: How-To * * * * * LCG-SPI: SW-Testing

12. SPI SW-Testing • SW-Testing Overview • LCG-SPI SW-Testing • Test Frameworks • Test Documents • User support Final Remarks • Software testing is a needed step in the software • development process. • SPI provides test frameworks, templates and help to • have a coherent automatic testing procedure with a • common interface to run and browse the test results. • The proposed testing tools were chosen following the • criteria of a simple approach, tools in use in the • experiments and freely available. • Following standards makes life easier to SPI and the users. • Automatic testing is always needed and is a way to consolidate • the software. • Bug reports and typical use-cases from the users • should be converted into tests • SPI SW-testing has to evolve and consolidate following • the needs of the LCG projects. • All SPI sw-testing tools, templates, documentation can be • used by any project. • Thanks to: • LCG-PI team • LCG-POOL team • LCG-SEAL team • LCH-experiments LCG-SPI: SW-Testing