LCG-SPI: SW-Testing LCG Applications Area

1. LCG Software Process & Infrastructure LCG-SPI: SW-TestingLCG Applications Area GridPP 7th Collaboration Meeting Manuel.Gallas@cern.ch LCG/SPI LCG Software Process & Infrastructure LCG-SPI: SW-Testing

2. LHC grid software applications(LHC experiments, projects, etc) SPI SW-Testing • Overview • Test FrameWorks • User support • Test policies • Test documents LCG Application Area POOL PI SEAL SIMU LCG Infrastructure SPI project SW-Testing http://spi.cern.ch/ Outline and Context LCG-SPI: SW-Testing

3. GOAL • Software testing will be an integral part of the software development process. • All level of software testing should be run as part of an automatic process. Nightly Building System SPI SW-Testing Automated testing Use in Exp. Software testing Acceptance test Examples LHC experiments System test System Tests Sw-testing team Integration test Integration Tests Software developer Work Package Test Unit test SW-Testing CODE SW- testing support SW-testing polices Test FrameWorks SW-testing doc Code Documentation Overview LCG-SPI: SW-Testing

4. Aim:to help developers: • to produce code for testing • to run tests in automatic way • Our constrains: • Avoid commercial software and • licensing problems. • Avoid “do it yourself solutions” • Try to adopt commonly used • open-source software. • Our inputs: • Contacts within HEP-community. • What is available as free open source code. SW-Testing Test FrameWorks • Requirements: • Work with different languages (C++, Python, …) • Allow two ways of testing: “test the output of the code” and “test inside the testing-code”. • Easy way to integrate existing tests. • Allow the creation of dependencies among tests • Organize the tests by components or packages. • Regression testing • Easy integration with the Nightly Building System. • Provide also a graphical interface. LCG-SPI: SW-Testing

5. A simple test: • Subclass the TestCase CppUnit or PyUnit class • Override the method runTest(). • When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass inan expresion that is true if the test succeeds CppUnit PyUnit QtUnit JUnit X-Unit family SW-Testing Test FrameWorks: X-Unit family • Scope: Unit –testing • The same “assertion style” in different languages. • Provides:- The name of the test case that failed. • -The name of the source file that contains the test. • -The line number where the failure occurred • Different platforms/compilers: (Linux/Solaris/Windows) LCG-SPI: SW-Testing

6. Oval build Test Source Code • Oval: • validation and regression • used in CMS Executable Oval run Ovalfile (configuration) Log file Reference file Oval diff SW-Testing Test FrameWorks: Oval • Can be use for Unit-test to Validation tests. • Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”) • It is possible to set different run environments. • Can run external scripts and external binaries. LCG-SPI: SW-Testing

7. QMTest SW-Testing Test FrameWorks: QMTest • Uses a graphical interface for creating and running tests • Runs tests in parallel • Organizes tests hierarchically • Supports execution of a single test or many at once • Records dependencies between tests • Can be run in batch mode -> easy integration with the Nightly-Building systems LCG-SPI: SW-Testing

8. Aim:to help developers: • to produce code for testing • to run tests in automatic way Oval QMTest Oval PyUnit CppUnit • Uses a graphical interface for • creating and running tests • Runs tests in parallel • Organizes tests hierarchically • Supports execution of a single • test or many at once • Records dependencies • between tests • In LCG AppArea runs what is in • the OvalFile and uses oval. • Automated tests • Regression test Test Scripts OvalFile QtUnit JUnit Log file ? Ref file X-Unit family Valgrind SW-Testing Test FrameWorks: Global picture LCG-SPI: SW-Testing

9. SPI SW-Testing • Overview • Test FrameWorks • User support • Test policies • Test documents SW-Testing LCG-SPI: SW-Testing

10. Policies Test FrameWorks HowTo Test doc SW-Testing User support: Web information http://spi.cern.ch/ LCG-SPI: SW-Testing

11. Provided as free software by the SPI External Software Service CppUnit Oval QMTest PyUnit SW-Testing User support: Tools supported • Different platforms: • (Linux/Solaris/Windows) • Different compilers • Integrated with SCRAM • and ready to be used • Installed in AFS/DFS • Instructions to install • them in a local machine How to use them ? LCG-SPI: SW-Testing

12. SW-Testing Follow sw-testing policies 1 2 3 User support: How-To * * * * * LCG-SPI: SW-Testing

13. SPI SW-Testing • Overview • Test FrameWorks • User support • Test policies • Test documents SW-Testing LCG-SPI: SW-Testing

14. For the projects For the developers For the testing responsible SW-Testing Sw-Testing policies http://spi.cern.ch/software_development.html • Sw-testing policies are • needed to ensure: • Software is created together • with the tests. • Test directory structure and • naming should be homogeneous • and this will imply easy life to • have automatic tools. • Common and well integrated • tools LCG-SPI: SW-Testing

15. SPI SW-Testing • Overview • Test FrameWorks • User support • Test policies • Test documents SW-Testing LCG-SPI: SW-Testing

16. Test Case template collects information about the existing tests and how they cover the expected functionality of the software. SW-Testing Test Documents • For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities. • We use a template LCG-SPI: SW-Testing

17. SPI SW-Testing • Overview • Test FrameWorks • User support • Test policies • Test documents SW-Testing Conclusions: Software testing is an important task and we have tools in place to do it. • Thanks to: • PPARC • LHC experiments • LCG Feedback and interaction are always welcome!! LCG-SPI: SW-Testing