Supervisor: Prof. Jyri Hämäläinen Instructor: Jari Simolin (M.Sc), Nokia Siemens Networks

1. Functional regression testing and testautomation in a 3G networkelementplatformenvironment Jyri Ilama Supervisor: Prof. Jyri Hämäläinen Instructor: Jari Simolin (M.Sc), Nokia Siemens Networks Espoo, 15.12.2009

2. Agenda • Testingterminology • Testautomationprocess & Benefits of TA • UMTS architecture • IPA2800 platform • Call Management domain • ContinuousIntegration • The CI project of Call Management • CI FRT pilotproject • Lessonslearned • Whatwentwrong? • Results and analysis

3. Testingterminology • (our) Functional testing • ”Using” the ”system” • Actually, a script uses a sub system of an individual entity (one, separate network element) • The actual, acceptance / system testing is done later in a System Verification phase • Regression testing • Testing that the software still works after changes are made in some unit

4. Testautomationprocess • Howcanthisbeachieved? What’s inside the blackboxes? • Pitfalls to avoid • Goodpractices

5. Benefits of Test Automation • The required effort for selecting a set of test cases and executing them takes a few minutes • More tests can be executed more often • Some cases are even impossible to perform by manual testing • Use of resources • Repeatability • Reusability • Simply: Saving resources and time!

6. UMTS architecture • We are interested in RNC’s and MGW’S

7. IPA2800 platform • Networkelementsarecomplexnetworks of interconnectedcomputerunits, communicating via messageconnections • Networkelementshaveverystrictrequirements on faulttolerance • ”the fiveninesavailabilityperformance” • 3G networkelementplatform based on DX200, running in RNC’s and MGW’s • Butmuchmoreefficient and clearer of itsarchitecture

8. Call Management domain • Responsible of all call related operations • Setup, release, and connecting of calls • Capacity and performance • Remaining calls in case of recovery actions • E.g. Unit failures

9. ContinuousIntegration (CI) • A practice of software development, where the members of a team integrate their work very frequently, usually at least daily • Whenever an integration is made, it is verified by an automated build to detect possible integration errors as quickly as possible • A complex system that involves lots of people, servers, tools and connections working tightly together

10. ContinuousIntegration (2)

11. CI Server • A computer that runs some CI software and executes integration builds whenever software changes are made • Tools • CruiseControl & Hudson • Java-based free software, e.g. Ant, Maven and Rake builds supported • Modularity: plug-in support

12. The CI project of Call Management • Started in May 2007: only PRB compilation builds in the beginning, unit tests added later • No decent FRT software available • Hipla: an output report of even hundreds of thousands of lines of plain text, not so nice • Summer of 2008: HiBot was designed for running old (Hipla) test cases in a new way • HIT2Python translator, all old Hipla code -> HiBot • Telnet functions implemented manually, in a hurry, by a person who left the company • This was the point where this tool was left in the hands of a summer trainee; me

13. The CI project of Call Management (2) • I got HiBot up and running quickly, against all estimates • Started implementing and/or debugging all the functionalities • Lots of occasional problems with the Telnet connections

14. CI FRT pilot project • MGW test cases of Call Resource Handling • Other pilots by other teams in Call Management • Even more work with correcting all the commands • Still random crashes and annoying connection and prompt detection problems • External problems • IPA Reporting Server • Other tools, such as CruiseControl and SVN

15. CI FRT pilot project (2) • Test automation problems leading to jams and crashes • The old, manual test cases had to be automated properly • June 2009: The "Telnet guy" came back • The problems were found and defects corrected • Time for me to focus on the essential: to find and fix defects in IPA2800 platform • I implemented a very useful script

16. CI FRT pilot project (3) • A very useful script for investigating results

17. Lessonslearned:Automatingindividualtestcases • Old manual cases are the most difficult ones • Changed outputs of functions: scanning strings • If the printout changes a little, does your macro still work? • The principle of modularity • No hard-coded values • Reusability of test cases • a Forever-loop is definitely not your friend

18. Lessonslearned: Automating the wholetest set • Consecutive execution • A test case should be possible to run in any kind of situation, no matter if there are e.g. other calls already, hanging, or whatever • Proper teardowns • At the end of test suites, so the execution of a new suite can be started from scratch • Execution order • Problematic suites: Which first? • Possible system breakers as last ones

19. Whatwentwrong? • New software: Focus on the essential • Critical features/functionality • R&D&T should be done instead of R&D • And regression testing should be as important as the testing of new features • Scrum mode: we’re too deep inside it. Tasks that are not backlog items won’t get enough priority, at least easily • Needed: RT sprints or a team dedicated to RT! • Close all external problems out of sight • Write totally new test cases if possible

20. Results and analysis • A stabile, working CI environment for our FRT • Real defects are found all the time • Also Backlog Item Testing can be done efficiently • The "output2report" script was found very useful • Good lessons about test automation • As well as writing modular, intelligent test cases • A very good quality level of the product release!

21. Results and analysis (2)