1 / 21

Supervisor: Prof. Jyri Hämäläinen Instructor: Jari Simolin (M.Sc), Nokia Siemens Networks

Functional regression testing and test automation in a 3G network element platform environment. Jyri Ilama. Supervisor: Prof. Jyri Hämäläinen Instructor: Jari Simolin (M.Sc), Nokia Siemens Networks. Espoo, 15.12.2009. Agenda. Testing terminology

giza
Télécharger la présentation

Supervisor: Prof. Jyri Hämäläinen Instructor: Jari Simolin (M.Sc), Nokia Siemens Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Functional regression testing and testautomation in a 3G networkelementplatformenvironment Jyri Ilama Supervisor: Prof. Jyri Hämäläinen Instructor: Jari Simolin (M.Sc), Nokia Siemens Networks Espoo, 15.12.2009

  2. Agenda • Testingterminology • Testautomationprocess & Benefits of TA • UMTS architecture • IPA2800 platform • Call Management domain • ContinuousIntegration • The CI project of Call Management • CI FRT pilotproject • Lessonslearned • Whatwentwrong? • Results and analysis

  3. Testingterminology • (our) Functional testing • ”Using” the ”system” • Actually, a script uses a sub system of an individual entity (one, separate network element) • The actual, acceptance / system testing is done later in a System Verification phase • Regression testing • Testing that the software still works after changes are made in some unit

  4. Testautomationprocess • Howcanthisbeachieved? What’s inside the blackboxes? • Pitfalls to avoid • Goodpractices

  5. Benefits of Test Automation • The required effort for selecting a set of test cases and executing them takes a few minutes • More tests can be executed more often • Some cases are even impossible to perform by manual testing • Use of resources • Repeatability • Reusability • Simply: Saving resources and time!

  6. UMTS architecture • We are interested in RNC’s and MGW’S

  7. IPA2800 platform • Networkelementsarecomplexnetworks of interconnectedcomputerunits, communicating via messageconnections • Networkelementshaveverystrictrequirements on faulttolerance • ”the fiveninesavailabilityperformance” • 3G networkelementplatform based on DX200, running in RNC’s and MGW’s • Butmuchmoreefficient and clearer of itsarchitecture

  8. Call Management domain • Responsible of all call related operations • Setup, release, and connecting of calls • Capacity and performance • Remaining calls in case of recovery actions • E.g. Unit failures

  9. ContinuousIntegration (CI) • A practice of software development, where the members of a team integrate their work very frequently, usually at least daily • Whenever an integration is made, it is verified by an automated build to detect possible integration errors as quickly as possible • A complex system that involves lots of people, servers, tools and connections working tightly together

  10. ContinuousIntegration (2)

  11. CI Server • A computer that runs some CI software and executes integration builds whenever software changes are made • Tools • CruiseControl & Hudson • Java-based free software, e.g. Ant, Maven and Rake builds supported • Modularity: plug-in support

  12. The CI project of Call Management • Started in May 2007: only PRB compilation builds in the beginning, unit tests added later • No decent FRT software available • Hipla: an output report of even hundreds of thousands of lines of plain text, not so nice • Summer of 2008: HiBot was designed for running old (Hipla) test cases in a new way • HIT2Python translator, all old Hipla code -> HiBot • Telnet functions implemented manually, in a hurry, by a person who left the company • This was the point where this tool was left in the hands of a summer trainee; me

  13. The CI project of Call Management (2) • I got HiBot up and running quickly, against all estimates • Started implementing and/or debugging all the functionalities • Lots of occasional problems with the Telnet connections

  14. CI FRT pilot project • MGW test cases of Call Resource Handling • Other pilots by other teams in Call Management • Even more work with correcting all the commands • Still random crashes and annoying connection and prompt detection problems • External problems • IPA Reporting Server • Other tools, such as CruiseControl and SVN

  15. CI FRT pilot project (2) • Test automation problems leading to jams and crashes • The old, manual test cases had to be automated properly • June 2009: The "Telnet guy" came back • The problems were found and defects corrected • Time for me to focus on the essential: to find and fix defects in IPA2800 platform • I implemented a very useful script

  16. CI FRT pilot project (3) • A very useful script for investigating results

  17. Lessonslearned:Automatingindividualtestcases • Old manual cases are the most difficult ones • Changed outputs of functions: scanning strings • If the printout changes a little, does your macro still work? • The principle of modularity • No hard-coded values • Reusability of test cases • a Forever-loop is definitely not your friend

  18. Lessonslearned: Automating the wholetest set • Consecutive execution • A test case should be possible to run in any kind of situation, no matter if there are e.g. other calls already, hanging, or whatever • Proper teardowns • At the end of test suites, so the execution of a new suite can be started from scratch • Execution order • Problematic suites: Which first? • Possible system breakers as last ones

  19. Whatwentwrong? • New software: Focus on the essential • Critical features/functionality • R&D&T should be done instead of R&D • And regression testing should be as important as the testing of new features • Scrum mode: we’re too deep inside it. Tasks that are not backlog items won’t get enough priority, at least easily • Needed: RT sprints or a team dedicated to RT! • Close all external problems out of sight • Write totally new test cases if possible

  20. Results and analysis • A stabile, working CI environment for our FRT • Real defects are found all the time • Also Backlog Item Testing can be done efficiently • The "output2report" script was found very useful • Good lessons about test automation • As well as writing modular, intelligent test cases • A very good quality level of the product release!

  21. Results and analysis (2)

More Related