1 / 17

Tactic-Based Testing

Peter Braun, Benjamin Flach, Reinhard Jeschull, Jan Philipps. Tactic-Based Testing. MBTUC 2011 | Berlin, 18-20 October 2011. Model-Based Testing. Behavior model. Model determines search paths. Model provides reference values. Search criteria. Test case generator.

Jims
Télécharger la présentation

Tactic-Based Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Peter Braun, BenjaminFlach,Reinhard Jeschull, Jan Philipps Tactic-BasedTesting MBTUC 2011 | Berlin, 18-20 October 2011

  2. Model-Based Testing Behavior model Model determines search paths Model provides reference values Search criteria Test case generator Model returns coverage data Test cases

  3. LessonsLearned Complex data spaces and algorithms Inclusion of prior knowledge of developers Modeling language and tool issues

  4. Tactic-Based Testing Tactic determines search paths Tactics Main idea: Seamless combination of model- and script-based testing Test cases Data model Tactic Interpreter & Coverage measurement Coverage over tactic, data and oracle models Oracle model Oracle model provides reference values

  5. Tactic-Based Testing - Framework Tactics JRuby as tactic language Test cases Data model Tactic Interpreter & Coverage measurement Data models based on EMF, Java, … Oracle model Test case and coverage data output as XML Oracle models in UML, Java or JRuby. Generic interface for other languages

  6. Tactic-Based Testing – Tactics • Use of tactics to build up arbitrary search strategies • The structure of the tactics defines the structure of the test cases Tactic parameters T Partial test case Set of partial test cases

  7. TacticBasedTesting – Sample Tactic written in JRuby incorrect_pins = [...] # List of 5 incorrect PINs tacSelFile = step("select", {"file" => ["1C38", "69A5", "CA4B"]}) tacNothing = identity(); tacVerifyOnce = step("verify_chv", {"chvValue" => CORRECT_CHV}) tacVerifyTwice= step("verify_chv", {"chvValue" => incorrect_pins}) >> step("verify_chv", {"chvValue" => CORRECT_PIN}) tacRead = step("select", {"recNo" => [1, 3, 10], "mode" => [RecordMode::ABSOLUTE]}) # Try toread a recordof a file after (in)correct/missingcard holder verification returntacSelFile >> (tacNothing | tacVerifyOnce | tacVerifyTwice) >> tacRead Results in 63 generatedtestcases Visualization of a tactic structure

  8. Validas AG • Founded 2000 • 17 employees • Competences • Model-based development • (Test-) Specification • Test automation • Tool qualification • AUTOSAR • Customers & Partners • BMW • EADS • ESG • Giesecke & Devrient • Infineon • Audi/AEV

  9. FreeRTOS – Interface Testing Interface and robustness testing of off-the-self components Functional dependencies Potential side effects Hardware configuration (memory size, address layout, …) Large parameter space Return code Sequences of interface calls Nontermination Parameter structures Crash Parameter data

  10. FreeRTOS – Test Setup 30.444 test cases, 153.851 test steps 41 API-interfaces tested 41 tactics Tactics FreeRTOS Code generator XML test cases Data model Tactic interpreter Test control Test framework About 850 lines of JAVA code Oracle model Compiled test cases Compiler About 1600 lines of JAVA code Test Case Compilation Test Case Generation Test Case Execution on SUT (Cortex M3) Execution time of about 7 hours

  11. FreeRTOS – Test Results Example: pvPortMalloc (232-5) • Expected result: NULL-Pointer • Observed result: Valid memory address • Reason: Integer overflow (232 – 5) + 16 = 232 + 11  11 Header of linked list block for heap management

  12. Smart Card – Command Testing Exhaustive testing of CREATE_FILE and its parameters to create elementary files on a smart card High variability of parameter structures Functional dependencies Configuration of the smart card (max. file size, reserved IDs, …) Complex parameters Statusword Command sequences Data Parameter structures CreateEF Crash Parameter data

  13. Create File – Test Setup Test Execution on SUT Test Case Generation Report Generation 16 tactics About 1000 test cases Tactics Test results Test cases Data model Tactic interpreter Report generator Test executer EMF-based, generated JAVA code Oracle model Report Smart card Execution time 7:30 minutes About 900 lines of Ruby code(CREATE_EF and 10 supporting commands)

  14. Create File – Data Model Generic and domain-specific data models based on EMF

  15. Create File – Test Execution Tactics to achieve test goals Test reports & analysis support Test execution environment

  16. LessonsLearned Complex data spaces and algorithms Inclusion of prior knowledge of developers Modeling language and tool issues

  17. Summary Tactics • Tactic-based testing for flexible control of test case generation • Precise specification of test cases with varying parameters and structures • Applicable to wide range of test objects Test cases Data model Tactic Interpreter & Coverage measurement Oracle model CreateEF Validas AG Arnulfstraße 27 80335 München Fon +49 (0) 89-53 88 669-0 Fax +49 (0) 89-53 88 669-10 info@validas.de www.validas.de

More Related