1 / 22

-Monica Kothari

QE Efforts Optimization using O rthogonal A rray T esting S trategy - A Case Study. -Monica Kothari. Execute Smartly?. OR. Execute Endlessly?. Major Challenges. Scope. What If. Less no. of tests cases but still high functional and code coverage.

tave
Télécharger la présentation

-Monica Kothari

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. QE Efforts Optimization using OrthogonalArrayTestingStrategy -A Case Study -Monica Kothari

  2. Execute Smartly? OR Execute Endlessly?

  3. Major Challenges Scope

  4. What If... • Less no. of tests cases but still high functional • and code coverage • Systematic scoping of test cases across the • domain with minimal manual effort • Find new bugs which could not be found using • conventional approach • Find bugs early in the cycle 60% Less Effort !

  5. Agenda • Complexity of the testing problem • Geometric View of Test Cases • Taguchi’s OATS approach • Example on Orthogonal Array • Why use OATS • Comparison of Test Planning Methods • OATS implementation • Applying the Technique • ROI Analysis • What works and What Doesn’t!

  6. Complexity of a Testing Problem Figure3Test Case Coverage Flow Diagram

  7. Key Limitations • Generating all combinations manually is error prone: • Total Combinations = Template(5)*FCType(4)*Item(8)*Action(8)*6(Verify) • = 7,680 combinations • Identifying optimal combinations to test without missing any complex combinations ensuring high functional and code coverage ensuring the confidence in quality of product. • Minimum testing cycles for execution and automation • development.

  8. Geometric View of Test Cases

  9. Orthogonal Array Testing Strategy Taguchi OATS Approach Orthogonal Array Testing Strategy created by Dr. Genichi Taguchi is a proven, systematic, statistical way of testing pair-wise interactions. • Orthogonal Array is an array of values in which: • Each column is an independent variable to be tested for interaction represented as – “Factor”. • Each Factor can take a certain set of values called “Levels”. • Each row represents a test case/combination.

  10. OATS - Example 3 Parameters– A,B,C 3 Values– 1,2,3 Applying OATS All possible cases involving 3 parameters: 3*3*3 = 27 cases TABLE1Sample Array using OA Total cases = 9 which cover all pair-wise combinations of the 3 variables.

  11. Why Orthogonal Array Testing Strategy? • Identify a minimum set of tests that will find all multi-mode defects (particularly serious defects) • All single-mode defect will be found if every option is tested at least once (unit elements) • 2-way or “Pair-wise” is combinations of 2 items (parameters) that cause a defect • 3-way or “Tri-wise” is combinations of 3 items (parameters) that cause a defect • “ Pair-wise” defect detection (pair-wise testing) finds most defects, ex. testing all pairs typically finds 75% of defects Source: Kuhn • NASA Deep Space Mission - study showed that 88% of bugs discovered using “pair-wise” defect detection testing • U.S. Food and Drug Administration – study showed 98% bugs discovered using “ pair-wise ” defect detection Source: 27 th NASA/ IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 Dec, 2002. • “ The Combinatorial Design Approach to Automatic Test Generation” Source: IEEE, Software 1996, Vol. 13, No. 5 • Baseline application under study resulted in a reduction in test plan time from 1 month to less than 1 week • In several experiments, “pair-wise” demonstrated good code coverage and defect detection ability

  12. Comparison of Test Planning Methods

  13. OATS Implementation - A Case Study Both the approaches were followed and analyzed: Traditional approach to generate the test spec based on the functional spec and domain experience. Orthogonal Array Testing Strategy to generate the optimized test set using OA Tool.

  14. OATS Implementation - A Case Study contd.. TE Effort: 9 mds TE Effort: 3 mds

  15. Applying the Technique STEP 1: Identify the Factors and Values for generating the Test Set. STEP 2:Identify any dependency of values between Factors to avoid incorrect combinations ex. Floating cart is enabled only when mini-cart is enabled.

  16. Applying the Technique STEP 3:Input the parameters as Factors and Values OA Tool that generates Orthogonal Array Test Set. FIGURE1OptimizedTest Table using rdExpert Tool

  17. Applying the Technique STEP 4:Analyze the test set generated to ensure that it covers required cases. • If not, modify the Factors and Values to generate another set. This may be done by splitting the Factors and getting 2 different sets. • Add any manual cases which are not part of the test set. • Incorporate any boundary values for parameters which are required to be tested.

  18. STEP 5:Generate the Test Spec in required format using the Script functionality of the Tool. Figure2 Script for converting matrix to Test Spec Applying the Technique

  19. ROI Analysis for Release OATS has been used extensively in many releases and off-cycles with excellent ROI:

  20. What works! and What Doesn’t! • Applying OATS manually is not advisable. • Focusing the testing effort on the wrong area of the application. • Picking the wrong parameters to combine. • Orthogonal array only tests the most optimal combinations and not all. • One must use their testing skills, expertise, and experience to improve the test cases produced by these methods.

  21. Execute Endlessly or Efficiently? Execute Endlessly? Use Conventional Approach Execute Efficiently? Use OATS • Enables to select a small test set • High functional and code cvg • Increased overall productivity • Systematic scoping of cases • Easy Analysis • Optimal implementation time • Minimal Test Execution • Simpler and less error prone • Useful for integration and regression testing OR

  22. References [1] Orthogonal Array Testing Strategy (OATS) Technique - Jeremy M. Harrell [2] Orthogonally Speaking, StickyMinds. (2001): http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=MAGAZINE&ObjectId=5031&tth=DYN&tt=siteemail&iDyn=2#table3go#table3go [3] http://www.pairwise.org/tools.asp [4] http://upiia.uab.es/teach/a21291/apunts/provaOO/OATS.pdf [5] http://www.combinatorialtesting.com/clear-introductions

More Related