1 / 19

Black-Box Test Reduction Using Input-Output Analysis

Black-Box Test Reduction Using Input-Output Analysis. ISSTA ‘00 Patrick J. Schroeder, Bogdan Korel Department of Computer Science Illinois Institute of Technology Chicago, IL USA. Presentation Overview. 1) Motivation 2) Black-Box Testing 3) Input-Output Analysis

majed
Télécharger la présentation

Black-Box Test Reduction Using Input-Output Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Black-Box Test Reduction Using Input-Output Analysis ISSTA ‘00 Patrick J. Schroeder, Bogdan Korel Department of Computer Science Illinois Institute of Technology Chicago, IL USA ISSTA '00

  2. Presentation Overview 1) Motivation 2) Black-Box Testing 3) Input-Output Analysis 4) Determining Input-Output Relationships 5) Conclusion ISSTA '00

  3. Motivation for Research • Experience in Industry as a black-box tester of large, complex, telecommunications systems • Root cause analysis of customer-found faults often discovered ineffective or redundant black-box tests • Additional information from the software implementation often improves black-box testing (after the initial release!) ISSTA '00

  4. Black-Box Testing • Black-box testing is testing from a functional or behavioral perspective to ensure a program meets its specification • Testing usually conducted without knowledge of software implementation - system treated as a “black box” • Black-box test design techniques include: equivalence partitioning, boundary value analysis, cause-effect graphing, random testing ISSTA '00

  5. Observations on Black-Box Testing • Testing from the black-box perspective generates a large number of tests (e.g. AT&T Definity PBX > 3 million tests) • All testers are faced with resource limitations; the number of tests must be reduced • Test reduction done randomly or based on “engineering judgement” may lead to ineffective or redundant tests ISSTA '00

  6. Combinatorial Testing • Combinatorial testing: For programs with multiple inputs, one must decide how to test different combinations of selected test data values • The most thorough approach is to test all combinations - this is quite often impossible due to the large number of tests • However, not to consider testing different input combinations could lead to an unacceptable number of undetected software faults ISSTA '00

  7. X1 X2 X3 . . . Xn Program P Combinatorial Explosion If n = 10 and 10 test data values are selected for each variable, the total number of combinatorial test cases is 1010 ISSTA '00

  8. Current Approaches • Testers may try to identify “important” combinations to test using specifications, historical fault data, and engineering judgement • Orthogonal arrays, or other combinatorial design techniques, may be used to tests pair-wise or n-way combinations of inputs • Difficult to determine impact on fault-detection capability of the test set using these techniques ISSTA '00

  9. Definition of the Problem • In combinatorial testing, black-box testers face the situation where the number of possible test cases far exceeds the time and resources available to execute them • How can we reduce the number of combinatorial tests while maintaining the fault-detection capability of combinatorial testing? ISSTA '00

  10. Proposed Technique • Observations: • Current techniques do not consider program output variables • For a class of programs, not all program inputs affect all program outputs • We propose the use of Input-Output (IO) Analysis to determine which program inputs affect program outputs • This information can be used to reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set ISSTA '00

  11. Simple Example using IO Analysis Input data selected using equivalence partitioning: A={1, 3} B={N,S,E,W} C={TDC, BDM} W Z Total Number of Combinatorial Tests: 2 * 4 * 2 = 16 ISSTA '00

  12. IO Analysis Applied A={1, 3} B={N,S,E,W} C={TDC, BDM} W Z ISSTA '00

  13. IO Analysis Applied A={1, 3} B={N,S,E,W} C={TDC, BDM} T(W)  T(Z) (1, N, TDC) (1, N, BDM) (3, N, TDC) (3, N, BDM) (1, S, TDC) (1, E, TDC) (1, W, TDC) W Z T(W) (1, N, TDC) (1, N, BDM) (3, N, TDC) (3, N, BDM) T(Z) (1, N, TDC) (1, S, TDC) (1, E, TDC) (1, W, TDC) ISSTA '00

  14. Optimization Problem • Create the smallest possible test set which covers all data combinations in the individual output test sets • Test set optimized using “Don’t Care” (DC) conditions T(W) (1, DC, TDC) (1, DC, BDM) (3, DC, TDC) (3, DC, BDM) T(Z) (DC, N, DC) (DC, S, DC) (DC, E, DC) (DC, W, DC) Toptimized (1, N, TDC) (1, S, BDM) (3, E, TDC) (3, W, BDM) ISSTA '00

  15. Determining Input-Output Relationships • Information may be gathered in a variety of ways (e.g. from specifications, code) • Manual collection of information difficult and time consuming • Automated collection possible through: • Static Analysis • Dynamic Analysis • Execution-oriented Analysis ISSTA '00

  16. Automated Analysis Techniques • Static techniques analyze a program’s source code - static program dependencies used to determine IO relationships • Dynamic techniques analyze a program’s execution trace across a set of test data - dynamic dependencies used to determine IO relationships • Execution-oriented techniques manipulate individual inputs across multiple program executions in a test harness to determine IO relationships ISSTA '00

  17. Technique Advantages/Disadvantages • Static techniques reliably determine the absence of relationships between inputs and outputs; however, they may identify relationships that are not actually present • Dynamic analysis and execution-oriented analysis reliably determine the presence of input-output relationships; however, they cannot guarantee that all relationships will be detected • Dynamic analysis and execution-oriented analysis may be used when static analysis, perhaps in over-estimation, indicates that all program inputs affect all program outputs ISSTA '00

  18. Experimental Study Results 1) Warehouse Management System Combinatorial Tests: 3,125 Using IO Analysis 40 2) DACS Configuration System Combinatorial Tests 22,500 Using IO Analysis 158 3) Liquidity Ratio Spreadsheet Combinatorial Tests 1.95 Million Using IO Analysis 625 ISSTA '00

  19. Conclusion • For the class of program and type of problem defined, input-output analysis can reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set • Research continues on identifying program types and testing situations where our approach is applicable • Additional experimentation with execution-oriented analysis to provide program dependencies, and other information useful in black-box testing, is underway ISSTA '00

More Related