1 / 20

Justifying Software Testing in the 21 st Century

Justifying Software Testing in the 21 st Century. Presented by Ian Gilchrist IPL Software Products Manager. Introduction. The Software industry is about 50 years old. Testing has generally been regarded as the mainstay verification technique But testing is an unpopular job.

kendra
Télécharger la présentation

Justifying Software Testing in the 21 st Century

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Justifying Software Testing in the 21st Century Presented by Ian Gilchrist IPL Software Products Manager

  2. Introduction • The Software industry is about 50 years old. • Testing has generally been regarded as the mainstay verification technique • But testing is an unpopular job. • New software techniques have emerged in the last 5-10 years • Can these be used to justify omitting testing? • Does testing still have a role to play in the 21st century?

  3. ‘New’ techniques • Use of these following new techniques can all be said to lead to ‘better’ software: • Formal verification, via automated analysis of pre-/post- conditions • Best-known form is SPARK for Ada code (sub-set of) • ‘Advanced’ defect detection, via deep static analysis of potential run-time errors • Best-known forms include Polyspace, Klocwork, and Coverity, mainly for C/C++, Ada, and Java • Automated code generation, direct from design, supposedly eliminating human coding-errors • Examples include Matlab/Simlink, and SCADE, generating C code

  4. New Software Techniques Formal Ver’f’n or Defect removal ‘Better’ source code Source code Hand code Compiler Design Spec Source code Code Gen. Object Code

  5. Claims • A speaker at a recent Ada conference (June 2005), stated, “It is rarely cost-effective to rely on testing for evidence of testability… The way forward is more to rely on use of…deep static analysis tools.” • Polyspace website used to claim that its use removed the need to unit test code. Now it says, “…our solutions enables organizations to reach unmatched levels of software reliability while reducing the time and cost of software testing.” • SCADE website says, “Coding errors are eliminated.”

  6. Context and Definitions • Context • Restricted to unit/module testing, of ‘high-integrity’ software, coding in Ada, C and C++ • Definitions • Verification is a set of techniques directed towards confirming that a software component “performs its intended function, and does not perform any unintended function.” [IEC 61508] ‘Good’ verification involves • Generation of evidence • Repeatability • Automated (within limits)

  7. Verification Categories • Verification techniques come in three categories: • Reviews/inspections • Analyses • Testing • It is conventional to use a (judicious) mixture of techniques from more than one category, to achieve an ‘acceptable’ level of confidence “Testing cannot show the absence of errors…Verification is typically a combination of reviews, analysis and testing.” [RTCA DO-178B]

  8. Aim of this Paper • Take a set of ‘old’ and ‘new’ verification techniques • representing commonly used methods in industry. • Subject them to three ‘usefulness’ criteria • Come to a conclusion about the continuing relevance of testing in the 21st century. • Still just as relevant? • Less relevant? • Irrelevant?

  9. The Selected Techniques • Reviews/Inspections • Enforcement of coding standards • Fagan Inspections • Analyses • ‘Advanced’ defect detection • Formal verification • Testing • Functional testing • Structural (i.e. with added coverage) • Statistical (use data to mimic operational conditions)

  10. The Three Criteria • To what extent does each technique contribute to providing confidence that the software performs its intended function? • To what extent does each technique contribute to providing confidence that the software does not perform unintended functions? • To what extent is each technique pragmatic: • Generates evidence • Can be automated • Is repeatable • Cost-effective

  11. Comparison of Techniques

  12. Results • The ‘results’ presented here are subjective. • They represent only the observations, opinions, and prejudices of the author!

  13. Reviews and Inspections

  14. Analyses

  15. Testing

  16. BIG ELEPHANT in the corner • The best static verification technique (formal verification) can only demonstrate that the source code ‘looks good’ • The weak link is the compiler • For code to work as intended the compiler must correctly convert source to object code • IPL’s experience is that all compilers have faults • Some are better than others • Ada compilers are better than C • C compilers are better than C++ • Source code can be ‘perfect’ but still not work correctly when compiled

  17. Wild Card • What about Code Generation? • Can we by-pass the need to verify software at all? • Do we 100% trust the code generator? • There is still the compiler issue • I think even generated code needs verification.

  18. The Strengths of Testingg • Testing is nearly the only technique that demonstrates that code ‘performs its intended function’ • It has a more limited role in demonstrating the absence of faults • It can be highly automated, repeatable, and generates evidence • It need not cost a lot • Be objective about when to stop testing

  19. Conclusion • Reviews and Analyses (IMO) at best serve to demonstrate that code is READY FOR TESTING • The new techniques however can justify the use of less testing than was considered necessary in the past • How to define ‘less’ testing…

  20. Any questions?

More Related