Risk-Based Testing – An Overview

1. Risk-Based Testing – An Overview Paul GerrardGerrard Consulting1 Old Forge CloseMaidenheadBerkshireSL6 2RD UK e: paul@gerrardconsulting.comw: http://gerrardconsulting.comt: 01628 639173 Assurance with Intelligence

2. Agenda • I Why Risk-Based Testing? • II Introduction to Risk-Management • III Risk and Test Objectives • IV Designing the Test Process • V Project Intelligence, Test Strategy and Reporting • V1 Close, Q&A • Here’s the commercial bit: • This material is based on: • Risk-Based E-Business Testing, Gerrard and Thompson,Artech House, 2002 • Visit www.riskbasedtesting.com for more information. Assurance with Intelligence

3. Why Risk Based Testing?

4. V-Model Requirements User Acceptance Test Is there ever a one-to-one relationship between baseline documents and testing? Functional Specification System Test Physical Design Integration Test Where is the static testing (reviews, inspections, static analysis etc.)? Program Specification Unit Test Assurance with Intelligence

5. “Traditional” approach Methodology Test stage Test stage Test stage Test stage Consider Schedule, Environments, Timescales etc. DevTest System Test Acceptance Build and Execute tests Not Done Not focused Not again! Stakeholder Involvement We have to Trust them Too detailed To understand Are these faults really severe?

6. Problems with tradition • Sequence of decisions • Stages  responsibility  capability  objectives • Guidance to developers and testers • None, except generic, text book mantras • “demonstrate software meets requirements” • Input of stakeholders • Only when system/acceptance tests reveal problems • Far too late! • Decision making • Timescale driven in early stages • Crisis driven towards the end • Unsatisfactory all round. Assurance with Intelligence

7. Acceptance Test Test the Requirements Install System Write Requirements System Test Test the Specification Build System Specify System Integration Test Test the Design Build Software Design System Write Code Unit Test W-Model Assurance with Intelligence

8. Plan assess product risks define test objectives test techniques, products to test Stakeholder Involvement Schedule responsibility estimation process Decide Implement assess product risks risk-based test reporting focused test design and execution Risk-based testing

9. Risk-based test planning • If every test aims to address a risk, tests can be prioritised by risk • It’s always going to take too long so… • Some tests are going to be dropped • Some risks are going to be taken • Proposal: • The tester is responsible for making the project aware of the risks being taken • Only if these risks are VISIBLE, will management ever reconsider. Assurance with Intelligence

10. How much testing is enough? • Enough testing has been planned when the stakeholders (user/customer, project manager, support, developers) approve: • TESTS IN SCOPE • They address risks of concern and/or give confidence • THE TESTS THAT ARE OUT OF SCOPE • Risk is low OR these tests would not give confidence • The amount and rigour of testing is determined by CONSENSUS. Assurance with Intelligence

11. Even penguins know how to manage risk! Assurance with Intelligence

12. Risk response planning • Do nothing! • Pre-emptive risk reduction measures • information buying • process model • risk influencing • contractual transfer • Reactive risk reduction measures • contingency plans • insurance • The risk that’s left is the residual risk. Where testing fits in Assurance with Intelligence

13. Risk and Test Objectives

14. Why use risks to define test objectives? • If we focus on risks, we know that bugs relating to the selected mode of failure are bound to be important. • If we focus on particular bug types, we will probably be more effective at finding those bugs • If testers provide evidence that certain failure modes do not occur in a range of test scenarios, we will become more confident that the system will work in production. Assurance with Intelligence

15. Risks and test objectives - examples

16. Risk-based test objectives are usually not enough • Other test objectives relate to broader issues • contractual obligations • acceptability of a system to its users • demonstrating that all or specified functional or non-functional requirements are met • non-negotiable test objectives might relate to mandatory rules imposed by an industry regulatory authority and so on • Risk assessment might miss something, or de-scope something important • Generic test objectives • ‘catch all’ measure – e.g. all requirements coverage • complete the definition of your test stages. Assurance with Intelligence

17. Generic test objectives Assurance with Intelligence

18. Designing the Test Process

19. Tester Activity Risk Identification • Consult business, technical staff • Prepare a draft register of risks Risk Analysis • Discuss risks • Assign probability and consequence scores • Calculate exposure Workshop Risk Response • Formulate test objectives, select test technique • Document dependencies, requirements, costs, timescales for testing • Assign Test Effectiveness score • Nominate responsibilities Tester Activity Review and Decision Test Scoping • Agree scope of risks to be addressed by testing • Agree responsibilities and budgets Test Process Definition • Draft the test process from the Test Process Worksheet • Complete test stage definitions Tester Activity Master Test Planning process Assurance with Intelligence

20. Test process worksheet Assurance with Intelligence

21. Plannedend start today master testplanning test objectives test process definition Residual Risks test specification test plan/ procedures test execution test results analysis test log release risk assessment Progress through the test plan Test products through the lifecycle test stages initial risk assessment Assurance with Intelligence

22. Project Intelligence, Test Strategy and Reporting Assurance with Intelligence

23. Goal Assessment Goal Assessment Goal Assessment Acceptance PI Management Stakeholder Involvement/Project Assurance/Governance PI and the project lifecycle Project Initiation Project Planning Project Intelligence Planning Results Chain Analysis PI Strategy Test Strategy Key: Project planning and initiation Project Intelligence Activities Development activities Review and Test activities Assurance with Intelligence

24. Project Phase Design Build Integ Systest UAT Trial Prod. Reqs Objectives for each test phase are easily identified PI Strategy overview PI Drivers Ass. Obj. Business goals Coverage goals Risks Assurance with Intelligence

25. PI Process Overview – designed to handle change Project Risk Management PI Reporting 3 Project or Process Risks 5 Product Risks Out of Scope 7 Goals Outstanding Risks Outstanding Coverage Outstanding 12 Goals Achieved Risks Addressed Coverage Achieved Risk will not be tested (no test objective) All new goals/risksare “outstanding” initially Risk is not product-related,or is programme-critical Test Process Management Test Phase Failed tests must be repeated when fixes received Testobjectivedefined 4 Evaluation & Analysis 6 Allocation to Test Phase 2 Classification 8 Define/ Design Tests 9 Run Test Goals and Risks Product risk Tests have revealed a new risk or changes to a risk New risk identified 1 Requirements Analysis/ coverage 10 Identify Regression Test Coverage Goals Change affects requirements (and tests) 13 Impact Analysis 11 Run Regression Test Change Change requires regression tests Assurance with Intelligence

26. Project Phase Design Build Integ Systest UAT Trial Prod. Reqs Risk and goal-based reporting PI Drivers Ass. Obj. Business goals Coverage goals Risks Progress towards goals is clearly seen. Outstanding risks are highly visible. Assurance with Intelligence

27. start today Plannedend Residual Risks residual risks of releasing TODAY Progress through the test plan Risk-based reporting all risks ‘open’ at the start Assurance with Intelligence

28. Closed Goal based test reporting Goal Goal Goal Goal Goal Benefit Objective Objective Objective Objective Objective Open Closed Open Risks Closed Closed Open Open Benefits availablefor release Assurance with Intelligence

29. Risk-based test approach: planning • RBT approach helps stakeholders: • They get more involved and buy-in to the approach • They have better visibility of the test process • RBT approach helps testers • Approval to test against risks in scope • Approval to not test against risks out of scope • Clearer test objectives upon which to design tests • RBT approach helps developers • Specifies their responsibility for testing in detail • “No hiding place”. Assurance with Intelligence

30. Risk-based test approach: execution and reporting • RBT approach helps stakeholders: • They have better visibility of the benefits available and the risks that block benefits • RBT approach helps management: • To see progress in terms of risks addressed and benefits that are available for delivery • To manage the risks that block acceptance • To better make the release decision. Assurance with Intelligence

31. Risk-Based TestingAny Questions?riskbasedtesting.comgerrardconsulting.comuktmf.com