Test Assurance – Ensuring Stakeholders get What They Want

1. Test Assurance – Ensuring Stakeholders get What They Want Paul GerrardGerrard ConsultingPO Box 347MaidenheadBerkshireSL6 2GU UK e: paul@gerrardconsulting.comw: http://gerrardconsulting.comt: 01628 639173

2. Why Test Assurance? • Most projects need to be kept ‘honest’ • Sponsors and business • Need someone ‘on the inside’ on their side • Need translations from technical, evasive spin to lay persons’ language • Testing is rarely done well - project needs support from someone capable and bias-free • Risks arising need someone to prod the project into action (who else will do this?)

3. Objectives of the role • Primary objective: Support • To provide advice, leadership and direction to the project team to improve the effectiveness of testing and in particular test reporting • Secondary objective: Assurance • To provide independent assurance to the management board on the thoroughness, completeness and quality of testing • Optional • Excluded from scope is any responsibility for the delivery of testing - specifically excluded to avoid any conflicts of interest

4. Support • Information Provision • Including specific points of expertise or experience • Decision Support • To support the decision making process, where points of technique, documentation, reporting or appropriate (not always 'best') practice may be suggested to the team • Advisory • To advise on process for traceability, coverage, sign-off for assurance purposes • Risk Awareness • To identify risks that must be addressed by test phases • Review support • Critical reviews of coverage, documentation, plans, results while under development.

5. Assurance • Test Audit • Independent audit of test records and process to include checks for traceability, coverage, consistency, accuracy etc. • Test Phase report • Phase report to document the audit, identify areas for correction, interpretation • Sign-Off • See later.

6. Assessment of Testing as a whole • Can the system can be used in a realistic business environment, processing realistic data? • Does the system meets performance, availability and reliability objectives? • Can the system can be supported by operations staff so that exceptions can be handled? • Can the system can be installed, started, stopped, backed up and restored from a range of failure scenarios? • Have all outstanding product risks of concern have been addressed • Have each of the test phases achieved their target level of coverage so that the project can be confident in their assessment?

7. Process and Coverage Assurance • Is the process sound; will it achieve the test phase objectives? • Was the process followed, exit criteria met, waived appropriately? • Did the appropriate, responsibility people perform sign-offs? • What process was used to design tests; were coverage targets set? • Were the targets met in test planning and execution?

8. Product Risk Assurance • What risks drove the test planning, prioritisation? • Were risks addressed through the testing? • Are there risks, not tested that are outstanding?

9. Interventions - Assurance • Test Assurance Notes: Where anomalies or uncertainties in the planning, scope or approach to testing are detected, or where a new risk to be addressed by testing appears, an Assurance Note is a challenge to the project to clarify the purpose of that aspect of the project. • Test Audit: Independent audit of test records and process to include checks for traceability, coverage, consistency, accuracy etc. • Test Phase report: Phase report to document the audit, identify areas for correction, interpretation. • Sign-Off: see slides that define precisely what Assurance Sign-off means (and doesn't mean).

10. Sign-Off by stakeholders and other participants • Indicates that the signatories: • Have read and understood it what they have signed • Have approved a deliverable, outcome or process • Made their decision in a transparent fashion • Commit to a defined course of action – usually to ‘accept’ and/or implement • Have reached a consensus view • Have based their decision on their (or delegated) judgements • Agree that their views have been taken into consideration • Signatories must be recognised authorities • Sign-off is usually irreversible – once signed, cannot be undone • Not granted lightly.

11. Test Assurance Sign-Off (is different) • The test approach, plan, specifications, scripts, logs, incident reports have been reviewed and test lead has been interviewed/consulted • The following have been ‘assured’: • Objectives and acceptance/exit criteria • The test design approach enables objectives to be met • Coverage target(s) ensure that enough information will be gathered to make the acceptance/exit decision • The exit criteria have been met or… • The remaining anomalies have been waived by the business • And… the right people have been involved • The test records indicate that the phase process has been followed and are a true reflection.

12. The politics of assurance • Who will pay for the assurance function? • Would your department employ a critic? • Can the Assurance Manager stay independent? • Or will he go native with the development group? • Can the Assurance Manager speak the language of business? • Are you a techy at heart – unable to see beyond code, bugs, incident reports? • If you made a recommendation, would the development organisation attack you, or the problem? • Assurance Managers sign-off – do they take responsibility for the release? • Do Test Assurance Management skills exist yet? • If you ‘did’ assurance, would it be a career limiting move?