1. Non-Functional Testing Why is this so often done badly or not done at all? Can we build a Test Managers toolkit to help? What would it contain? Paul Godsafe Non-Functional Testing

2. Non-Functional Testing • Reliability - The capability of software to maintain its level of performance under stated conditions for a stated period of time. • Maturity, Recoverability, Fault Tolerance • Usability - The effort needed for use, and the individual assessment of such use, by a stated or implied set of users. • Learnability, Understandability, Operability • Efficiency - The relationship between the level of performance of the software and the amount of resources used, under stated conditions. • Time Behaviour, Resource Behaviour • Maintainability - The effort needed to make specified modifications. • Stability, Analyzability, Changeability, Testability • Portability - The ability of software to be transferred from one environment to another. • Installability, Replaceability, Adaptability, Conformance (e.g. to a particular database standard) Ref. ISO 9126 Non-Functional Testing

3. Why is Non-Functional Testing so often done badly or not at all? • Need not clearly defined or understood • Creates problems, not solutions • Need to define requirements • Where do you start (or end)? • Often hard to understand and/or interpret results • Expensive to do • People, kit, software, time • Risk/reward is hard to quantify in advance • Are you an optimist or pessimist? • Solutions rarely available on demand • Cost, lead times, reliability, access to skills……. • Intermittent need Non-Functional Testing

4. Oh what to do, what to dooooo? Non-Functional Testing

5. Well, we could……… • Maintain an idealistic view • Just complain • Do nothing and accept the status quo • Talk to other TMs and agree that nobody loves us • Develop some pragmatic solutions based on………. Non-Functional Testing

6. Genuine Need, Realistic Demand and Access to solutions (tools/skills/services) • Attitude to risk • Commercial/operational drivers • Attitude towards testing • Centralised/devolved organisation • What can be realistically delivered • Attitude may be influenced by: • Past experience • The way you provide advice, guidance & solutions Non-Functional Testing

7. So can we build a Toolkit to Help? Possibly but………. • A toolkit can only be used on a defined problem. Are you looking for, analysing or solving non-functional defects? In addition you may need: • A strategic objective to drive the scale/scope • Demand to use your toolkit • Where will the non-functional requirements come from? • An interface to development projects • A strategy for delivering your toolkit - self-service, internal or 3rd party • Scalability and demand management • A delivery policy/mandate • A means of payment Non-Functional Testing

8. Does your toolkit address the causes of inertia…. • Does it define need and improve understanding? • Does it provide a complete solution? • Is it cost-effective? • Does it deliver value-add? • Is it available when needed? • Virtual NFT – the power of suggestion Non-Functional Testing

9. The End of the Beginning Non-Functional Testing