Thoughts on Systematic Exploratory Testing of Important Products

1. Thoughts on Systematic Exploratory Testingof Important Products James Bach, Satisfice, Inc. http://www.satisfice.com James@satisfice.com

2. Exploratory Testing and Risk • If the stakes are high, how can you justify using unproductive approaches to testing? • Highly scripted testing means testing without using your brain. This is unproductive. • If you are using your brain; if you are designing tests throughout the project based on all the information available; then you are doing exploratory testing.

3. Exploratory Testing and Risk • I predict that within ten years it will widely be considered reckless to ship a product that has not been tested systematically using exploratory methods. • Exploratory testing is extremely productive. All testers do it. The problem is that very few take it seriously, or have any training in it.

4. I teach testing as a sort of martial art. • The “questions” consist of various ways of configuring and operating the product. (Asking questions about the product without operating it is usually called review rather than testing) • The product “answers” by exhibiting behavior, which the tester observes and evaluates. • To evaluate a product is to infer from its observed behavior how it will behave in the field, and to identify important problems in the product. Testing is questioning a product in order to evaluate it.

5. Exploratory Testing Defined Definition Exploratory testing is simultaneouslearning, test design, and test execution. freestyle exploratory pure scripted fragmentarytest cases vague scripts charters roles When I say “exploratory testing” and don’t qualify it, I mean anythingon the exploratory side of this continuum.

6. Exploratory Testing Can Be Done… • …on an idea before it’s a product. • …on a product to model it for test planning. • …to augment or improvise upon scripts. • …to investigate a problem uncovered by some other test process. • …to test a product after changes are made. • …as the principal method of finding problems on a complex product.

7. Explore the meaning of requirements. • How do you test this requirement? • “When the user presses a button on the touchscreen, the system shall respond within 300 milliseconds” • Possibilities: • Performance analysis instrumentation? • Use a stopwatch? • Estimate it by eye? • Just report whether the product seems annoyingly slow? • Key questions: • What does this requirement really mean? • What risk corresponds to it? • Why is it written that way? • How important is it compared to other requirements?

8. Explore implications and scenarios. • How do you test this requirement? • “The movie [of x-ray images] shall be played at the frame rate set by the frame rate slider control.” • Possibilities: • Performance analysis instrumentation? • Use a stopwatch? • Estimate it by eye? • Key questions: • No tolerance is given. How close is close enough? • How much variation is acceptable? • Could a doctor come to a false diagnosis because of problems with this feature?

9. Things I explore when I play with a product. • Composition • Affordances:Ways the product can be used. • Dimensions & Variables: Product space and what changes within it. • Relationships & Interactions: functions that cooperate or interfere. • Navigation: Where things are and how to get to them. • Conformance • Benefits: What the product is good for-- when it has no bugs in it. • Consistencies: Fulfillment of logical, factual, and cultural expectations. • Oracles: Specific mechanisms or principles by which you can spot bugs. • Bugs and Risks: Specific problems and potential problems that matter. • Context (of the Product) • History: Where the product has been, and how it came to be. • Operations: Its users and the conditions under which it will be used. • Conditions (of Testing) • Attitudes: What your clients care about and what they want from you. • Complexities & Challenges: Discover the hardest things to test. • Resources: Discover tools and information that might help you test.

10. learning • Model Product • Identify Risks • Select Coverage • Determine Oracles • Determine Operations design • Understand Testing Mission • Understand Project Environment • Understand Risk Factors & Areas • Understand Product & Problem Domain • Configure Test Lab planning Exploration is a feedback loop • Configure • Operate • Observe • Evaluate execution Follow Risk Rumors

11. “How do you organize your work?” "cidtestdsfdpocruspicstmplfdsfscura" Function testingDomain testingStress testingFlow testingScenario testingClaims testingUser testingRisk testingAutomatic testing CapabilityReliabilityUsabilityScalabilityPerformanceInstallabilityCompatibilitySupportabilityTestabilityMaintainabilityPortabilityLocalizability StructuresFunctionsDataPlatformsOperations CustomersInformationDeveloper relationsTeamEquipment & toolsScheduleTest ItemsDeliverables

12. Analyzing Non-Routine Risks isAbout Learning, not Calculating • This learning… • is a heuristic process, not a rote procedure. • unfolds over time. • involves many different sources and people. • is a model-building process. • involves comparing risks predicted to risks manifested. • is highly social and psychological. • is largely self-correcting. Testing is a core part of that learning process.

13. Develop Your Skills! TESTING SKILL CONSISTS OF THE SKILLS OF • General Systems Thinking • Critical Thinking & Research Methods • Rapid & Exploratory Learning PLUS KNOWLEDGE OF • Cognitive Science & Human Factors • Decision & Utility Theory • Folklore & Heuristics (most testing textbooks are folklore) PLUS • Context-Specific Technology and Social Skills