Structured Exploratory Testing – Method in Madness

1. Structured Exploratory Testing – Method in Madness Vivek Devarajan Senior Consultant, Wipro Technologies

2. 1 Scripted versus Non-Scripted Testing Some Exploratory Testing Ideas Session Based Test Management Benefits of Exploratory Testing Case Studies 2 3 4 5

3. Scripted versus Non-Scripted Testing

4. Scripted versus Non-Scripted Testing Leveraging the benefits of both Testing Styles Madness Methodical • Elaborate Documentation • Static Test Suite • Higher Testing Costs • Limited Defect Detection • Concise Documentation • Dynamic Test Optimization • Lower Testing Costs • Better Defect Detection • No Accountability • No Predictability • High Possibility of Defect Slippage Scripted Testing Freestyle Testing Structured Exploratory Testing / Session Based Test Management

5. Some Exploratory Testing Ideas

6. Exploratory Testing using Question Patterns • While testing a system, how often have you felt “I have tested something similar before?” • The Domain Factor • On the other hand, how often have you been bogged down by an application, just because it was from a different domain that you are not so familiar with? • The Reuse Factor • Is there a way to develop testing ideas once and reuse them across domains / applications? • The Knowledge Factor • Is there a way an experienced tester or domain expert can impart his/her experience to others? • The Exploratory Testing Context • Is there a way to explore the application systematically and find bugs?

7. Question Patterns and Exploratory Testing • Question Patterns: A Set of interrelated Questions grouped together, focused on an aspect of the system under test that • Enable someone to understand the application better • Can be used as a technique for requirements capture • Patterns help the tester explore the application systematically and find bugs • Example: Parent and Child Entity Pattern • Scenario A).Consider an online shopping application where the user can add / modify / delete multiple items in his shopping cart order. In this case, the shopping cart is a business entity which has multiple child entities (items) within itself (a one-to-many relationship between the cart and the items) where the child entity (item) can be added/updated or deleted. • Scenario B). Consider an application where the user can apply for an insurance policy online, to which he can add / modify / delete multiple beneficiaries. In this case, the policy is a business entity which has multiple child entities (beneficiaries) within itself (a one-to-many relationship between the policy and the beneficiaries) where the child entity (beneficiary) can be added/updated or deleted.

8. Question Patterns and Exploratory Testing • Questions that can be asked: Parent and Child Entity Pattern • What are the business entities involved? • What is the relationship between them? (One to Many, One-to-One, etc) • What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search) • What are the field level and business validations for each entity? • Answers to such questions enable testers to find gaps in requirements (or even capture requirements). • Answers to such questions also serve as a guideline and a checklist during exploratory testing.

9. Question Patterns and Exploratory Testing • Questions that can be asked: Parent and Child Entity Pattern • What are the business entities involved? • What is the relationship between them? (One to Many, One-to-One, etc) • What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search) • What are the field level and business validations for each entity? • Answers to such questions enable testers to find gaps in requirements (or even capture requirements). • Answers to such questions also serve as a guideline and a checklist during exploratory testing.

10. Exploratory Testing by Looking for Specific Problems • Purpose of Testing – Looking for Bugs/Problems that could impact the stakeholders • Defect Seeking v/s Defect Finding • Make a list of possible issues • Design tests to find ways to stimulate the possible issues identified

11. Exploratory Testing by Analyzing Data Flow • Most Applications process / manage data • Identify the input and output data • Verify the output data for various combinations of input data (including green path values, invalid values, boundary conditions, large values, etc) • Try modifying the data at various stages in the flow and see the impact • Identify the right combinations / representatives of data

12. Exploratory Testing by Analyzing Flows • Identify end-to-end flows and branching • Depict the branching graphically, if needed • Try parallel flows • Try varying the sequencing / timing of the flows • Analyze the impact of input data values on the flows • Check the main flows as well as alternate/exceptional flows • Check for state transition flows

13. Session Based Test Management – Accountable and Manageable Exploratory Testing

14. Session Based Test Management - Approach • Test Execution is divided into time-boxed Test Sessions, where the tester does (uninterrupted) exploratory testing with a specific charter • Tester takes brief notes about the testing done and the issues found • Tester uses a checklist of testing ideas as a guide for testing • Further tests are dynamically optimized depending on the results of the current test • The Test Lead conducts a ‘Debrief’ session with the testers on the test findings and plans further executions based on the findings

15. Session Based Test Management – Session Contents • The contents of a typical test session are as follows: • Charter: Contains the mission of the given session. Typically it mentions the application areas to be covered / explored, at a high level. • Tester name, Session Time Start and End • Task Breakdown: Time taken for Test execution v/s Bug Reporting, Charter (Original Session Charter) v/s Opportunity (Diversions taken) • Test Notes: Notes of what the tester executed, in brief. • Bugs/Issues found

16. Session Based Test Management – Accountability • Tester’s Activities: Tracked via Test Session reports • Coverage and Testing Progress Tracking: • High Level Coverage: Number of functional areas and quality criteria covered • Granular Coverage: Number of checklists/items/test ideas covered for each functional area • Traceability: Each test idea and item in the checklist can be mapped to requirements

17. Benefits of Exploratory Testing

18. Exploratory Testing - Benefits • Lesser Preparation and lesser effort • Quicker Bug Detection • Leverages on Tester’s Abilities, experience and creativity • Works even if requirements are not fully frozen

19. Skilled Testing v/s Deskilling – The Classic Debate • Deskilling via Detailed Test Cases leads to higher test design and maintenance costs and limited bug detection capabilities • Testers with lesser experience can be trained in skilled exploratory testing • Skilled Testing is more cost effective and results in better product quality

20. Case Studies

21. Case Study 1 – The Problem • A Police IT Application for a Government Customer • The Context: • Previous customer concerns over quality in beta releases • Large volume of functional tests to be executed in a short time span • - Conventional Testing Estimations: 3000 test cases to be designed and executed in 2 months (including KAP phase), requiring 14 testers • Inexperienced Testers with no domain knowledge • Customer more focused on bugs/issues than in testing processes

22. Case Study 1 – Testing Approach and Results • A Police IT Application for a Government Customer • Question Patterns used to arrive at checklist of testing ideas • Entity Relationship pattern • State Transition Pattern • Master Data – Transaction Data Pattern • Checklists consisting of Business Conditions and Testing Ideas • Initial Test Sessions focused on exploration of the business operations of entities • Deep Dive Test Sessions focused on finding specific problems • Test Sessions for End to End Testing focused on Application Flow and Data flow • Results: 65% reduction in testing efforts and 60% better defect detection

23. Case Study 2 – The Problem • An Order Management Application for a Healthcare customer • The Context: • The application is intended to enable healthcare providers and pharmacies to place orders quickly • Bugs not being found early enough • High cost of testing • Frequent patch releases with incomplete functionality • Mix of experienced and inexperienced testers, with fair domain knowledge • Parallel run of conventional testing and Session Based Test Management

24. Case Study 2 – Testing Approach and Results • An Order Management Application for a Healthcare customer • Question Patterns used to arrive at checklist of testing ideas for regression testing • Checklists consisting of Business Conditions and functional changes to be checked for • Initial Test Sessions focused on the exploration of new functionalities • Subsequent Test Sessions focused on user experience • Results: 50% reduction in test design efforts and quicker defect detection

25. Thank you Vivek Devarajan Senior Consultant, Wipro Technologies vivek.devarajan@wipro.com

26. Appendix

27. Artifacts Created • Checklists Created for Session Based Test Management: • Sample Test Session:

28. Session Based Test Management SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles Exploratory Testing Freestyle Testing Scripted Testing Session Based Test Management

29. Session Based Test Management SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles

30. Comparison of Structured v/s Freestyle Testing Scripted Testing Freestyle Testing Benefits: • Very Low Costs for Test Creation/Maintenance • Highly effective in quick Defect Seeking, especially effective for tricky bugs • Does not need well documented requirements Benefits: • High Predictability • Deskilling (Test Execution) • High Accountability Drawbacks: • Very High Costs (Resources + Tools) for Test Creation as well as Maintenance • Limited Defect Detection Capabilities (Lack of Proactive/Creative Approaches) • Needs well documented requirements Drawbacks: • Low Accountability • Relies on the creativity of the Tester

31. Session Based Test Management SBTM - Leveraging the benefits of both Testing Styles Scripted Testing Freestyle Testing • Elaborate Documentation • Static Test Suite • Higher Testing Costs • Limited Defect Detection • Concise Documentation • Dynamic Test Optimization • Lower Testing Costs • Better Defect Detection • No Accountability • No Predictability • High Possibility of Defect Slippage Automated Testing Session Based Test Management Exploratory Testing

32. Evolution of Session Based Test Management Session Based Test Management Exploratory Testing Evolution Adhoc / Freestyle Testing

33. Salient Features of Session Based Test Management • Application Learning, Test Design and Test Execution happen in Parallel • A Structured, measurable and accountable method of Exploratory Testing • Crisp Checklists to guide the tester on what needs to be tested • Execution happens in Test Sessions (Uninterrupted Test Execution Time) with specific charters, after which tester takes notes and makes a report • Tester takes a different execution route, if required, depending on issues found in a Test Session • Future test sessions are planned based on results from current session

34. Metrics for Session Based Test Management Test Session Metrics Execution Metrics Defect Trend Cumulative Defect Trend