1. Exploratory Testing Or how to find bugs faster…

2. Summary • What is Exploratory Testing? • Why are we doing black box testing? • Session Based Testing • Where do we test? • Types of testing • How outlines can help • How flow charts could help

3. What is Exploratory Testing? • Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution. CemKaner, who coined the term in 1983,[1] now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."[2] • While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run. Exploratory testing is often thought of as a black box testing technique. Instead, those who have studied it consider it a test approach that can be applied to any test technique, at any stage in the development process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement of the tester, and the tester's responsibility for managing his or her time.[3] • - http://en.wikipedia.org/wiki/Exploratory_testing

4. Exploratory Testing • is not just adhoc and saying adhoc is a turn off to people who don't understand exploratory testing. Why? True adhoc is literally monkey pressing buttons. • … is a learning process. • The code will change as the project progresses. Doing the same thing over and over again has diminishing returns • Not a replacement for doing/running test cases

5. Exploratory Testing • Is not a replacement for automation testing • It’s a supplement to reduce the risk of the unknown quantity of bugs • Comment : I would implore various strategies of depth and breadth type testing in trying to reduce the risk of an unknown quality and severity of bugs to make them known.

6. Why do we do blackboxtesting? • Because human thoughts and behavior changes from individuals as well as over time. We don’t necessarily do the same thing over and over again like automation testing does • Bottom line : people think. And people think differently.

7. Human Cognitive Behavior • Be aware of yourself and why you may miss bugs: • http://viscog.beckman.illinois.edu/media/mistakes.html • http://viscog.beckman.illinois.edu/flashmovie/15.php

8. Testing • Be a scientist! Poke around! • Try to figure things out • Go beyond just doing the test case… • Talk to each other about it, learn from each other • If you don’t understand the bug, then how do you expect other people to? • It’s like chess, try to think in steps ahead • Test cases aren’t the rule book; they are a guideline • The more you test the more experience you will gain. You can learn from any source; I often learn from people I have mentored as much as I’ve learned from the people who have taught me!

9. Session Based Testing • Session-based testing is a software test method that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. The method can also be used in conjunction with scenario testing. Session-based testing was developed in 2000 by Jonathan and James Bach. • Highly recommend looking at this… this is what top blackbox testers do in finding bugs as well as other things discussed later on.

10. Where to test? • New Code Change • Looking at code change • Trends in bugs • Talking to Dev; what might they be worried about? • Talking to other QA; what seems fragile? • Risk areas • What’s dangerous if it fails for human safety • Embarrassment (functionality) • Complexity • untested

11. Types of testing • MVC = Model View Controller • UI Testing • Functional/Business Logic testing • Data/Input Testing • UI Testing • Does user flow make sense? (Usability) • Culturally effective? Which culture? • Does it look cluttered?

12. Types of testing • Functional Testing (Positive Testing) • Negative Testing • Boundary Testing • Race Conditions • Environment conditions • Device specific • Localization • Initialization • Load/Endurance • 508 Compliance

13. Examples • Functional Testing (Positive Testing) • Use Cases and specifications • Negative Testing • Error handling, like bad email and incorrect website address • Boundary Testing • Max characters, or with integers -1, 0, 1, max -1, max, max + 1 • Race Conditions • Running music app, video app, and getting a phone call at the same time

14. Examples • Environment conditions • Rural area testing of telephony vs urban • Device specific • Battery running low, graphics issues • Localization • Date format, starting day, fitting in ui, punctuations • Initialization • Start up of device, FTE, start up of apps versus OTA and continual usage

15. Examples • Load/Endurance • Loading tons of facebook, versus keeping music app running for days on end • 508 Compliance • Handicap ie : blind, deaf user interfaces

16. How outlines/test plans can help • From 1.0 • https://wiki.mozilla.org/B2G/QA/Gaia_Test_Plan/Browser • https://wiki.mozilla.org/Releases/Firefox_25/Test_Plan • https://wiki.mozilla.org/QA/Fennec/Milestones/2.0/TestPlan:Virtual_Keyboard/Android_VKB • https://wiki.mozilla.org/QA/Fennec/24/Beta/8

17. B2G Specific Example • https://docs.google.com/spreadsheet/ccc?key=0Arku3jleCA0UdF9ZZWtyYm9iV1NLOGRTWkQzWHFsS2c#gid=5

18. How flowcharts can help • Mapping out user flow • Demo

19. Resources • QA conferences • They can be pricey and advertisey • QA newsgroups/forums/websites • QA magazines/books/articles • http://en.wikipedia.org/wiki/Software_testing • http://www.amazon.com/exec/obidos/ASIN/0471358460/softwarqaandtest

20. B2G Architecture • https://wiki.mozilla.org/B2G/Architecture • https://github.com/sotaroikeda/firefox-diagrams/wiki/Firefox-Diagrams • GitHub: • https://github.com/mozilla-b2g/gaia/commits/master • Compare to sources.xml file : https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-central-unagi/latest/sources.xml • Tool for searchingcode • https://mxr.mozilla.org/