Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests

1. Large-scale Test Automation Production Test Automation is No Longer Optional. Automated Exploratory Tests or Structured Automated Tests Which One is Better and Can You Do Both? June 16th, 2010 – San Mateo, CA Cem Kaner - Hung Q. Nguyen Moderated by Michael Hackett

2. Common Challenges • Automation Technology • Multiplatform (cross-platform) support • Ability to interface with multiple platforms • Ease of extensibility to new platforms • Reusability of test suites/scripts across multiple platforms • Output/Object validation support • High rate-of-change • Interface changes • Functionality changes

3. Automation Cost of Ownership • Technology = Tool Licensing + Development/Customization Cost • Production Cost = *One-time Cost + **Recurring Cost *NEW–One-time Cost of Design, Implementation & Execution • Test Case or Test Script Creation • Test line creation • Sequencing test logic or order of test lines (test steps) • Testing and debugging test script • Function or Keyword Creation • Interface Capturing or Mapping • Test Dataset Creation **EXISTING—Recurring Cost Test Execution & Maintenance Cost • Monitoring test suit execution • Investigating and troubleshooting false negatives • Maintaining Test Case or Test Script • Test line modification • Updating test logic or re-sequencing order of test lines (test steps) • Testing and debugging updated test script • Function or Keyword Modification • Interface Recapturing or Remapping • Test Dataset Modification

4. Large-scale Automationis Method Dependent Automation is Method-Centric—it is more than about the technology; it is about the effective application of technology • Maintainability—handling high rate-of-change • Reusability—ability to reuse common test components without having to program/code • Scalability—ability to automate high volume of tests through reusability (and team-based cost efficiency) • Visibility • Tests auditable by management and non-programmer staff • Productivity is structurally measurable

5. Efficiency is Key! Efficiency Key Contributors • Test design Definition: A test (case) creation activity with intent to serve one or both of the following objectives: • Exposing bugs/errors (or show otherwise, it works as intended), and • Optimizing maintainability and scalability (especially for automation-readiness). • Test automation (large-scale) • Method-centric, not tool-centric • Team-based, not individual coders

6. Rules of Automation Built-to-Last • Tests are treated as product asset, along with the source code. • Tests, good or bad are dependent on the design. • Tests, manual or automated must be optimized for visibility, reusability, scalability and maintainability. • Tests must be automation-ready. • Tests, if they are worth automating, should follow the 5% rule: • No more than 5% of all tests should be executed manually • No more than 5% of all efforts around testing should involve automating the tests • No more than 5% of coded test scripts against non-coded test scripts.

7. Automation Strategy Must Include • Efficient Practices • Technology • High extensibility • Object validation capability • Methodology • High maintainability (low maintenance cost) • High reusability • People and process • High scalability (high-volume and team-based methodology) • Just-in-time automation—test engineers are skilled at test design and automation can be done before code-complete and/or concurrent to exploratory testing • High manageability (high visibility)

8. "Agile" System Development • Establish objectives (global) • Develop tests • Develop the system • in iterations, one subsystem at the time • test-driven • Release • Agile “Home Ground*” • Low mission-critical • Senior developers • Requirements change often • Small number of developers • Culture that thrives on chaos * Barry Boehm and Richard Turner Balancing Agility and Discipline: A Guide for the Perplexed Addison-Wesley, 2004

9. Role of Automation • Effective automation can allow tests to support agile system development • Automation should not dominate. Don't make it into an "agile automation" project • A keyword driven method is, in my view, essential for successful test development w/ automation in agile development • Automation is separate from test development • Usually doesn’t need user interaction (only some with testers) • Automation focuses on actions, not on tests

10. TEST DEVELOPMENT Testers Stakeholders SYSTEM DEVELOPMENT Developers Stakeholders Agile Test Development AUTOMATION Tests can drive the development

11. Recommendations • Have a global test-design scheme • Keep it short and simple: focused on breakdown of the tests • This global test design can (and usually should) be revisited throughout the project cycle • Separate (1) test objectives and (2) test cases • Test objectives are easier to create/describe than test cases • Use your stakeholders (specialists, power-users, etc.) wisely • Focus the efforts on the relevant tests • Don’t bother them more than needed • Avoid involving them in low- and medium-level tests • Observe various types of input: • How does this work? • What do we need to test? • What is interesting to test/How can we break the system? • Keep automation separate • Test developers are not consumed by automation • Use keyword-driven approach (e.g., Action Based Testing)

12. PEOPLE & PROCESS Production Layer FRAMEWORK ARCHITECTURE Test Layer Low-level / System Action High-level / Application Action Extensibility Integratability Automation Layer EAS1 EAS2 EAS3 EAS4 EASx 3rd-Party Tools Device x SUT / AUT • Communication protocol • Interface/Object recognition • Object content validation EAS: Extended Automation Support

13. Scalability Illustration 6000 250 3000 200 22 20 Application (High-level) Actions Built-in/System (Low-level) Actions Tests

14. Division of Work Note: Programming skill is required for Automation Engineer

15. Key Takeaways • Fully understand automation cost-of-ownership • Don’t underestimate the challenge of keeping maintenance costs low • You need to get efficient—optimize your volume of test to exceed 50% coverage • Efficiency is key, and it will come from excellent test design and automation methodology (e.g., action-driven), and a well architected framework technology • Minimize programming tests • High scalability comes from high reusability of common “actions” and team-based staffing model • High maintainability comes from keeping maintenance activities at the lowest level • High visibility in your automation program to give you control and measurability, which ultimately leads to manageability • Practice just-in-time automation • Have a global test-design scheme that separate test objectives from test cases; and automation from test development

16. Appendix Action Based Testing example

17. TA Agent Example withTestArchitect™ (TA) TA Client TCP/IP Wireless Mobile API

18. Test Example

19. Action Definition for “send sms message and check” action

20. TA Connects to Window Mobile via TCP/IP TA TCP/IP Wireless

21. Agent Connects to Device via Mobile API Mobile API

22. Interface Definition for “message editor” Entity

23. About LogiGear Corporation LogiGear is a world leader in high-volume software test automation production, serving hundreds of companies, from Fortune 100 to startups. Draw from 15+ years of research and practice, 5 books, including one bestseller, 18 pragmatic training courses, a proven automation methodology and a software framework that elevates automation test coverage to exceed 95% on a diverse set of technology platforms, and 400+ skilled engineers in the US and Vietnam, it delivers unbeatable value that precisely meets each customer’s specific needs. hungn@LOGIGEAR.com