Driving Test Automation through the Continuous Deployment Process

1. Driving Test Automation through the Continuous Deployment Process Adam Guida

2. Introduction Adam Guida • Senior QA Automation Developer Work History: • 2011 – Present: • Deutsche Bank Global Technologies Cary, NC • 2006-2011: • lulu.com - Raleigh, NC Email: guidadic@gmail.com

3. Automated Testing is a Liability Unless you have Stable • Test Environments • Deployment Automation Strong • Communication • Collaboration • Technical Expertise

4. The Truth About Automated Testing Automated Testing: • When done correctly, it can significantly increase the repeatability and effectiveness of testing efforts • Augments classical test methodology but does NOT replace manual testing • Increases regression consistency • Reduces tedium • Allows human testers a greater focus on new and creative ways of breaking things However: • Automated testing is ineffective without an approach that considers all facets of the development automation ecosystem

5. Purpose of this lecture • Lessons Learned • Poorly written test • automation isn’t sustainable • Writing good tests alone • isn’t enough • Continuous Automation in QA • Test Strategies • Best Practices • Environments • Tools & Technologies • Communication

6. Continuous AutomationPractices Continuous Integration (CI) • A practice where code is automatically compiled and unit tested on every developer code check-in Continuous Deployment • If CI phase passes, application(s) are automatically deployed to one or more pre-production environments, triggering additional layers of automated testing

7. Continuous Automation Sweet! All tests passed! Unit test failed! I’ll take a look at what broke and try again public class Feature { private int 1; public void doSomething(){ if (a == b){ return true; } My bad! I’ll check-in a fix for that right away. Latest build is up on CI for testing! Awesome! Let’s grab some lunch! Feature: Test A and B are True Scenario: Verify that A Equals B Given I open the “Feature” When I click on “A” and “B” Then the result is “…….” Hey, can you take a look at the last run? Looks like a regression test failed. This one is my fault. Looks like I forgot to adjust some tests after that last build. I’ll check-in a fix & restart the automation DEV & QA Test SCM App SCM Let me take a look. I see! You’re calling the wrong resource. I’ll IM you the correct name Dude, I’m having a problem getting this test working. The web service is returning a 404 Automation Pipeline AutomatedTesting Build + Unit Test Deploy Report Test Environment

8. Continuous Automation Collaboration • Establish partnerships & comradery with your development team • Treat QA & Developers as a single team • Close low-level knowledge gaps in functional testing • Develop workflows for how DEV & QA address test failures as a team • Foster a culture that values cross-team learning, exchange of knowledge and professional growth

9. Continuous AutomationBest Practices Demo

10. Continuous AutomationBest Practices Automated Test Suite should be run as often as possible • Automated tests should run……………………...automatically • On every successful automated deployment • Automation executes using the fewest clicks possible • Start Small and Simple – It won’t all happen overnight

11. Automation Test Team:Strategies • Delegation - Separate the technical from test responsibilities for a given task • Increases efficiency and focus by reducing context switching • Pair-testing - Pair technical & non-technical testers together for greater impact • Increases technical experience and promotes mentorship of non-technical testers • Increases usability & readability of tests • Reporting - Test Lead/Manager - keep a close eye on automated test reporting. • If tests start failing, seek to understand the underlying causes, then try to assist test team by removing roadblocks • Understanding daily/weekly trends in test results helps guide risk assessments

12. Continuous AutomationBest Practices Treat Your Test Suite like an Enterprise Application Automated testing *IS* a form of programming and bound by the same laws as software development • Strive to be the best programmer you can & adhere to software development best practices • Use a Source Control Management tool • Unit Test your test automation (Yes, that means testing your tests) • Practice Code reviews • Seek architectural guidance and input from your development team • Defect tracking (automated tests apps get bugs too!)

13. Automated Test Strategies The Perfect Automated Test • Fast – Speed impacts every aspect of a quality automation suite • Reusable – Reuse reduces code complexity and increases simplicity of a test suite • Relevant – Test indicate a problem we care about and is not redundant • Independent – Is self sufficient and does not depend on the ordering of other tests • Focused – Tests a specific aspect of the application without being too broad • Clear – Determines a pass or failure result automatically, ideally narrowing down the exact cause of the failure • Understandable – Is written in such a way that documents behavior other stake-holders can understand • Flexible – Change is the only constant – write tests in ways that can handle changes within reason

14. Automated Test Strategies Managing the Complexity Versus Coverage Tradeoff • System Integration / End-to-End –Provides the most comprehensive test coverage at the cost of speed and complexity. • Unit – Isolated testing of individual classes, methods and functions • Component – Tests are isolated to the component itself. Inputs/Outputs are simulated through test adapters or mock

15. Automated Test Strategies Example of Component Testing End-2-End Data Transformer Testing TEST 1 MOCK DATABASE H2 User Database Marketing Aggregator Web Server Browser WEB SERVICE CALL Product Database Order Database TEST 2 TEST 3 Web Service Testing MOCK AGGREGATOR UI Testing

16. Automated Test Strategies Managing the Complexity Versus Coverage Tradeoff • Diversify your test portfolio using a combination of Unit, Component and System testing • Promote Unit Testing w/ Devs - allows you to start trusting some basic elements of the system, so you can focus on more challenging test scenarios • Minimize System/End-to-End tests - to fewer high-level smoke tests based on areas of importance • Maximize your Component Testing - focus on lower level functionality, API’s and data model interactions • Use Test Adapters – for both greater control & reliability of the application input/outputs

17. Choosing the right Tools & Technologies • Don’t Reinvent the Wheel – someone has already solved your problem • Carefully Evaluate All Technologies – Require experience prior to adoption. Invest time in evaluating the pros & cons of a variety of different solutions. Build POC’s and give stakeholders (developers) a role in decision making • Straightforward Integration – Utilize tools which accept open formats for read/write – ensure each tool can fully integrate with others as necessary • Active Community of Users – Favor technologies where an active online support community exists over managed (paid) support solutions

18. TechnologiesProgramming Language Rule of thumb: Test Automation should be based on the same programming language of your application • Enables sharing of development resources, support and contribution • Shared libraries and other resources between testing & dev • Should be capable of utilizing a highly functional IDE’s such as Eclipse, Intellij, Visual Studio, etc

19. Tools: IDEs (Integrated Development Environment) Intellij IDEA / RubyMines • Fully integrated support for Cucumber • Syntax highlighting • Code Editing/Refactoring • Supports many languages • Build-in database browser • Debugger

20. Tools: Continuous Integration Server • CI (Continous Integration Server) More than just a build agent • Manages the end-to-end flow of your automation pipelines and reporting • Is both the glue and front-end that binds the continuous deployment, testing and reporting processes

21. Tools: VMs - Virtual Machines • VirtualBox • Virtualized OS’s • Linux • Windows • Android • OS X • Contained sandbox environments • Backup/Restore, share and take snapshots of your VM environments

22. Tools: • Create and configure lightweight, reproducible, and portable development environments • Easy to setup/tear down and deploy to a standardize test environment • Works with VirtualBox and other virtual environment products • Support integrated into Intellij IDEA via plug-in

23. Tools: Deployment Orchestration • Help automate aspects of environment provisioning & deployment • Install & Configure Servers & other resources • Helps facilitate continuous deployment of your application • Many great frameworks to choose from - each has it’s own set of strength & weakness

24. Tools & Technology: Database Management • Treats SQL like version-able code • XML, JSON, YAML and generic SQL syntax supports multiple database technologies • Flexible runtime: Maven Plugin and standalone support

25. Final Thoughts • Automation is an investment, that when managed effectively, provides a particular set of benefits which increase efficiency and repeatability of testing. AUTOMATION IS NOT A BENEFIT IN ITSELF • Treat automation as a 1st class citizen of your project • If you’d like a copy of this presentation, please send me an email at: guidadic@gmail.com