1 / 39

It’s Automation, Not Automagic !

It’s Automation, Not Automagic !. …or proving Einstein was right about insanity . Presented by Jim Hazen. Introduction. Jim Hazen Veteran of the Software Testing Trenches Experience in software testing, both commercial and consulting work. Working with automation and tools since 1989.

avidan
Télécharger la présentation

It’s Automation, Not Automagic !

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. It’s Automation, Not Automagic! …or proving Einstein was right about insanity Presented by Jim Hazen

  2. Introduction • Jim Hazen • Veteran of the Software Testing Trenches • Experience in software testing, both commercial and consulting work. • Working with automation and tools since 1989. • Implemented Automation Tools on DOS, OS/2, Windows and Web platforms. • Experience with tools; including HP/Mercury, IBM/Rational, and other ‘scripting’ languages.

  3. Sage Words to Remember • “Insanity: doing the same thing over and over again and expecting different results.” – Albert Einstein

  4. Agenda • What is ‘Test Automation’ • A Brief History • Different types of test automation, tools and uses • Automation Myths • How to correct them • Resources & Skills needed • Setting expectations for Management & Others • Summary • Q&A

  5. What is Test Automation?

  6. It’s a Big Topic to Discuss

  7. Bigger Than We Think!

  8. Is It This?

  9. Or More Like This?

  10. Who Builds the Automation?

  11. Some Are Seen This Way

  12. Most Look Like This Instead

  13. What ‘Test Automation’ Really Is • “Automation” definition • Webster’s – “the technique of making an apparatus, a process, or a system operate automatically” • In Software Testing we use ‘tools’ to drive our processes • People only think of tools that execute a test as ‘Automation’ . • Other tools can be used for Test Design, Test Data Generation, Defect Tracking, Test Management, etc. • “Automation” is using a tool during any part of the process of testing.

  14. Who Test Automator’s Really Are

  15. A Brief History • Test “Execution” tools have been around a long time! • Mainframe – Home grown ‘macro’ tools1, Hyperion • PC DOS – AutoTester, SuperKey (macro tool) • PC GUI – Automated Test Facility (ATF), Rational Robot, QA Run, Winrunner, etc. • PC Web – QTP, SilkTest, Selenium, TestComplete, etc. • Mobile – MonkeyTalk, Robotium, etc.

  16. Other Types of Test Automation Tools • Other Test Execution Tools • xUnit (JUnit & NUnit), SOAPUI, FitNesse, etc. • Test Design & Test Data Generation • Hexawise, AllPairs, Tosca Test Suite, etc. • Test Execution Management • HP ALM, MS Team Foundation Server, TestLink, etc. • Test Environment Management • VMWarevCloud Director, etc. • Defect Tracking & Reporting • Bugzilla, JIRA, etc.

  17. Automation Myths & Misconceptions

  18. Old Myths • You can automate 100% of your tests • Automation will “cure” all problems with testing • Automation tools are usable by anyone • Record/Playback is all you need • Automation allows Management to cut test staff • One tool is all you need • Automation will speed up testing process * Oldies, but goodies!!

  19. New Myths • You don’t have to know how to program • Opensource tools are “free” • Commercial tools are “expensive” • GUI level automation is a waste of time • “Codeless” and “Scriptless” tools • Commercial tools can’t support agile • Developers only need to focus on Unit Tests

  20. How to Correct Them • 100% Automation – Certain tests shouldn’t or can’t be ‘automated’. 80/20 Rule; automate 20% of functionality used 80% of time. • Automation solves all testing problems – it helps, but it creates new problems too. • Automation can be done by anyone – Not everyone has the ‘mindset’, it is both learned and inherent (logic). • Record/Playback is all you need – Proven time & time again to be false, and to fail.

  21. How to Correct Them cont. • One tool is all you need – There isn’t a one-size fits all. • Automation speeds up testing process – The “Illusion of Speed” problem. Both execution and creation. • You don’t need to know programming – Anyone can Automate problem. • Opensource tools are free – Costs due to ramp up time (money) and skilled staff to do it (more expensive).2 • Commercial = Expensive – Not really in grand scheme of things.2

  22. How to Correct Them cont. • GUI level automation is a waste of time – Not really

  23. How to Correct Them cont. • Codeless/Scriptless tools – Just a different UI. You’re still writing a ‘test’ which is a set of ‘instructions’ (code). • Commercial tools don’t do Agile – Yes they do, it’s how you use the tool that counts. • Developers only create Unit Tests – Other tools require someone to ‘build’ the fixtures/code to ‘talk’ to the SUT.

  24. Why Failures Happen Source: IDT Survey3 85% Failure Rate! Too High!!

  25. Einstein was right! • We keep repeating history, why? • Always a ‘new’ group of management and testers. • Companies always looking to do testing faster, cheaper and with less resources. • Vendors and others who sell “Automagic”. • Instant gratification mindset. Automation takes time. • Not learning from past mistakes. • Software not designed with “testability” in mind. • Testers not involved early with decisions, not included by management.

  26. Resources for Automation • Dedicated Staff • Skilled & experienced staff • Dedicated Equipment/Environments • Reduce risk of competition of resource or teams stepping on each other • Dedicated Budget • Automation isn’t cheap, plan appropriately • Dedicated Management Support • Can’t have any of the other 3 without it

  27. Skills for Test Automation • It’s a different view and purpose for the script code. • Creating ‘software’ to ‘test’ software; i.e., PROGRAMMING! • Need basic understanding of programming methods. • Understand different types of technology layers • GUI, API, System, Web, Services, Database, etc. • You need to know how to program in different languages when needed. • C#, Java, VBScript, Jscript, XML, SQL, etc.

  28. Test Automation Skills Cont. • Need to understand how to ‘hook’ the software under test and interact with it, and at what level. • API, or Object, or Service, or Protocol, etc. • Understand how tool ‘interacts with’ the software under test; what functions/methods are to use. • GUI Object ‘methods’ (functions) • Ex: SwfWindow(“LoginForm").SwfEdit(“ID”).SET “Jim” • Need to understand how to “pick apart” the software and “drive it” programmatically.

  29. Axioms to remember • SMART • Standalone • Test/Script able to run by itself, minimize dependencies • Modular/Maintainable • Scripts and code need to be modular in design and easy to maintain • Appropriate • Build according to need for project, don’t try to automate 100% • Reusable • Build for reuse; keep script code compact and focused. • Testable & Traceable • Make it easy to debug your code, and be able to link together.

  30. Axioms to remember cont. • SEARCH4,5 • Setup • Test/Script needs to check for pre-test conditions and flag if needed. • Execute • The logic and data needed to run the test itself • Analyze • Validations, Error Handling, etc. • Report • Progress, Error, Summation of run information (Logfile) • Cleanup • Test/Script needs to ‘cleanup’ (temp files, reset data, etc.) • Help (or Home)

  31. Setting Realistic Expectations • Management & Other Groups • 100% automation is an unreachable goal.6,7 • Automation is its own form of software development. • Ramp-up time will be required, longer than you think. • Benefits realized over time, no immediate payback for investment.

  32. Expectations cont. • Management & Other Groups • No tool supports all systems & technologies out of the box. • Not all testers can write scripts. Specialized resources are a must and need to be dedicated to the work. • Nothing is “Free”, there are always costs involved. • Don’t believe the marketing hype! Don’t buy the snake oil.8 There is no Magic!

  33. Otherwise, the result

  34. Summary • What is ‘Test Automation’ – A lot more than you think! • History – Lots of tools over the last 30 plus years. • Different types of test automation tools – Not just for execution of a test only. • Automation Myths – Old and New; lots of failures due to them. • How to fix the problems – Be realistic, dispel the myths. • Resources and Skills needed for test automation – Again, a lot more than you think. It is Programming! • Setting expectations for Management & Others – Without proper expectations you are painted into a corner.

  35. Remember… It’s automation, Not Automagic!

  36. Q&A

  37. Contact Info • Jim Hazen • Company: Freelance Contractor • Home email: calkelpdiver@gmail.com • LinkedIn profile: http://www.linkedin.com/profile/view?id=2835000&trk=tab_pro

  38. References • “Experiences of Test Automation: Case Studies of Software Test Automation”, Chapter 5, Graham & Fewster, Addison-Wesley, 2012 • “Test Automation - Let's Talk Business”, Igor Gershovich, Connected Testing, 2006. • “Implementing Automated Software Testing”, IDT Survey, ElfriedeDustin et al., Addison-Wesley, 2009. • “How We Test Software At Microsoft”, page 187, Alan Page, Microsoft Press, 2009. • “How to Automate Testing: The Big Picture“, Keith Stobie & Mark Bergman, 1992. • “Seven Steps to Test Automation Success”, Bret Pettichord, http://www.io.com/~wazmo/papers/seven_steps.html, 2001. • “Success with Test Automation”, Bret Pettichord, http://www.io.com/~wazmo/succpap.htm, 2001. • “Test Automation Snake Oil”, James Bach, 1996

  39. Thank you for attending this session. Please fill out an evaluation form.

More Related