Chicago Quality Assurance Assocation

1. Chicago Quality Assurance Assocation Secrets of Successful Automation ProjectsPresented byPaul M. GrossmanSoftware Quality Assurance AnalystQTP Automation Framework Designer

2. Portfolio of Custom Software • Software Development: .Net, Java, C# • eCommerce (Oracle ATG) • Mobile Applications: iOS, Android, Windows Mobile/WP7, Blackberry • Quality Assurance Testing • LoadRunner, QTP, HP ALMS • Custom Tool Development and Frameworks • Performance Testing • Mobile Testing: iOS, Android, Windows Mobile, Tablets • Developed Applications in the following sectors • Games & Entertainment • Healthcare • Social Networking • 4tegroup.com

3. PaulM. Grossman • 11 years automation experience with HP Tools • HP Quality Center 9.0 Certified Product Consultant • HP Quick Test Professional 9.0 Certified Product Specialist • Beta Tester of QTP 9.5 – 11.5 • Speaker at Mercury/HP Events • Became Automation Engineer due to Perfect Timing! Manager caught me practicing magic tricks while waiting for system to reboot.

4. Session Goals • Key success factors in test automation today • Demonstrating test automation ROI • Effective framework design techniques • Successful script design for Agile environments

5. Automation Challenges • Lack of communication of changes/details to testing group • Lack of understanding of should be automated • Lack of variation from plan in test scripts • Insufficient standards • Insufficient Analysis & Design • “Silver bullet syndrome” (automated testing is the “silver bullet” that will end all testing problems on a project) • Lack of change control

6. Case Study Automation Project Failure at a Medical Device Manufacturer Year 1: QA Manager wants automated testing. Buys WinRunner, accepts offer at new company. Year 2: New QA Manager wants automated testing. Hired Lead Developer (C++), given an Assistant (VBscript). Goal for automation: 100% coverage. Year 3: Lead Developer moves to new department. Assistant told automation is high-priority, spare-time project. Year 4: QA Manager moves to new division Assistant retires. Results: 0 Defects found. Negative Return On Investment. Unsuspecting manual tester assigned to automation team. New Division Manager expresses interest in automation. Manager promotes me to Team Lead… of defunct Automation Project.

7. Lack of research & planning Lack of apparent progress Lack of experienced resources Not a full time project No buy-in 100% automation goal resulted in development in all product areas 20% development in all areas with nothing to show Project handed over to assistant with no product training Misperception: FT not needed because automation by it’s definition is automatic, right? Testers & developers fueled rumors of tool’s stability Top 5 Project Pitfalls

8. Lack of research & planning Lack of apparent progress Lack of experienced resources Not a full time project No buy-in 10% Automation goal POC Pilot Project Training Full-time commitment Repeat defects manually on a non-automated system Mitigation Strategies

9. Tips And TricksAvoiding "100% Automation" Trap • 90% Automation can be approached but it is generally not cost effective • New releases will have more functionality with constantly decreasing code coverage • A percentage of the code could be unreachable code, such as uncalled functions • Pick one: Functionality or Browsers Do the math: You won't have time for everything! 20 - 40% Automation makes a significant impact on the testing process. Math: App Developers outnumber your team? Do they have 6-18 Months head start? Do they have more than one tool?

10. Case Study A - HA ! The Pilot Project • Automate the Acceptance Test (“Smoke Tests”) • Short tests, but too wide of coverage • Commitment to six weeks of full-time development • Became four weeks, 10 days of manual testing • Used “Record & Playback” to create the entire test script • Overwhelming maintenance • Demonstration to follow at the end of the project • Management convinced they purchased the right tool • Asked for training on Winrunner – Lessons Learned Request to get training approved!

11. Case Study Functional Decomposition • Crafted functions from Demo Script to create “TimeSaver” utilities • Wrote discrete state change functions: TRUE, Worked! FALSE, Uh-Oh! • Modular functions made maintenance manageable • Automated the common interfaces of two additional products • Proposed automating easy features first –ShowStopper #1:Memory Leak. First 13-hours of endurance test. –Developers stop blaming tool, now ask for ad-hoc scripts Request to attend Mercury World Conference Approved!

12. Estimated Return On Investment “Best Bang For The Buck”: Project TimeSavers • Automate tedious manual tasks • Create 64 Annotations - 30 Seconds • Perform 128 End to End Procedures - 30 Minute Drive Space Test • Register 128 New Patients - 30 Minute Boundary Test • These tests were small, little maintenance • Gets testers on your side, reduces their “pain” • Scripts were easy to change into reusable functions First Quantifiable Return On Investment: $40 (1 Hour) Final estimated ROI for the company:$2 Million

13. Estimated Return On Investment Manual Tester: $40 Avg. hourly wage, including overtime and benefits.

14. Estimated Return On Investment ROI Formula: Estimate Equivalent Cost of Automation How much would it cost to hire someone to do this work manually? Estimated hourly wage (Plus OT & Benefits): $40Per Hour Average Determine hours of runtime - 10 hour runs, 5nights a week 10 x 5= 50 Hours per system - on 3systems 50 x 3 = 150Hours per week ----------------------- 150x $40 = $6,000 Extrapolated to the end of the yearX 52 Weeks ----------------------- $312,000 Four years of run time: $1,248,000

15. Estimated Return On Investment What would some of the worst defects found by automation have cost the company had they gotten to the field? • Case Study Examples • ShowStopper #2: Transient Boot Up Failure • Replicated with automation and high end UPS • ShowStopper #3: Data Transfer Problem • Duplicated issue within five days. • Reason: Related to Network Load limitation. • We pegged the ROI of the worst detected defects by estimating the cost of a single lost customer sale: $100,000 • Potential revenue loss from 7 ShowStoppers: $700,000 • Plus 1.2 Million not paid to a manual tester: 1.9 Million Est. ROI Used Functional Test Tool to uncover Network Load issue?

16. Estimated Return On InvestmentThat was then…this is now What would some of the worst defects found by automation have cost your companyhad they gotten to the field? • ShowStopper #1: Credit cards fail to authenticate? • Store is closed world wide. • ShowStopper #2: Search path leads to infinite loop? • Customers frustrated with site jump to competitor. • ShowStopper #3: 12-second+ response time? • Customers abandon order • Is customer support being effected? • Slows down automation code development and debugging • Slows down development programmers • Slows down manual testers What if it's Dev, QA or Staging Environment?

17. Let's take a break! When we come back… • Tips And Tricks • Dealing with Failure • What is a Framework anyway?

18. Capture/replay Function-based Data-driven Keyword-driven Exhaustive Random Lexical Analyzer Hybrid What is a Framework?Framework Options

19. What is a Framework? • Driver • Loads & Executes Multiple Tests • Facilitates Result Reporting • Object Repository or Identification • Identifies objects by Class and Property Values • Uses Regular Expressions for dynamic values • Data • Pre-Defined • Generated • Recovery Mechanism • Browser closes • Object invalid • Error generated A framework is a set of techniques: • Reduce Maintenance • Abstract Complexity • Extend Coverage

20. Tips And TricksInitial Project Design • Return To Home State Functionality • Get back to the starting point from anywhere in the application, even if it is not running. • Navigationand Population • Low level custom actions to Click buttons, Open menus, Selectitem fromlists. • getValidObject (sClass, sName) • A single function that returns a valid object reference from many close matches. • Object exists (height, width & Abs_x > 0), is enabled, above underlay, closest to text. • Uses Descriptive Programming or xPath to identify object • Tokens • Stores reusable values, such as an Order Number • Can return intrinsic values like Today's Date. • Assume Failure, Prove Success • Use a global variable & Return Codes for overall success. This creates a single exit point for script execution. • Modify, Recover, Repeat HomeState "^QA_URL", "Admin" 'URLToken, UserRole EnterInto "Order No.", "^dt_OrderNumber" 'Token Lookup SelectFrom "Month", "^ThisMonth" 'Intrinsic Token Replacement Click "Generate New Order" NegativeTest VerifyText "Failed to submit order" CaptureAfter "Order Number:", "^OrderNo" 'Token capture for reuse Recovery Cancel_New_Order "^OrderNo" RecoveryEnd RepeatOnNextRow

21. Tips And TricksModular Design • Functions… but No Subs • Don’t waste time deciding which to use • Subroutines have no built in functionality for return codes • Sub FileExists(strFile, rc) ‘ Return Code Workaround • All Functions need Return Codes • FileExists = TRUE • IF FileExists(strFile) THEN ProcessFile (strFile) • Use Option Explicit in VBscript • Reduces errors due to variable misspellings • Three Level Keyword Framework • Components (Param1, Param2) • Keywords (Param1, Param2) • All other supporting functions Test Scenario# 3 Purchase Order With Discount Code and Pay with Declined Credit Card Enter_CC"Declined" getValidObject CLICK "Ok"

22. Tips And TricksGlobal Variables and Dictionaries • Minimize global variables: • gloObject: Reference set by GetValidObject • Use a Dictionary Object • Global • No Naming Convention • Stores Object References • Can be Persistent! • Case Sensitive Key Names

23. Tips And TricksDealing with Failure dicFramework("AlreadyFailed") = FALSE Function CLICK (sButtonName) If dicFramework("AlreadyFailed") = TRUE Then Exit Function End If blnFound = getValidObject "WebButton", sButtonName'Sets gloObject If blnFound = TRUE then gloObject.Click Else dicFramework("AlreadyFailed") = TRUE End If End Function That's an odd naming convention? "glo" = "Global Object

24. Tips And TricksDealing with Failure dicFramework("AlreadyFailed") = FALSE Function CLICK (sButtonName) … If blnFound = TRUE then On Error Resume Next gloObject.Click reportErr ("Click") On Error Goto 0 Else dicFramework("AlreadyFailed") = TRUE End If … End Function Function reportErr (sFunc) If err.Number <> 0 Then Report err.Description & " in " & sFunc End If End Function

25. Thinking Outside The Box getValidObject (sClass, sText) • getValidObject ("Link", "Login") 'Sets gloObject • Three Attempts • Descriptive Programming (Exact Match) • oBrowser.oPage.Link("innerhtml:=(.*)?Login(.*)?") • False? Add "Index:=0" - True if there multiple objects • Search Object Collection (First Valid Match) • oDesc("class") = "Link" : oDesc("Name") = ".*Login.*" • Set oLinksCollecton = oBrowser.oPage.ChildObjects(oDesc) • Switch Class • oBrowser.oPage.WebButton("innertext:=Login") • oBrowser.oPage.WebElement("innertext:=Login")

26. Thinking Outside The Box getValidObject (sClass, sText) • getValidObject ("Link", "Login") 'Sets gloObject • Validation – Is this object Visible? • Width > 0 • Height > 0 • Abs_x > 0 • Source_index: Appears over an underling object • This is not the List object you are looking for • List contains the string we want to select

27. Agile environments • Version Control • Not vulnerable to local system crash • Code Change Comparison • Merge Changes • Revert Changes

28. Agile environments • Continuous Integration with Jenkins • Run scripts when changes are made or at Night • Report overall Pass / Fail • Replaces Driver • Customizable We are real programmers: We should use similar tools and processes as the App Developers.

29. Secrets of Successful Automation Projects • Successful Projects Require • Modular Design, Demonstration, Low Maintenance • Plan for Common Pitfalls • Leverages Agile Processes • Automated tests • Keywords & KW-Based Components • Supporting functions • Dynamic Reusable data • Object identification • Recovery & Cleanup • Potential loss can be quantified as ROI • Training Increases ROI • Always think outside the box!

30. Resources • Training Classes • HP Certification classes • HP Discover • LinkedIn Users Group Meetings On the Web • cqaa.org • stickyminds.com • sqaforums.com • advancedqtp.com • wilsonmar.com

31. Thank you! Learn more at CQAA.org Questions? QTPmgrossman@gmail.com Additional Thanks to Lee Barnes Utopia Solutions www.UtopiaSolutions.com