Building Stable Selenium Tests on a Foundation of Jello

1. Building Stable Selenium Tests on a Foundation of Jello Dan cuellarLead software engineerZoosk – Test TEAM

2. A Bit about • Zoosk is a romantic social network with tens of millions of users that operates in 25 languages and in over 70 countries. • Products include: • web site • mobile web site • Facebook app • iPhone app • iPad app • Android app • Adobe Air app • The development pace is rapid, features are churned out in around 2 weeks time (including testing) • Product management is always running experiments and adjusting plans based on data from instrumentation • Developer to quality assurance ratio hovers somewhere around 10:1

3. Agenda • So, What Do You Mean by Jello? • How Good Tests Go Bad • Automation Philosophy • Self Testing Automation

4. SO What Do YOU Mean By Jell-o® • Jell-O [jel-oh] • Site wide A/B tests where ½ of users will see a complete site redesign and the other ½ will use the site as it was before. • Facebook app users can also split between different A and B designs at times. • Each A & B may have its own JS, CSS, and business logic to go along with it. • Not to mention that different versions of the API were being used by the web, Facebook app, and mobile apps. • All the while, different features were being enabled and disabled for different subsets of the A/B test groups

5. Things Can Get A Bit Crazy

6. A B

7. How Good Tests Go Bad

8. What Can Go Wrong? • General • Not enough information is retrieved to reproduce the failure • Elements • The type can change (e.g. dropdown becomes a checkbox) • An element can change location • An element can have multiple ways of identifying itself depending on an A/B test • Business Logic • Logic can change for one platform • Logic can change for some users and not others (A/B test) • Logic can change for all platforms

9. How do we combat these problems? • General • Not enough information is retrieved to reproduce the failure • Separate test actions from controller code, add abundant logging, and unify the patterns by which common operations are accomplished (e.g. selecting from dropdown) Don’t access selenium directly in any scripts. Create a wrapper that logs everything and performs basic checks. (e.g. is the choice available on the dropdown) • If libraries are used then you can log just about everything possible every time without having to re-paste all the logging code

10. How do we combat these problems? (cont.) • Elements • The type can change • Abstracting away all uses of the item into a library • An element can change location • Centralize all hooks into the UI, you can change it once and it will be fixed for all • Using reflection, you can verify that all of your hooks are functional • An element can have multiple ways of identifying itself depending on an A/B test • ID’s are great, but you can also use an XPath Or expression in your centralized hook

11. How do we combat these problems? (cont.) • Business Logic • Logic can change for one platform • Update the library for that platform • Logic can change for some users and not others (A/B test) • Detect the A/B test in the library • Logic can change for all platforms • Generic Libraries or test scripts which coordinate platform specific libraries are a great place to address this

12. AutomationPhilosophy

13. In One Slide • Separate logical actions from calls to controller (e.g. Selenium) code into libraries and interfaces • Centralize your site model and logic, so that all automation runs through one code path • Write code that can test the test automation. Investigating test script failures can be costly, but if you test code can test itself you will be able to pick out certain kinds of problems easily. • Write platform-agnostic test scripts when possible

14. Hooks • means by which test automation can identify pieces of the user interface • to reduce redundancy and lower the cost of test repair, it’s important to centralize your hooks into a single location • If a hook changes, it only needs to be updated in a single location

15. Controllers • pieces of code that control a component or platform • perform low-level actions to manipulate a component • Web Controller Action Examples • click • go to URL • gathering cookies • iPhone Controller Action Examples • tap • pinch • rotate

16. Libraries • methods (groups of calls to controllers) that perform product-level actions using a controller on a particular platform • contains methods that orchestrate controllers into higher level actions • Web Example • A login method in a library would check if the user is signed in by looking at a cookie first. • If it then determined that the user was not logged in, it would then fill out the login form and submit it and validate the proper cookie is set.

17. Interfaces • used to remove the platform context from libraries • each platform supports common tasks like authentication, messaging, searching, and more • implementing an interface allows for tests to be written in a platform-agnostic fashion. • Example • A user deletes their Zoosk account • Write a test that deletes an account, then verifies on each platform using a sign in method in the interface to valid authentication fails on each platform.

18. Test Scripts • coordinate interface calls (or library calls if the test is platform specific.) • test a particular product scenario and report back if the behavior is as expected. • when tests call the interface, the business logic of the test can be maintained in a single place, minimizing the amount of code changes that need to be made in test automation when business logic changes.

19. CONSOLIDATING AND Abstracting Product level actions • Libraries handle almost all calls to the controller • Product Level Actions (e.g. Sign In To The Site) are stored in libraries, so that there exists one good copy that everyone can use • The libraries implement generic interfaces so the platform can be abstracted away • Test scripts orchestrate interface, library, and controller calls into scenario test cases • Test script calls to interface > calls to library > calls to controller

20. Separation of control from scripting • An extra layer is great to log information and perform additional actions that Selenium can’t assume it needs to do • We use a layer between that wraps logging, error handling, and some other things on top of most WebDriver commands

21. Library Example • [C#] • namespaceAutomation.Libraries{public SeleniumActionLibraryWebSignUpLib: ISignUpLib {public voidSignIn(string email, string password){if (IsSignedIn()) return; • if (!Driver.Url.contains(“login.php”);Driver.Navigate().GoToUrl(“https://www.zoosk.com/login.php”); • Hooks.Web.LoginPage.EmailField.type(email); • Hooks.Web.LoginPage.PasswordField.type(password); • Hooks.Web.LoginPage.SubmitButton.click(); • WaitForElementToNotExist(Hooks.Web.LoginPage.Spinner); } }}

22. LIBRARY INTERFACE • [C#] • namespaceAutomation.Libraries{interface ISignUpLib{void SignIn(string email, string password);boolIsSignedIn(string email, string password); void SignOut(); }}

23. Platform Agnostic Automation Scripts • Many tests that need to be run are logically identical across all platforms. • Deactivate an account, attempt to sign in, expect failure • Send a message, verify it shows up in sent items • Take down some web service, verify an error is produced • Implemented these tests at an abstract level allows you to have one test for all platforms • When the logic changes, the logic only exists one place in your automation code

24. An Abstracted Test • [C#] • namespaceAutomation.Tests.PlatformAgnostic • {public static classSignInWithDeactivatedAccountTest: Test • {public staticvoid Run() • { • varaccount = APIUtility.Accounts.CreateTestAccount(); • APIUtility.Accounts.Deactivate(account); • client.SignUpInterface.SignIn(account.email, acount.password); • Log.VerifyFalse(!client.IsSignedIn()); • } • } • }

25. Self-Testing Automation

26. Centralizing Your Site Model FOR SELF -Testing • All hooks (means by which we retrieve WebElements) are stored in a single class, namespace, or something similar • Elements are stored lazily; the means to retrieve them are stored, but the actual WebElement is not retrieved until run-time. We use classes that wrap calls to By or methods which will return a WebElements • Hooks are grouped relevantly and each group has private self-test setup methods which will navigate the client (e.g. selenium) to where the hooks exist • Reflection is used to iterate through all of these groups and run the private self-test setup methods and then validate the functionality of the elements • Annotations (Java) or Attributes (C#) are used to exclude elements from the tests

27. About INTROSPECTION& REFLECTION • Introspection (Java) and Reflection (C#) is the process by which a computer program can observe, do type introspection, and modify its own structure and behavior at runtime. • In the example to follow we store information about how to use various hooks (references to pieces of user interface) in the code so that later at runtime we can use the information to determine how to test our library of hooks.

28. Centralized Hooks • [C#] • namespaceAutomation.Hooks.Web{public staticclassLoginPage {publicstaticWebHookEmailField = By.Id(“email”);publicstaticWebHookPasswordField= By.Id(“password”);publicstaticWebHookSubmitButton = By.Id(“submit”); • [OmitFromSelfTest]publicstaticWebHookSpinner = By.Xpath(“//div[@class=’wait’]”); • privatestaticvoid _StartSelfTest() { _SelfTestSeleniumInterface.Open(“http://www.zoosk.com”) } }}

29. Simple Self-Test • [PseudoCode] • // iterate in depth-first-search orderforeach(class c in Automation.Hooks.Web){ if c.hasMethod(“StartSelfTest”)c.executeMethod(“StartSelfTest”); • foreach(WebHookh in c)Log.Verify(IsElementPresent(h)); • ifc.hasMethod(“StopSelfTest”)c.executeMethod(“StopSelfTest”); • }

30. More advanced Techniques • Subclass WebHook to things like TextBoxHook and have the test check for reading and writing from the text field using Selenium • Do the same for Links, Images, etc. and write custom test methods • Randomize the order of which elements are tested to check for robustness • Add random data to the tests, e.g. type a random string into the text box. • Write custom test methods for structures that require it

31. The End