Higher-Level Testing

1. Higher-Level Testing

2. Higher level testing Integration Testing Testing the interaction among a number of interactive components System Testing Testing the system as a whole, considering various environments, external exceptions Acceptance Testing Final user testing Usually on GUI 2

3. Integration Testing Strategies Big Bang Top down Bottom Up Sandwich Path-based strategy 3

4. Big Bang! Prepare all relevant components Data, Global variables… Put them together Pray! 4

5. Usage Scenario of Big Bang Quite common in small projects Requires no extra effort for integration May work well if all interfaces are well-defined … Not likely to happen… 5

6. Top down strategy Feature decomposition A hierarchical structure of all software features May not require extra efforts: You should use it in requirement and design 6

7. An example feature decomposition 7

8. Top down strategy Focus on the validation of the whole system first Test an empty system with no components at first All components are replaced with test doubles, e.g., stubs Gradually add components Until all components are added Advantages Easier to understand the process of the integration May have a working system earlier (or always have a working system): important for modern development, such as XP 8

9. Top down strategy Disadvantages Requires to write test doubles Since DOC matters in integration, so more complex test doubles is usually required Centralized So the integration cannot be done parallelly 9

10. Bottom-up strategy Focus on the integration of lowest units at first Start from an unit depending nothing else When all sub-unit of a component are integrated, integrate the component Until the whole system is built 10

11. Bottom-up strategy: Illustration Bottom Level Subtree Middle Level Subtree Top Level Subtree 11

12. Bottom-up strategy Advantages No test stub is required Requires test driver but may re-use unit test cases (view the whole component as a unit) Support parallel integration Issues No working system is available until the end Higher risk Require more interactions among sub-teams working on each component 12

13. Sandwich strategy A combination of top-down and bottom-up strategies Do top-down and bottom-up integration simultaneously Do bottom-up integration for common sub-systems, and top-down integration can be used for the integration of the sub-system 13

14. Sandwich strategy 14

15. Path-based strategy A special strategy for testing the integration related to some key features Scenario: A management software A report needs to be generated, then reviewed, approved by multiple people (e.g., student, admin, chair) It goes through the sub-systems for student, admin, chair, … An integration along the paths of the report 15

16. Application of strategies Big bang is widely used for small projects Pure top-down or bottom-up strategies are rarely used For large projects, usually sandwich strategy is used with the supplement of path-based strategy for key data or control flows 16

17. Higher level testing Integration Testing Testing the interaction among a number of interactive components System Testing Testing the system as a whole, considering various environments, external exceptions Acceptance Testing Final user testing Usually on GUI 17

18. System testing Test the system as a whole Usually test against specifications For each item in the specification Work out a test case and a test oracle Test boundary values Test with invalid inputs Test with environment issues 18

19. Environment issues Building Compile options and configurations Environment variables Third party dependencies Underlying platforms OS, database, application server, browser Compatibility is a huge problem Different platforms (if cross-platform) Different versions of dependencies and platforms Different configurations 19

20. Environment issues Third-party services Dropbox, google API, amazon, etc. Testing for both normal interaction and service failures Environment failures Power failure Network failure Hardware failure Distribution issues Runtime update Potential conflicting software 20

21. Higher level testing Integration Testing Testing the interaction among a number of interactive components System Testing Testing the system as a whole, considering various environments, external exceptions Acceptance Testing Final user testing Usually on GUI 21

22. Command line testing For software with command line user interface Writing test scripts With different options and inputs Try invalid cases : missing option parameter, invalid paths, etc. Be careful about tearing down between test cases Try to put comparison code inside the scripts Write well-structured scripts and comments Note: you always need to change test cases as the software evolves 22

23. GUI testing Testing Graphics User Interface is easier But harder to automate And harder to compare results with oracles Manual testing is still widely performed for GUI testing Manually explore the user interface Record the steps in the test for future testings Observe the GUI for errors 23

24. GUI testing: Tools Three category of GUI testing tools Event-based tools The tool feed in GUI events to the event handlers in the GUI code Screen-based tools The tool will feed in screen click / swipe / drag /… events to the operating system Computer-vision-based tools The tool identifies the object on the GUI with computer-vision techniques 24

25. Event-based tools Somewhat like unit-testing on GUI classes Generating Fake Event Objects Feed the events to the event handlers Need to design event feeding sequences carefully, otherwise may cause false positives When infeasible event sequences are fed Pros & Cons Robust for display change Easy to automate Feasible event sequences are hard to generate Sensitive to internal code changes Not for end-users 25

26. Appium: Sample Code @Test public void example() throws Exception { // find an element by tag name WebElement button = driver.findElement(By.tagName("button")); button.click(); // get the value of the element WebElement texts = driver.findElement(By.tagName("staticText")); assertEquals(texts.getText(), "some expected value"); } 26

27. Screen-based tools Imitates end-user interactions Generate OS-level events You can see the cursor move on the screen Provide co-ordinations of events Pros & Cons Great for end-users Robust for GUI-implementation changes No false-positives: all sequences are feasible Difficult to find co-ordinations Sensitive to display changes and screen size / resolution 27

28. MonkeyRunner: Sample Code # Runs The Componentdevice.startActivity(component=runComponent)# Presses The Menu Buttondevice.press('KEYCODE_MENU', MonkeyDevice.DOWN_AND_UP)# Dragging To Specific Coordinate (x,y)device.drag((15, 80),(15,400),5,10)snapshot_screen(Actual_device)device.drag((15, 400),(200,400),5,10)snapshot_screen(Actual_device)device.drag((200, 400),(200,80),5,10)snapshot_screen(Actual_device) 28

29. CV-based tools An emerging technique for GUI testing An improvement on screen-based tools Specify which button to click instead of give the co-ordinations CV-based tools will find the button on the screen and click it Pros & Cons Great for end users Solve the problem of display changes and different screen/resolution Bring in uncertainties 29

30. GUI testing: Script generating Record and Replay Screen record and replay Record click on the screen and keyboard inputs Replay all the clicks and inputs Not robust, affected by screen size, resolution, resize, … Event record and replay Record all the UI events in the UI framework (swing, MFC, android etc.), and outputs (texts) in the UI Re-trigger all the events and compare the UI texts More robust, but requires more preparation and overhead 30

31. Review: Higher level testing Integration Testing Strategies: big bang, top-down, bottom-up, sandwich, path-based System Testing Environment issues: building, platforms, web services, environment failures, distribution issues Acceptance Testing GUI testing Event-based, Screen-based, CV-based Record and replay 31