Steven Borg | Co-founder & Strategist, Northwest Cadence

1. 04b | Manage Test Execution (2 of 2) Steven Borg | Co-founder & Strategist, Northwest Cadence Anthony Borton | ALM Consultant, Enhance ALM

2. Module Overview • Run Tests • Perform Exploratory Testing • Manage Bugs • Use Lab Center • Analyze Recommended Tests • Perform Analysis • Manage Work Items

3. Manage Bugs

4. What the Study Guide says… • Manage bugs • tracking bug metrics (bug trends, status) • verifying bugs (create Test from bugs) • analyzing bug reports • managing bug workflow

5. Manage Bugs

6. Use Lab Center

7. What the Study Guide says… • Use Lab Center • creating new environments • creating copies of environments • running Tests on remote environments • executing Test Case in a Lab Center Environment

8. Lab management automation Development and Testing Environments Repro environments Development environments Test environments Lab management services Environment provisioning Environment templates Environment snapshots

9. Create a new environment using virtual machine templates in SCVMM • Create new environments on existing machines Continuous Build-Deploy-Test Modern applications need to be deployed and tested in multiple environments. Whether you store your test environments on bare metal or in the cloud, Lab Manager enables automation and increases collaboration, reducing the build-deploy-test cycle time. • Create an environment using machines running on SCVMM

10. Quickly choose multiple test suites to run after deployment Automation: engage When developers and testers collaborate, quality wins. Developers build the application, deploy it to an environment and run automated test cases in a single step, preparing the way for testers. • Choose your environment, and build, deploy and test in that environment from a single dialog

11. Spin up virtual test environments from within Test Manager, or attach a virtual environment to a manual testing session Manage Environments Testers spin up environments to run manual tests, and test agents record all the data required to recreate any bugs. Snapshot environments and attach to bug reports so developers can easily reproduce in the exact environment where the defect was found. • Snapshot environments during manual testing to increase reproducibility

12. Test run results are attached to specific builds and environments Intelligent analysis Enable data collectors and generate detailed test results to ensure any bugs that are found are actionable. Team Foundation Server keeps track of all the pieces, keeping your entire team on the same page and reducing time to repair. • Collect all the data required to recreate bugs from virtual environments

13. Use Lab Center

14. Analyze Recommended Tests

15. What the Study Guide says… • Analyze Recommended Tests • selecting the build in use • comparing the current build to a previous build • viewing Recommended Tests • analyzing related Work Items

16. Analyze Recommended Tests

17. Perform Analysis

18. What the Study Guide says… • Perform analysis • analyzing reports (Requirements-User Stories) • analyzing by Test Suite • analyzing by configuration • identifying areas where quality is low • identifying Test Plan status

19. Software testing with Visual Studio Microsoft Test Manager Visual Studio IDE Test planning and management Test case management Lab manager Manual testing Exploratory testing Code quality analysis Unit testing UI automation testing Performance testing Load testing Microsoft Feedback Client User testing Team Foundation Server Work item tracking Source control Lab management services Test execution services Build services Reporting and BI Pluggable components Lab environments Test runners Diagnostic data collectors Build engines

20. Monitoring Quality Metrics Testers Stakeholders Solution managers Users Operations Developers • Test planning and management • Test execution and defect reporting • Lab management • Requirements management • Acceptance criteria definition • Production monitoring • Integrated incident management • Actionable feedback • Exploratory testing • Developer testing • Build automation • Actionable diagnostics Quality metrics and reporting Role tailored tools feed data to transparent quality reports Data Warehouse Custom SSRS/SSIS Test Failure Analysis Stories Overview Bug Reactivation Bug Trends Pre-built and fully customizable cross project reporting

21. Filters allow you to control which builds you see Visualize quality The build report is a barometer of quality and a harbinger of success or failure. As the development cycle nears the end, defects and failed builds should trend towards zero. Build quality reports provide teams clear insight into the development process and can indicate where teams need to focus their efforts. • View the test and code coverage results for each build to determine quality

22. Right out of the gate Waiting until the end of a cycle to address defects can adversely impact schedule and resources. Desired ATDD behavior should see a high spike in failing acceptance tests early in the cycle with the spike tapering down as the cycle progresses indicating that tests are being run early and uncovering defects early on.

23. Bug bashing Development teams can use defect tracking reports to estimate levels of deliverable code quality and to provide insight into a team’s ability to address defects. A defect should ideally be detected and solved once, with a corresponding test implemented to ensure the fix addresses the desired behavior. Bug reactivations should remain close to zero.

24. Line-by-line highlighting illustrates which code is covered, uncovered, or partially covered by test Filling in the gaps The only thing constant is change and code that changes must be tested. To ensure your code base is fully covered by test leverage the code coverage report and identify your testing gaps. Establishing a code base fully covered by tests ensures new changes will not break your code! • Results show which methods are covered by test and total coverage is indicated by a percentage

25. Track active and resolved bugs by user story • View real-time status of each user story I can see clearly now A picture is worth a thousand words and a Stories Overview report gives that big picture! The Stories Overview gives a comprehensive view of each implemented user story including completion status, test results and bug status. A thousand words, indeed! • Track total percentage hours completed by user story • Test results quickly display quality status of each user story

27. Perform Analysis

28. Manage Work Items

29. What the Study Guide says… • Manage Work Items • validating requirements • Work Item relationships (eg, what it means when a test case is associated with a requirement) • creating Work Item queries • performing bulk updates in Microsoft Excel

30. Manage Work Items

31.  EXAM BEST BETS • Become very familiar with work items, work item queries, and how work items are related • Understand the lifecycle of a bug and its associated test case • Know what Excel reports are available for each process template • Know what SSRS reports are available for each process template • Understand the basics of the Lab Center tab, including environments and an overview of Lab Management