How Agile Ended Our Defect Report-Fix-Check-Rework Cycle

1. How Agile Ended Our Defect Report-Fix-Check-Rework Cycle Richard Leavitt Rally Software Development rleavitt@rallydev.com

2. Objectives for Today Observation: • Traditional workflows and project metrics evolved to manage large defect inflows during long testing phases Question: • How relevant are these management processes in dramatically shorter development cycles? Agenda: • Defect Workflow & Metrics in Test-Last • Our Agile Transition • New Dev & Test Processes, Workflows and Reports • Benefits and Bruises

3. Agile Doesn’t Change Our Goals • Customers judge our software as high value and high quality • Our business judges our efforts as timely and providing high ROI • We like our jobs

4. Agile Doesn’t Change Our Questions We desire: • A small set of useful and valid measures, provided by • Standard reports that everyone anticipates and understands, with • The ability to drill down and roll up Status How’s feature X doing? Readiness When can we release? Progress What’s blocking?

5. Before: Phased Development Process • Advantages: • Requirements and design “up front” • Logical • Prescriptive • Plan based • Disadvantages: • Requirements “freeze” paradigm • Dreaded system integration phase • High defect counts and “surprises” • Hard to adapt to changing requirements • Plan based

6. One of My Test-Last Environments • Release size: 16 FTEs x 14 Months • Detailed work breakdown with task complete dates • Unit testing? What’s that? • QA drops with high bug counts (100’s) • Islands of information • Test effort & results • Traceability matrix • Defect/Issue/Request lists and metrics • Project plan • Requirement docs (MRD->PRD->SRS) • Test cases in Excel • Build results

7. High Defect Counts Add Weight • “Push” workflow systems with complex rule engines for routing & signoffs • Fine-grained permissions to control ownership and changes • Lengthy bug reviews & management burdens • DELAY!

8. But Defects are Easy to Count & Graph! Statistical gymnastics • Defect rates • Densities, cycle times • By phase • By owner, type, status, severity, module, how found… • In general, lots of totals, percentages, averages and ratios

9. And how else can we answer our questions? After all, our typical experience goes like this: “I’ve never heard of any direct measure of project status, program readiness for release, or progress toward meeting milestones, etc. Instead, we’ve used surrogate measures and metrics.” The Darker Side of Metrics Douglas Hoffman, Software Quality Methods, LLC

10. QA Drops Problems with Defect Metrics Readiness When can we release?

11. Problems with Defect Metrics Readiness When can we release? Bug Review Meetings • Defects “closed” as • Deferred • Duplicate • Low priority • Can’t Reproduce • Enhancement Request Close Rate2 Close Rate1

12. So, Phased Development Processes Encourage a Defect-Driven Approach And its accompanying problems: • Large defect counts with heavy process burdens, high management effort and delay • Defect-based metrics that don’t relate well to the real questions we’re trying to answer • All metrics programs cause unintended side effects on our behaviors, but can we do better?

13. Why Agile Development? • Agile promotes rapid delivery and feedbackover elaborate upfront planning Short iterations give all stakeholders timely and regular visibility into readiness, status and progress • Shared commitments simplify workflows… but everyone needs real-time status information! • Agile handles change better • Accepts late-binding requirements • Continuous integration and early testing means less “integration hell” and associated defects & surprises In ShortMinimum Process, Maximum Value

14. The Agile Paradigm Shift Waterfall Agile Cost Schedule Requirements Constraints Value /VisionDriven PlanDriven Estimates Cost Schedule Features The Plan drives cost & schedule The Vision drives features

15. Waterfall Iterative Iterative and Incremental Parallel AcceptanceTest Driven Moving to Iterative & Incremental Agile Development ProjectMgmt Critical Path through Phases Critical Drop Schedule Time Boxes Requirements & Design Detailed Design Inside Iteration Manage Scope Creep Freeze & Signoff Development Process Increment at a time Multiple Drops to QA All Features in Parallel Test & Fix Process Acceptance Test Inside the Iteration “Test What’s Working” Last Phase Only

16. Agile Team Practices • Time box everything • Two levels of planning: rough for releases, fine-grained for iterations • Just-in-time requirements elaboration • Focus on the highest priorities • Early and continuous testing • Try to remove roadblocks and defects immediately Slide 2

17. The New Planning Process • Release cycle: 8 or 10 weeks (4 or 5 iterations) • Prod Mgmt defines themes & features/stories • Dev provides rough “planning” estimates • Release Backlog is then negotiated & prioritized • “Hardening” iteration at the end tackles technical debt • Iteration cycle: 2 weeks with one ½ day planning meeting • Recap previous iteration • PM presents new and elaborated stories & use cases • Dev details tasks & estimates just-in-time • Priorities are set and the team commits

18. Story Acceptance is Common Goal • One schedule, one budget, one team, but independent dev & test people and tasks • Features are squeezed, NOT testing and schedule • Dev codes highest priority requirements all the way to acceptance • QA elaborates & automates acceptance tests in parallel with coding • Dev-QA collaboration versus separation improves code robustness and testability right away. • Quality no longer “tested in”

19. Early & Continuous Testing Drives Down Defect Counts • Unit tests run on code check-in • Nightly build runs automated acceptance tests • Team tries to resolve failures immediately • When combined with short iterations, we get • Tiny defect counts (dozens, max) with little management burden and team overhead In Short, We Stay Close To Shippable Code

20. Iteration Decisions & Metrics Meeting iteration intent and schedule means responding to changing realities • Should we cut or adjust any features? • What progress have we made on each story? • How much is there left todo? • What’s our remaining budget? • What’s standing in our way of acceptance? • Blocks? • Which story card tests are failing? • Are any defects open against our iteration stories?

21. New Iteration Status Views Readiness When can we release? 4 2 Status How’s feature X doing? Progress What’s blocking?

22. Tie Acceptance Test Results To Story Progress What’s blocking?

23. So, Failing Tests Replace Many Defects for Simple, Pull-based Workflow • Dev and Test now communicate via failing tests instead of routing defects Test-driven Advantages • Failures are easy to reproduce • Failures are transient, less management overhead • We’re done when the tests pass • Lighter process with less delay

24. Summary: Instead of Focus on Defects

25. We focus on Acceptance

26. Summary: Instead of Complex, Push-based Workflow

27. We use Simpler, Pull-based Workflows

28. Benefits and Bruises • Agile has dramatically changed our project planning, tracking and workflow. We are more responsive to new requests and late changes. >Everyone needed education on paradigm shift • Everyone focused on incremental, meaningful functionality, not defect find/assign/fix/regress >Demands high collaboration and real-time status. >Upstream analysts must provide steady flow of business needs and priorities

29. More Benefits and Bruises • Time boxes mean we always know when we’re going to release >Though the scope may be slightly different! • QA plays large role in requirement elaboration and defect prevention. >Early resistance to TDD – required a change agent (Mike Cohn) >Struggles to automate tests inside iteration >New skills required in test design and automation.

30. More Benefits and Bruises • Tiny bug lists (dozens) with little management overhead >Most “bugs” found during exploratory testing in hardening iterations. >Re-factoring tests is critical to keep up velocity. • Simple, pull-based workflow with team members actively grabbing the next most important item off the stack >Need culture of shared ownership that “takes responsibility”

31. Suggested Process Brown bag discussions Guest lectures & webinars Local Agile/XP user groups Agile Conferences Agile2005 – Denver, July 05 Where to start on the web Agile Alliance Web Site www.agilealliance.org Rally Ecosystem Pages www.rallydev.com/ecosystem Suggested Reading Agile Project Management w/ SCRUM www.controlchaos.com/ Agile & Iterative Development www.craiglarman.com Lean Software Development www.poppendieck.com/ Agile Project Management www.jimhighsmith.com/ Tactical Management of Iterative Development www.rallydev.com Suggestions for Getting Ready

32. Questions?

33. Acknowledgments • The Darker Side of Metrics, Douglas Hoffman, PNSQC 2000 • It’s Not About the Bugs, Harry Robinson, Stickyminds 2003 • Managing Software Requirements: A Use Case Approach, Dean Leffingwell, 2003 • The Test Manager at the Project Status Meeting, Brian Marick, Steve Stukenborg, 1997