Driving Fit & Finish in UX Quality: Desirability is in the Details

1. Driving Fit & Finish in UX Quality: Desirability is in the Details Yingzhao Liu, Program Manager, WEX User Interface PlatformDonna Wallace, Senior UX Manager, DDE User ExperienceKynan Antos, UX Designer, Windows Home Server Dan Harman, Program Manager, Dynamic Systems FoundationAugust 2007

2. About this presentation • Introduction • What is Fit & Finish • Driving Fit & Finish for Windows Vista • Fit & Finish everywhere • Visual Studio • Windows Home Server • Server Manager • Best practices • Timeline • Culture Change • Ingredients for a successful process • Warning signs • Take-aways

3. What is Fit & Finish • A term in the manufacturing industry in general and typically refers to the process of doing the final assembly:  putting the pieces together perfectly (Fit) and adding the final layer of paint, polish, removing fingerprints, etc. (Finish). • In software as in manufacturing, it can not wait to the end.

4. Vista: requires high Fit & Finish

5. Some examples of Fit & Finish bugs

6. Some feedback on Release Candidate • “There are still too many pre-Vista icons. Take a look inside the Windows folder, for example.  The “Offline web pages” icon is 95-style, and the “Downloaded program files” icon is 2000-style.  These have got to be updated!  There is ZERO excuse for not doing it… • “There are too many slightly-different shades of blue/green throughout the product. • “Each of the following apps have different back/forward buttons: Windows Explorer, IE, Help & Support (*quite* different), Windows Media Player/Photo Gallery (these two are the same—yay!), Windows Defender…”

7. Fit & Finish bugs • 10,000 FnF bugs were entered; 8,000 unique; 4400 fixed. • Bug Categories: • Visual glitches • Incorrect text or graphics • Layout errors • Animation errors • Visual clarity problems • Windows Vista UX Guidelines violations (http://uxguide) • UI implementation errors

8. Driving Fit & Finish for Vista • Got VP approval • Focused on areas of the UI with the highest impact • Conducted review meetings, enforced UX Guide and exit criteria • Made it easy for anyone to enter visual bugs • Staffed with our own testers and developers • Partnered with Basics: usability and accessibility • Attended central shiproom

9. About this presentation • Introduction • What is Fit & Finish • Driving Fit & Finish for Windows Vista • Fit & Finish everywhere • Visual Studio • Windows Home Server • Server Manager • Best practices • Timeline • Culture Change • Ingredients for a successful process • Warning signs • Take-aways

10. Visual Studio Whidbey Donna Wallace, Senior UX Manager, DDE User Experience

11. Visual Studio Whidbey (v7) “One big piece of feedback from [our recent VP reviews] that I agree with is that we need to focus more on end-to-end UI consistency across these products…. …When you look at a product that is completely consistent in its UI model versus one that isn’t, I personally jump to all sorts of conclusions about the consistent one being faster, more productive, more stable, more compelling, etc. all because of that one attribute… in the end, the consistency has a multiplier effect that helps sell the overall value.” — Craig Symonds, GM

12. Whidbey consistency initiatives • Promote designing for consistency • Via design guidelines • Target high profile inconsistencies • Via core scenario walkthroughs • Address “death by 1000 paper cuts” • Finding and fixing the right issues • Invest in engineering for consistency • Via greater implementation sharing

13. Whidbey consistency initiatives • Promote designing for consistency • Via design guidelines • Target high profile inconsistencies • Via core scenario walkthroughs • Address “death by 1000 paper cuts” • Finding and fixing the right issues • Invest in engineering for consistency • Via greater implementation sharing

14. Components of UX issue management • Identify and prioritize all UI elements • Determine prioritization scheme • Systematic method, communicated broadly • Find the right issues / bugs • Fix the right issues / bugs • Define success: set quality targets • Triage team keeps teams honest and ensures consistent severity ratings across the division • Track and report

15. Identify UI • UX : track UX reviews, Fit & Finish progress • UE, Loc: track UI work and changes • Today some of the list is generated, most is manual • Whidbey UIComponents.xls

16. UI Element Priority: EP rating • Upgrade priority based on • Part of main scenario: EP1 • Market focus or affecting user SAT EP1 EP2 EP2 EP3

17. Severity definitions

18. Example: Severity 1 UI Bugs Breadth: 90% of customers will regularly see issue Depth: Prevents usage Impact: Same impact as any Sev 1 technical bug

19. Example: Severity 2 UI Bugs Breadth: > 50% of customers will regularly see issue Depth: Significantly impacts usage Impact: Customer questions overall stability

20. Example: Severity 3 UI Bugs Breadth: ~ 25% of customers will see paper cut Depth: Annoying but doesn’t significantly impact usage Impact: Multiple bugs portray sloppiness & lack of professionalism

21. Example: Severity 4 UI Bugs Breadth: Only the trained eye will notice paper cut Depth: Doesn’t impact daily usage Impact: Customers won’t question specific bug, but multiple bugs make UI appear unfinished

22. Prioritization matrix defines goals

23. Bug finding: UI reviews • Conducted by UI experts and feature teams together at each milestone start or end • Evaluate both task flow and UI details • Avoid surprises: Disclose upcoming feature work • Get a “B” in coverage: Get first look at low priority areas • Gain consensus, set expectations accordingly • Provide the management team a milestone checkpoint on the “Overall Product UI Health” scorecard • Many issues can be fixed without logging bugs

24. High profile inconsistencies Smart Tags Early designs: similar needs, but multiple implementations, very different appearance and behavior

25. High profile inconsistencies Final designs: fewer implementations with a common look and feel.

26. Bug finding: UI testing • Ad hoc UI bugs (from UI Guidelines posted for teams) • Systematic approach: UI test cases • Derive objective test cases from guidelines • Format test cases to fit QA tools • Teach QA teams how to review UI • Categories of bugs: • Communication • Ease and efficiency • Error prevention • Visual design • Consistency • Engineering

27. Detail design Fit & Finish: Setup Before:Pseudo-web appearance and navigation Brand imagery not consistent Nonstandard themed controls Nonstandard font face & color nonstandard colors Use of links to perform actions nonstandard grid

28. Detail design Fit & Finish: Setup After: Updated use of brand imagery Standard themed controls Appropriate use of links Standard foreground color & fonts Standard grid Standard colors Use of buttons for action commands

29. Reporting • Due to the security push, seeing lower fix rates over those weeks • 102 UX bugs added since 11/12 • 73 UX bugs resolved as Fixed since 11/12 • 13 UX bugs closed as Won’t fix or Postponed since 11/12

30. In summary • Things that worked well • Key: had clear directive from executives • Built reviews into divisional schedule and set expectations • Systematic approach resonated well with engineering practices • Ensured flexible process to tailor fit individual teams • Had adequate reporting and communication both up and down • Things that we’re still working on • Understanding how to better integrate into Feature Crew model • Tuning quality gates and exit criteria • Measuring success beyond using only bug counts (find or fix rates) • Defining a method to better identify “dirtiness factor”

31. Windows Home Server Kynan Antos, UX Designer, Windows Home Server

32. Fit & Finish is about quality “From what we’ve seen so far, we’d rate Windows Home Server as one of Microsoft’s most polished and most impressive 1.0 releases to date.” — APC Magazine

33. Examples of FnF issues we encountered… Improperly rendered transparency…

34. colors didn’t match… Aliased edges: lack of transparency Background mismatch

35. placeholders were still placeholding… Placeholder icons…

36. deep investigation led to big issues… Collision…

37. poor implementation created issues… Artifacts from stretching…

38. creative solutions are often necessary btn_end_lft.png btn_end_rt.png btn_bg.png

39. sometimes things were all wrong… Coordinates and spacing…

40. all of these issues add up fast…

41. it isn’t risky (or difficult) to fix them…

42. ...fixing these issues has a big impact. • “We don’t want to award anyone before the final product is out, but we’ll say that we came out of the closed beta very impressed.” • — CNET • “I found Home Server to be intuitive to use... Its simplicity and automation will make it a winner with many home users.” • — Computer World Magazine

43. What worked • Adoption of the Vista Fit & Finish • Involve team in prioritizing feature areas • Use soft skills, build relationships • Get out of your officeWork directly with developers to fix issues • Write good bugs, always use screenshots • Focus on quality in the product, not on bug count

44. What didn’t work • Lack of early executive buy-in • Started too late in the schedule • Feature slip was prioritized over Fit & Finish schedule • No time to file, triage, and manage bug overhead • No ship criteria with teeth • No incentives for building quality

45. What we are doing next time • Getting early executive buy-in • Getting Fit & Finish on the schedule earlyScheduling Fit & Finish within each milestone after ZBB • Using Product Studio for tracking Fit & Finish issues • Developing and enforcing Fit & Finish ship criteria • Giving UX the authority to get results • Sharing the Fit & Finish process

46. Server Manager Dan Harman, Program Manager, Dynamic Systems Foundation Presented by Yingzhao Liu

47. Server Manager

48. Review Process for Server Manager “We were able to fix numerous issues that would not have been fixed otherwise, and collectively, the impact on the UX has been significant. Combined with the changes to our team culture after having witnessed the cost versus benefit, we’re impressed with the results.” —Dan Harman, PM

49. Review Process for Server Manager • Used Vista Fit & Finish review process as a baseline • Conducted two 2-hour review sessions in March 2007 • Focused on high priority areas: • Core console elements • Core wizard framework elements • Custom controls • Dynamics in the UI • Key experiences • Involved participants from 4 disciplines: PM, design, dev, and test • PM drove process, took notes, and filed bugs • Design contributed to reviews and assigned severity to issues • Test contributed to reviews and provided test environments • Dev triaged, costed, and fixed bugs

50. Filing Bugs • Used same bug categories as Vista Fit & Finish reviews • Visual glitches • Incorrect text or graphics • Layout errors • Animation errors • Visual clarity problems • UX Guide violations • UI implementation errors • Followed Vista Fit & Finish process for filing bugs • Filed bugs for all issues found • Assigned Priority 2, Severity 2 for most bugs • Added “RmtVisual” keyword for tracking purposes