Performance Analysis using Windows Performance Toolkit

1. CL16 Performance Analysis using Windows Performance Toolkit Michael Milirud Program Manager Microsoft Corporation

2. Session Objectives And Takeaways • Performance is critical • Live production system analysis • WPT Kit, Performance Analyzer tool suite, and xperf tools • How to • Collect a performance trace • Do basic analysis of a performance trace

3. Agenda • Motivation • Overview of Windows Performance Toolkit • Key Concepts • Case Studies • Conclusion

4. Why care? • Motivation • Overview of Windows Performance Toolkit • Key Concepts • Case Studies • Conclusion

5. Ensuring good performance • Responsiveness • Unresponsive system  Customer dissatisfaction • Resource utilization • CPU, disk, memory, network, battery • Resources are limited and shared, use them effectively • Good citizenship • It takes one component to impact system performance

6. Resource Usage Concepts • Resource usage point of view • Machine constraints = reality • Applications/services/drivers/OS compete for: • Processor time • Physical memory space • Virtual address space • Disk service time • Disk space • Network bandwidth • Battery power • … • Machine constraints = resource usage limits

7. Performance Analyzer Preview demo High level overview What is going on behind the scenes?

8. Performance Analyzer tool suite • xperf: • Trace capture, processing, and command-line analysis • xperfview: • Visual trace analysis • Xbootmgr (and xbootmgrSleep): • On/Off transition trace capture • hwpower2etw: • TDMS-to-ETL converter • symcachegen: • PDB-to-SymCache generator

9. Basic operations • Trace capture • Graphs • Frame List • Legend • Selection • Overlaying • Summary tables • Gold bar • Column Chooser • Sorting

10. I Get It, But Where Do I Get It? • Motivation • Overview of Windows Performance Toolkit • Key Concepts • Case Studies • Conclusion

11. Windows Performance Toolkit • Official kit • Built and used by the Windows organization • Wide support range • Cross platform • {Vista+} x {client, server} • Cross architecture • Publicly available • Contains Performance Analyzer (WPA) tool suite • WPA includes XPerf • Also includes GPUView and WpfPerf tools • Strategic investment

12. Where is WPT Kit applied today? • Bread and butter of Windows Engineering group • Windows 7 performance work • Industry-wide OEM performance effort

13. What is Performance Analyzer? • Tools for • Capture • Post-processing • Analysis • Very mature • 10+ years in development • 4th generation • Product quality code • Enterprise grade level • Core code reused inbox • Dev/Test/PM/UA – the whole deal  • Documentation on MSDN

14. Why Use Performance Analyzer? • Holistic performance analysis • All processes/threads • User + kernel mode • DPCs and ISRs • Scheduling • Disk and file I/O • Memory • Network • Analyze what customer experiences • Catch the problem as it happens • Capture-Anywhere-Decode-Anywhere • Very low overhead vs. alternatives • System-wide temporal “debugger”

15. Issues Performance Analyzer Can Help Root-cause • Responsiveness issues • Long delays in applications • High CPU usage • High Disk usage • Slow On/Off transitions • Poor battery life • And much more!

16. The Big Picture XML file XPerfView 6. CLI trace analysis via actions 5. GUI trace analysis via graphs and summary tables MergedETL file 3. Controls logging sessions and enables/disables providers Action System and Symbol Information XPerf Control/Status Control/Status Post Processing 4. Metadata injection Control/Status ETW ETL file ETW Session Event Providers 1. Collection of configurable in-memory buffers that is managed by the kernel Event Providers 2. Any component that has been instrumented with Event Tracing API Control/Status Dataflow

17. How Does Performance Analyzer Pull This Off? • ETW is the magic behind the curtains • Event Tracing for Windows • Core component since Windows 2000 • High performance, low overhead, highly scalable • ~2.5% CPU usage for sustained rate of 10,000 events/sec on a 2GHz CPU • For details, see “Event Tracing” on MSDN

18. You Got My Attention – What’s Next? • Motivation • Overview of Windows Performance Toolkit • Key Concepts • Case Studies • Conclusion

19. Capture a Performance Trace • A performance trace is a persisted form of a (complex) performance measurement • …Perform the scenario in question… • > xperf –start perf!GeneralProfiles.InSequentialFile • > xperf –stop perf!GeneralProfiles.InSequentialFile trace.etl

20. What is CPU Time Spent On? • xperf trace.etl

21. CPU Summary Table Grouping columns Aggregated columns

22. CPU Summary Table

23. Now Show Me Some Action • Motivation • Overview of Windows Performance Toolkit • Key Concepts • Case Studies • Conclusion

24. Disk Usage Case Study #1 Marking 875 messages in Outlook 2007 folder as read is terribly slow

25. Marking 875 messages in Outlook 2007 folder as read is terribly slow

26. Marking 875 messages in Outlook 2007 folder as read is terribly slow Throughput can be cut by 90%

27. Disk Usage Study Summary • Multiple components working well in isolation could impact performance when executed in parallel • Fighting over disk dramatically impacts performance for all involved • Disk seeks can cut throughput by 90% • Schedule all work that can wait as Idle tasks • See Task Scheduler documentation on MSDN Microsoft Confidential

28. Everything is slow Case Study #2 User complained that system was very slow across multiple scenarios including boot

29. User complained that system was very slow across multiple scenarios including boot

30. User complained that system was very slow across multiple scenarios including boot

31. Case Study: Slow Disk • Symptom: • Very slow system across multiple scenarios, including boot • Key Findings: • Very high disk utilization throughout the trace • Disk was taking a long time to service I/Os

32. CPU Usage Case Study #3 User browsed E:\ (SD) drive from Explorer and opened a photo from the drive.Audio playback glitched.

33. User browsed E:\ (SD) drive from Explorer and opened a photo from the drive. Audio playback glitched.

34. CPU Case Study Summary • One bad component can disrupt numerous good components • High usage of any single resource can cause performance issues • Spending too much CPU at DPC impacts performance • Use DMA instead of PIO when transferring data

35. What Have We Learned? • Motivation • Key Concepts • Overview of Windows Performance Toolkit • Case Studies • Conclusion

36. Best Practices • Resource efficiency • Responsiveness • Good Citizenship

37. Summary • Good performance  customer satisfaction • WPT = official perf tools from/for Windows • Performance Analyzer • xperf: Trace capture, processing, and command-line analysis • xperfview:Visual trace analysis • xbootmgr:On/Off transition trace capture

38. Call to Action • Download WPT Kit (see resources) • Capture traces across key scenarios • Evaluate resource utilization • Identify resource bottleneck • Invest in ETW instrumentation • Visit Windows Performance Analysis Dev Center • Use the public forum

39. Resources • Windows Performance Analysis Dev Center: • http://msdn.microsoft.com/en-us/performance • Event Tracing for Windows MSDN docs: • http://msdn2.microsoft.com/en-us/library/aa363787.aspx • Windows Internals • 5th edition is now out!

40. YOUR FEEDBACK IS IMPORTANT TO US! Please fill out session evaluation forms online at MicrosoftPDC.com

41. Learn More On Channel 9 • Expand your PDC experience through Channel 9 • Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses channel9.msdn.com/learn Built by Developers for Developers….

43. Addendum

44. Key new features since PDC’08 • WPT 4.5 (Win7 RC) • Generic Events support • Hierarchical legends • Configuration of symbol/symcache repository from the GUI • Stack tracing support for manifest-based events • Hardware power analysis • Idle CPU analysis

45. Key new features since PDC’08 • WPT 4.6 (Win7 RTM) • Column reordering in summary tables via the column chooser • Assisted resizing of graphs • Auto-snapping selection in check-point and count graphs • System Configuration dialog box tabs are now fully featured summary tables • Flush visualization in the detailed disk I/O graph • Single pass multiple action execution • Auto-scaling of Y-axis in graphs based on presented data series • Per-provider filtering in dumper • ReadyBoot analysis