StickyMinds and Better Software magazine presents…

1. StickyMinds.com and Better Software magazine presents… Whose Environment is It Anyway? • Performance Tuning in a Virtual Environment. Sponsored by Empirix • Non-streaming participants should call 1-866-761-8643 • International Non-streaming participants should call 1-904-596-2362

2. A Heuristic Approach Scott Barber, (A.K.A. “The Perf Guy”)sbarber@perftestplus.com Chief Technologist, PerfTestPlus, Inc.www.perftestplus.com Vice President & Executive Director, Association for Software Testingwww.associationforsoftwaretesting.org Co-Author, Performance Testing Guidance for Web Applications www.amazon.com/gp/product/0735625700

3. Testing in a Non-Virtual World Developers write code Developers release code for test Testers test code, often in various environments Testers assess system performance in not very production-like environments Managers make decisions based on test results Some defects get fixed before release The team gets surprised by defects in production The team puts in lots of overtime to fix the defects

4. What is Virtualization? • Wikipedia lists 8 categories of platform virtualization and 4 categories of resource virtualization • Platform Virtualization • Software simulates a computer environment on a host machine, usually called “virtual machines” • Applications run on “virtual machines” as if on a stand-alone hardware platform • As if stand-alone ≠ Isolated • Platform virtualization software has overhead • (http://en.wikipedia.org/wiki/Virtualization)

5. What is Virtualization? • Resource Virtualization • Identify a resource and exclusively assign it’s use to a specific application • Sometimes partitions single resources, sometimes spans multiple resources • Resources may or may not be co-located on a physical machine • Exclusively assigned ≠ Isolated • Resource virtualization has overhead • (http://en.wikipedia.org/wiki/Virtualization)

6. Testing in a Virtual World Developers write code Developers release code for test Testers test code, often in various configurations Testers assess system performance in not very production-like environments Managers make decisions based on test results Some defects get fixed before release The team gets surprised by defects in production The team puts in lots of overtime to fix the defects

7. What’s the Difference? • Virtualization, if done well, can: • Dramatically reduce the overhead involved in managing test environments • Create opportunities for unique, targeted, ad-hoc environments without additional hardware costs • Simplify configuration and release management • Put the test team in control of their test environments • But what about performance testing?

8. “With an order of magnitude fewer variablesperformance testing could be a science,but for now, performance testing is at best a scientific art.” --Scott Barber

9. Testing Virtual Performance?!? • Test environments are still test environments • Virtual test environments are less likely to match virtual production environments • There are more places for performance issues to hide in virtual environments • Exclusively assigned ≠ Isolated • As if stand-alone ≠ Isolated • Virtualization adds overhead • Study: “virtualized servers can handle 14% lower [load] than traditional server configurations” (http://www.sys-con.com/read/503412.htm)

10. “Linear extrapolation of performance test results is, at best, black magic. Don’t do it. (Unless you learned from Connie Smith, PhD or Daniel Menasce, PhD)” --Scott Barber

11. Testing Virtual Performance?!? • If done well, there can be benefits: • Dramatically reduce the overhead involved in managing test environments • Easier for performance testing to have it’s own test environment • Enable collaborative tuning without having to “borrow” hardware resources • Put the test team in control of their test environments • Virtualization does not change the core principles of successful performance testing

12. “Only performance testing at the conclusion of system or functional testing is likeordering a diagnostic blood test after the patient is dead.” --Scott Barber

13. Core Principles Project context is central to successful performance testing. Business, project, system, & user success criteria. Identify system usage, and key metrics; plan & design tests. Install & prepare environments, tools, & resource monitors. Script the performance tests as designed. Run and monitor tests. Validate tests, test data, and results. Analyze the data individually and as a cross-functional team. Consolidate and share results, customized by audience. "Lather, rinse, repeat" as necessary.

14. Contact Info Scott Barber Chief Technologist PerfTestPlus, Inc E-mail: sbarber@perftestplus.com Web Site: www.PerfTestPlus.com

15. IMAGE Whose Environment Is It Anyway? Dan Koloski Web BU CTO and Director of Strategy Empirix

16. Testing & Monitoring Solutions for: Privately held, founded 2000 380+ employees Worldwide presence; HQ in Boston & offices in Japan, UK, Germany and South Korea Patents in test, monitoring and VoIP diagnostics methodology Empirix – At a Glance Web Applications & SOA Carrier-Class VoIP & Telco Contact Center & CRM

17. Empirix in the Quality Ecosystem • Empirix Offerings Delivered Outcome PRODUCTS e-TEST suite™ OneSight™ Do-it-yourself MANAGED SERVICES Remote Validation Performance Tuning e-LoadExpert™ EXPERT SERVICES Supplement Your Team We Do it for You Onsite Consulting, Training, Mentoring

18. A few points to add to Scott’s presentation… • Scott said… • Virtualization does not change the core principles of successful performance testing • Why? • Hardware is not virtual • Whose environment is it, anyway? • Rapid Bottleneck Identification Tuning a Virtualized Context

19. Hardware is not virtual • Virtualization is app/OS or hardware emulation specific • Underneath it all is a fixed pool of HW resources! • Performance Testing is designed to understand the impact on the hardware – as seen from the perspective of the user Common Virtualization Types. Source: QAZone Whitepaper, Virtualization from the perspective of a QA/IT Operation Professional, http://qazone,empirix.com.

20. The Best Performance Data in a Virtual Environment • Understand the impact on the hardware… • Requires accurate workload stimulus representing activity on and through all VMs associated with that device • Requires measurement of hardware characteristics at the base layer as well as OS and app-level characteristics in each VM • Requires real-time collaboration during load testing to differentiate intra-VM bottlenecks (business logic) vs Hardware bottlenecks (CPU, Memory, NIC) • …as seen from the perspective of the user! • Your users don’t care about Virtualization • Metrics that matter are end-user perceived performance • Performance testing from inside the VM helps create stimulus but doesn’t report on end-user-perceived performance

21. Rapid Bottleneck Identification Load Testing in a Virtual Environment • What is Load testing? “Testing conducted to isolate and identify the system and application issues (bottlenecks) that will keep the application from scaling to meet its performance requirements”

22. What is a Bottleneck? • Any resource (hardware, software or bandwidth) that places defining limits on data flow or processing speed: your application is only as efficient as its least efficient element • On the Web, bottlenecks directly affect performance and scalability • Most untested systems have more than one bottleneck, but they can only be identified and resolved one at a time.

23. Where are the Bottlenecks? The vast majority of bottlenecks are caused by throughput limitations – not concurrency. Source: Empirix

24. Sample throughput limitations in virtual environments • Disk I/O • Multiple applications accessing shared disk resources • Virtual Memory Paging on shared disk resources • NIC • Multiple VMs shipping data through the same physical NIC • CPU • Processor-intensive (i.e., database) applications will compete for shared CPU time. • Memory • Depends on type of VM configuration how much of an issue this can be

25. Whose Environment Is It Anyway? • Virtualization blurs the line between Dev/Test/Ops environments and ownership • Can exacerbate “silo” problems between Dev/Test/Ops • However, there is an opportunity… • …A common set of measurements can un-blur the line • Test data should inform production SLAs • Common measurement/metrics can bridge the gap • Measurement consistency is the cornerstone of ALM

26. Load Testing Tools DON’T Collaborate • Load Testing Tools Apply Load and Make Measurements • People (and their expertise & experience) Find Bottlenecks & Resolve Them • Collaboration With Experts Resolves Issues Faster & Cheaper • Key to Successful Collaboration = Access to People & Access to Measurements • Tools Should Support Collaboration

27. e-TEST suite is designed for collaboration • Distributed, high-scale load control • Multi-user Web UI for real-time individual analysis Each user can view results from the same load test as it runs

28. Summary • Hardware is not virtual – virtualization does NOT change core performance test best practice • However… • Virtualization introduces new test planning challenges to cover • Workload/stimulus • Measurement of impact on end-user • In a world of scarce time, focus efforts on throughput-related workload to identify most common bottlenecks

29. Visit QAZone for the slides • http://qazone.empirix.com • Free resource, open to all (Empirix & non-Empirix users) • Forums (both QA-centric and Product related discussions) • Knowledge Base • Resource Center • Events, Announcements, Community & Industry News • Allows members to interact with peers, industry experts and a larger Empirix audience

30. Q & A Have a question for the speakers? Ask now.