Capacity Planning and Benchmarking Chapter 9 of The Art of Computer Systems Performance Analysis by Raj Jain

1. Capacity Planning and Benchmarking(Chapter 9 of �The Art of Computer Systems Performance Analysis� by Raj Jain) Presenter: Mikael Fernandus Simalango WISE Research Lab

2. Agenda Definitions and Terms Steps in Capacity Planning and Management Problems in Capacity Planning Common Mistakes in Benchmarking Computer System Performance Measurement Load Drivers Remote Terminal Emulation Summary and Conclusion

3. Definitions and Terms Capacity planning: Ensuring that adequate computer resources will be available to meet the future workload demands Should be cost-effective and meet the performance objectives Capacity management: Ensuring that currently available computing resources are used to provide the highest performance Performance tuning: The process of adjusting system parameters to optimize the performance. Benchmarking: Comparing the performance of two competing systems Is run on systems using automatic load drivers

4. Steps in Capacity Planning and Management Steps in capacity planning process

5. Steps in Capacity Planning and Management Steps in capacity management

6. Problems in Capacity Planning Common problems: No standard terminology Different definitions from different vendors No standard definition of capacity Maximum throughput ? Maximum number of users? Different capacities for the same system Nominal capacity: maximum achievable throughput under ideal workload conditions Usable capacity: maximum throughput achievable without exceeding a prespecified response time limit Knee capacity: optimal operating point of throughput and workload No standard workload unit Users, sessions, MIPS?

7. Problems in Capacity Planning Common problems: Forecasting future applications is difficult Regression vs technology advancement which may invalidate the previous assumption No uniformity among systems from different vendors Same workload but different amounts of resources on different system Model input cannot always be measured Input defined in simulation may not be conforming in real world Validating model projections is difficult Difficult to control the workload and configuration on a real system Distributed environments are too complex to model Performance is only a small part of capacity planning problem

8. Common Mistakes in Benchmarking List of repeated mistakes: Only average behavior represented in test workload Skewness of device demands ignored Loading level controlled inappropriately Caching effects ignored Buffering sizes not appropriate Inaccuracies due to sampling ignored Ignoring monitoring overhead Not validating measurements Not ensuring same initial conditions Not measuring transient performance Using device utilizations for performance comparisons Collecting too much data with very little analysis

9. Computer System Performance Measurement � Load Drivers Load driving: putting loads on the system Load drivers: mechanism or tool for putting loads on the system Purpose of load drivers: Performance measurement Component certification -> ensuring functionality of component under different workload demands System integration -> verifying compatibility of components under different environments Stress-load analysis -> Analysis of system stability for high load Regression testing -> ensuring previous capabilities are functional along with the new ones

10. Computer System Performance Measurement � Load Drivers Load driving techniques: Internal driver: loading programs directly into the memory and executing -> applying a batch job Operators: having live operators utilize the system Remote Terminal Emulators (RTEs): using computers to simulate many users in a very controlled and repeatable fashion

11. Computer System Performance Measurement � RTE About RTE: A full fledged computer which includes disks, persistent storage, and at least one console terminal Executes load driving by sending commands to the SUT at appropriate intervals through an automated script. Has three components: Pre-emulation: defining system configurations (terminals, users, programs) Emulation: collecting data of SUT by RTE Post-emulation: transferring data to another computer system for reduction and analysis

12. Computer System Performance Measurement � RTE RTE Components

13. Computer System Performance Measurement � RTE Limitations of current RTEs: The conditions for sending successive requests may not be realistic (should use probabilistic transmissions with varying contents of requests instead) It may not be possible to vary the input in successive repetitions Modern user interfaces are not emulated (mouse click, function keys, etc) Users are the shared resource (multiple requests at the same time) Interfaces and definitions are different across RTEs

14. Summary and Conclusion Capacity planning and management is a chain of activities Ensuring that user requests can be served at the optimized cost and performance And anticipating future workload demands RTEs: Trying to emulate workloads of SUT Resulting in optimized system profile/configuration for processing the workloads Having some limitations which may render analysis of emulated data inappropriate with real-world scenario

15. Summary and Conclusion Linking capacity planning with cloud computing, we may derive: Understanding that capacity planning is related to performance and scalability of the cloud Questions about what parameters are essential for measuring the capacity of the cloud in relation with its performance Questions about how to model a cloud system and get the optimized capacity Questions about how to implement a high availability cloud based on the derived capacity model