CPSC 531: Experiment Design and Performance Evaluation

1. CPSC 531: Experiment Design and Performance Evaluation Instructor: Anirban Mahanti Office: ICT 745 Email: mahanti@cpsc.ucalgary.ca Class Location: TRB 101 Lectures: TR 15:30 – 16:45 hours Class web page: http://pages.cpsc.ucalgary.ca/~mahanti/teaching/F05/CPSC531 Slides courtesy Professor Carey Williamson (minor modifications by Anirban Mahanti). CPSC 531: Experiment Design

2. PERFORMANCE EVALUATION • Often one needs to design and conduct an experiment in order to: • demonstrate that a new technique or concept is feasible • demonstrate that a new method is better than an existing method • understand the impact of various factors and parameters on the overall system performance CPSC 531: Experiment Design

3. PERFORMANCE EVALUATION • There is a whole field of computer science called computer systems performance evaluation that does exactly this • One of the best books is Raj Jain’s “The Art of Computer Systems Performance Analysis”, Wiley & Sons, 1991 (listed in bibliography) • Much of what is outlined in this presentation is described in more detail in [Jain 1991] CPSC 531: Experiment Design

4. PERF EVAL 101: THE BASICS • There are three main methods used in the design of performance studies • Experimental approaches • measurement and use of a real system • Analytic approaches • the use of mathematics, queueing theory, Petri Nets, abstract models, etc • Simulation approaches • design and use of computer simulations and simplified models to assess performance CPSC 531: Experiment Design

5. EXPERIMENTAL DESIGN AND METHODOLOGY • The design of a performance study requires great care in experimental design and methodology • Need to identify • experimental factors to be tested • levels (settings) for these factors • performance metrics to be used • experimental design to be used CPSC 531: Experiment Design

6. FACTORS • Factors are the main “components” or “things” that are to be varied in an experiment, because their impact on performance wants to be understood • Examples: switch size, network load, number of buffers at output ports • Need to choose factors properly, since the number of factors affects size of study CPSC 531: Experiment Design

7. LEVELS • Levels are the precise settings of the factors that are to be used in an experiment • Examples: switch size N = 2, 4, 8, 16 • Example: buffer size B = 100, 200, 400, 800 • Need to choose levels realistically • Need to cover reasonable portion of the design space CPSC 531: Experiment Design

8. PERFORMANCE METRICS • Performance metrics specify what you want to measure in your performance study • Examples: packet loss, packet delay • Must choose your metrics properly and instrument your experiment accordingly CPSC 531: Experiment Design

9. EXPERIMENTAL DESIGN • Experimental design refers to the organizational structure of your experiment • Need to methodically go through factors and levels to get the full range of experimental results desired • There are several “classical” approaches to experimental design CPSC 531: Experiment Design

10. Experiment Design - Classification • One factor at a time • vary only one factor through its levels to see what the impact is on performance • Two factors at a time • vary two factors to see not only their individual effects, but also their interaction effects, if any • Full factorial • try every possible combination of factors and levels to see full range of performance results CPSC 531: Experiment Design

11. Example: Web Proxy Caching Filtered stream (misses) Filtering Input stream CPSC 531: Experiment Design

12. Example: Factors and Levels • Cache size • Cache Replacement Policy • Recency-based LRU • Frequency-based LFU-Aging • Size-based GD-Size • RAND • FIFO • Workload Characteristics • One-timers, Zipf slope, tail index, correlation, temporal locality model CPSC 531: Experiment Design

13. Example: Performance Metrics • Document Hit Ratio (DHR) • What percentage of the documents requested by the clients are satisfied directly by the proxy, without having to obtain from the Web server? • Byte Hit Ratio (BHR) • What percentage of the bytes requested by the clients are satisfied directly by the proxy, without having to obtain from the Web server? CPSC 531: Experiment Design

14. Example: Performance Results Workload: Weak temporal locality Workload: Strong temporal locality Document Hit Ratio CPSC 531: Experiment Design

15. OTHER ISSUES • Simulation run length • choosing a long enough run time to get statistically meaningful results (equilibrium) • Simulation start-up effects and end effects • deciding how much to “chop off” at the start and end of simulations to get proper results • Replications • ensure repeatability of results, and gain greater statistical confidence in the results given • Presentation of results CPSC 531: Experiment Design

16. Presentation of Results • Graphs and tables are the two most common ways of illustrating and/or summarizing data • graphs can show you the trends • tables provide the details • There are good ways and bad ways to do each of these • Again, it is a bit of an “art”, but there are lots of good tips and guidelines as well CPSC 531: Experiment Design

17. Table Tips • Decide if a table is really needed; if so, should it be part of main report, or just an appendix? • Choose formatting software with which you are familiar; easy to import data, export tables • Table caption goes at the top • Clearly delineate rows and columns (lines) • Logically organize rows and columns • Report results to several significant digits • Be consistent in formatting wherever possible CPSC 531: Experiment Design

18. Graphing Tips • Choose a good software package, preferably one with which you are familiar, and one for which it is easy to import data, export graphs • Title at top; caption below (informative) • Labels on each axis, including units • Logical step sizes along axes (10’s, 100’s…) • Make sure choice of scale is clear for each axis (linear, log-linear, log-log) • Graph should start from origin (zero) unless there is a compelling reason not to do so CPSC 531: Experiment Design

19. Graphing Tips (cont’d) • Make judicious choice of type of plot • scatter plot, line graph, bar chart, histogram • Make judicious choice of line types • solid, dashed, dotted, lines and points, colours • If multiple lines on a plot, then use a key, which should be well-placed and informative • If graph is “well-behaved”, then organize the key to match the lines on the graph (try it!) • Be consistent from one graph to the next wherever possible (size, scale, key, colours) CPSC 531: Experiment Design

20. SUMMARY • Great care must be taken in experimental design and methodology if the experiment is to achieve its goal, and if results are to be fully understood • Computer systems performance evaluation defines standard methods for designing and conducting performance studies • Please follow these guidelines (where applicable) when doing assignments and course projects CPSC 531: Experiment Design