1 / 48

Methodology

This comprehensive guide delves into the critical aspects of performance evaluation in systems. It discusses the importance of defining operational loads and metrics, the significance of hidden factors, and the application of the scientific method in experiments. The text outlines how to identify overbearing bottlenecks and variations in performance under different conditions, emphasizing that proper experimentation is essential for valid conclusions. The reader will learn to distinguish between intensity and nature of loads, and gain insights into benchmarking and multidimensional metrics that drive performance analysis.

brigid
Télécharger la présentation

Methodology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Methodology easy but important 1

  2. ToC • What is performance evaluation about ? • Metrics, Load and Goals • Hidden Factors • The Scientific Method • Patterns 2

  3. Whatis Performance Evaluation ?

  4. Load • You need to define the load under which your system operates • Make the difference between • Intensity of the load (e.g. nb jobs per second) • Nature of the load • Statistical details that may matter: e.g. job sizes are heavy tailed or not • Benchmarks are artificial load generators; we will play with one of them

  5. Metric • Define a metric;examples • Response time • Power consumption • Throughput • Defineoperational conditions underwhichmetricismeasured (« Viewpoint », seeChapter 11)

  6. Compare Windows vs Linux 6

  7. Syscall Benchmark 7

  8. Memory Access Time 8

  9. Ghostscript 9

  10. Metrics are often Multidimensional 10

  11. A and D are non dominated 11

  12. Know your goals • A1 and A3 are comparisons, A2 is an absolute statement • E2 is an engineering rule

  13. 3. Hidden Factors • Factor: an element that may impact the performance • (desired factors): intensity of load, number of servers • (nuisance factors): time of the day, presence of denial of service attack 13

  14. TCP Throughput Increases with Mobility

  15. TCP ThroughputDecreaseswithMobility

  16. Why were we fooled ? • Hidden factor had a more important role than the factor we were interested in • We interpreted correlation as causality • Need to be aware of all factors and incorporate in the analysis • Or randomize experiment to reduce impact of hidden factors

  17. Simpson’sParadox • A wellknownphenomenon -- Special case of Hidden Factor paradoxwhenmetricissuccess rate and factors are discrete

  18. Berkeley Sex Case 1973 (source: wikipedia) 18

  19. Take Home Message • Pitfallnumber 1 is the presence of hidden factor • Anystudyis susceptible to it • Easy for opponents to find

  20. 4. Be Scientific • Joe measures performance of his Wireless Shop: • what would you conclude ?

  21. Scientific Method • Joe buys 2 more Access Points • improvement ? Before After 21

  22. Scientific Method • A conclusion can only proven to be wrong • Do not draw conclusions unless the experiment was designed to test the statement • Measurement 1 suggested that the wireless network was congested, but the experiment was not designed to test this statement • Joe should: design an experiment to validate:H1: “the wireless network is the bottleneck” • for example: measure the number of collisions / packet loss • result: collision · 1%; conclusion: H1 is not valid • hypothesis H2: the server is saturated • experiment: measure memory utilization : result ¼ 100% 22

  23. Performance After Doubling Server Memory 23

  24. Example from Nitin Vaidya, Mobicom 2000 Tutorial, slides 298-299 24

  25. 25

  26. 26

  27. 27

  28. Use of Scientific Method TCP throughput may increase with mobility (1) Duration of link failure period is impacted by speed(2) It has a negative impact on TCP throughput Do more simulationsmeasure distrib of link failure period Verify (1) and (2) How ? • Recognize a fact • Pose a hypothesis • Verify the hypothesis on simulations / measurements designed to test it 28

  29. Is ATM-ABR better than ATM-UBR ? 29

  30. Take Home Message • You should not conclude from an experiment without trying to invalidate the conclusion • (Popper, 1934): you should alternate between the roles of • Proponent • Adversary

  31. 5. Patterns • These are common traits found in different situations • Knowing some of them may save a lot of time

  32. Bottlenecksmaybeyourenemy • Bottlenecks are like non invited people at a party – theymay impose their agenda • Previousexample: whatwe are measuringis the bottleneck, not the intended factor 33

  33. Bottlenecks are YourFriends

  34. Bottlenecks are YourFriends • Simplifyyour life, analyzebottlenecks ! • In many cases, youmay ignore the rest 35

  35. Behind a Bottleneck May HideAnotherBottleneck

  36. i i’ i” i” 37

  37. i i’ i” i” 38

  38. 39

  39. Congestion Collapse • Definition: Offeredloadincreases, workdonedecreases • Frequent in complexsystems • May be due to • cost per job increaseswithload • Impatience • Rejection of jobs beforecompletion • Designer must do something to avoid congestion collapse • Eg. Admission control in Apache servers • Eg. TCP congestion control • Analyst must look for congestion collapse 40

  40. Sources use TCP (= fair scheduling). Increase capacity of link 5 to 100 kb/s; what happens ? 41

  41. Competition Side Effect • System balances resources according to some scheduling • Apparent paradox: put more resources, some get less 42

  42. No TCP, users send as much can • Increase capacity of link 2 from 10 to 1000 kb/s 10 kb/s 43

  43. Competition Side Effect • Apparent paradox: put more resources, all get less 44

  44. Museum Audio Guide Lowspeed USB connections atdocking station High speed 45

  45. Latent Congestion Collapse • System is susceptible to congestion collapse • Low speed access prevents congestion collapse • Adding resources reveals congestion collapse 46

  46. Take Home Message • Watch for patterns, they are very frequent • Bottlenecks • Congestion collapse • Competition side effects • Latent Congestion collapse 47

  47. Now it’s your turn… 48

More Related