Performance and Power Benchmarks

1. Zhang Yu Performance and Power Benchmarks

2. A long time ago in silicon valley far far away…. Tired? Let’s begin with a story

3. A few years ago (like centuries in electronic world) frequency was considered the key element of a cpu`s performance, because of Intel`s dominance in the market place and their continuous bragging about frequency. P4 vs K8

4. In November 2000 Intel's heavily advertised advances in clock speed reached an extreme with the release of the Pentium 4 which sacrificed per-cycle performance and used a deep instruction pipeline to gain higher clock speeds, ignoring problems that this introduced of heat production and power consumption. A time bomb was buried…

5. Intel: “I am the fastest!” AMD, Apple etc. : “No you are not!” Argument aroused

6. No argue but benchmark! They decided to settle this…. Let`s race!

7. Benchmark start

8. Bunny is fast indeed but unfortunately prone to make mistakes…. Game over

9. Deep instruction pipeline (e.g. if a prediction is wrong performance penalty will be greater because of deeper pipeline.) higher clock speeds introduced heat production and power consumption mistakes

10. As computer architecture advanced, it became more difficult to compare the performance of various computer systems simply by looking at their specifications. Therefore, tests were developed that allowed comparison of different architectures Benchmarks are particularly important in CPU design, giving processor architects the ability to measure and make tradeoffs in microarchitectural decisions How to determine performance?

11. Benchmarks are designed to mimic a particular type of workload on a component or system How does it work

12. Industry Standard (audited and verifiable) • Business Applications Performance Corporation (BAPCo) • Embedded Microprocessor Benchmark Consortium (EEMBC) • Standard Performance Evaluation Corporation (SPEC) • Transaction Processing Performance Council (TPC) Common benchmarks

13. BAPCo: MobileMark, SYSmark, WebMark Futuremark:3DMark, PCMark Whetstone PiFast Super PI WinSAT, exclusively for Windows Vista, providing an index for consumers to rate their systems easily Microsoft Windows benchmarks

14. Benchmarking is not easy and often involves several iterative rounds in order to arrive at predictable, useful conclusions. Vendors tend to tune their products specifically for industry-standard benchmarks Challenges

15. Benchmarks do not address questions which you did not ask… only see what you measure • Specific application benchmarks will not tell you about the performance of other applications without proper analysis => extrapolation is not straightforward • You must understand the benchmark itself to understand what it tells you • General benchmarks will not tell you about the details of your specific application Limitations and Pitfalls ofBenchmarks

16. When more power is used, a portable system will have a shorter battery life and require recharging more often. This is often the antithesis of performance as most semiconductors require more power to switch faster Power issues

17. nVidia`s Geforce graphic card: The cooling fan on this card is so powerful that it can be used to clean the door way. Nvidia’s joke

18. In 2004 problems of overheating led Intel to abandon further development of its Netburst microarchitecture Say goodbye to P4

19. The newest version of SPEC (SPECpower_ssj2008) counts power consumption as one of the key factor in performance. overall ssj_ops/Watt Nowadays we often use Gflops/Watt as an indicator of performance Trend: Power! Power! power!

20. http://en.wikipedia.org/wiki/Computer_benchmark http://www.spec.org/power_ssj2008/ http://en.wikipedia.org/wiki/Megahertz_myth http://en.wikipedia.org/wiki/Performance_per_watt reference