1 / 19

Analyzing Ratio Games: Metrics, Guidelines, and Case Studies in Performance Comparison

This presentation explores the concept of ratio games in performance evaluation of computer systems. It covers various examples and appropriate ratio metrics to contextualize performance comparisons. Key topics include the use of percentages, guidelines for ratio games, and numerical conditions for accurate assessments. Through detailed case studies, such as the comparison between the 6502 and 8080 processors, as well as RISC vs. CISC architectures, the presentation highlights the importance of context and methodology in performance evaluations, ultimately aiming to enhance decision-making in system selection.

carter-burris
Télécharger la présentation

Analyzing Ratio Games: Metrics, Guidelines, and Case Studies in Performance Comparison

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ratio Games Raj Jain Washington University in Saint LouisSaint Louis, MO 63130Jain@cse.wustl.edu These slides are available on-line at: http://www.cse.wustl.edu/~jain/cse567-08/

  2. Overview • Ratio Game Examples • Using an Appropriate Ratio Metric • Using Relative Performance Enhancement • Ratio Games with Percentages • Ratio Games Guidelines • Numerical Conditions for Ratio Games

  3. Case Study 11.1: 6502 vs. 8080 • Conclusion: 6502 is worse. It takes 4.7% more time than 8080. 1. Ratio of Totals

  4. 6502 vs. 8080 (Cont) 2. 6502 as the base: 3. 8080 as the base: • Ratio of Totals: 6502 is worse. It takes 4.7% more time than 8080. • With 6502 as a base: 6502 is better. It takes 1% less time than 8080. • With 8080 as a base: 6502 is worse. It takes 6% more time.

  5. Case Study 11.2: RISC vs. CISC • Conclusion: RISC-I has the largest code size. The second processor Z8002 requires 9% less code than RISC-I.

  6. RISC vs. CISC (Cont) • Conclusion: Z8002 has the largest code size and that it takes 18% more code than RISC-I. [Peterson and Sequin 1982]

  7. Using an Appropriate Ratio Metric Example: • Throughput: A is better • Response Time: A is worse • Power: A is better

  8. Using Relative Performance Enhancement • Example: Two floating point accelerators • Problem: Incomparable bases. Need to try both on the same machine

  9. Ratio Games with Percentages • Example: Tests on two systems 1. System B is better on both systems 2. System A is better overall. System A: System B:

  10. Percentages (Cont) • Other Misuses of Percentages: • 1000% sounds more impressive than 11-time. Particularly if the performance before and after the improvement are both small • Small sample sizes disguised in percentages • Base = Initial. 400% reduction in prices  Base = Final

  11. Ratio Games Guidelines • If one system is better on all benchmarks, contradicting conclusions can not be drawn by any ratio game technique

  12. Guidelines (cont) • Even if one system is better than the other on all benchmarks, a better relative performance can be shown by selecting appropriate base. • In the previous example, System A is 40% better than System B using raw data, 43% better using system A as a base, and 42% better using System B as a base. • If a system is better on some benchmarks and worse on others, contracting conclusions can be drawn in some cases. Not in all cases. • If the performance metric is an LB metric, it is better to use your system as the base • If the performance metric is an HB metric, it is better to use your opponent as the base • Those benchmarks that perform better on your system should be elongated and those that perform worse should be shortened

  13. Numerical Conditions for Ratio Games • A is better than B iff • Raw Data: • With A as the Base: • A is better than B iff

  14. Numerical Conditions (Cont) • A is better than B iff • With B as the base:

  15. Numerical Conditions (Cont) 2 B is betterusing all 3 Ratio of B/A response on benchmark j 1 A isbetterusing all 3 Base B Raw Data Base A 0 1 1 1 2 3 Ratio of B/A response on benchmark i

  16. Summary • Ratio games arise from use of incomparable bases • Ratios may be part of the metric • Relative performance enhancements • Percentages are ratios • For HB metrics, it is better to use opponent as the base

  17. Exercise 11.1 • The following table shows execution times of three benchmarks I, J, and K on three systems A, B, and C. Use ratio game techniques to show the superiority of various systems.

  18. Exercise 11.2 • Derive conditions necessary for you to be able to use the technique of combined percentages to your advantage.

  19. Homework • Read chapter 11 • Submit answer to Exercise 11.1

More Related