David J. Lilja Department of Electrical and Computer Engineering University of Minnesota

1. Exploiting Incorrectly Speculated Memory Operations in a Concurrent Multithreaded Architecture(Plus a Few Thoughts on Simulation Methodology) David J. Lilja Department of Electrical and Computer Engineering University of Minnesota lilja@ece.umn.edu

2. Acknowledgements • Graduate students (who did the real work) • Ying Chen • Resit Sendag • Joshua Yi • Faculty collaborator • Douglas Hawkins (School of Statistics) • Funders • National Science Foundation • IBM • HP/Compaq • Minnesota Supercomputing Institute

3. Problem #1 • Speculative execution is becoming more popular • Branch prediction • Value prediction • Speculative multithreading • Potentially higher performance • What about impact on the memory system? • Pollute cache/memory hierarchy? • Leads to more misses?

4. Problem #2 • Computer architecture research relies on simulation • Simulation is slow • Years to simulate SPEC CPU2000 benchmarks • Simulation can be wildly inaccurate • Did I really mean to build that system? • Results are difficult to reproduce • Need statistical rigor

5. Outline (Part 1) • The Superthreaded Architecture • The Wrong Execution Cache (WEC) • Experimental Methodology • Performance of the WEC [Chen, Sendag, Lilja, IPDPS, 2003]

6. Hard-to-Parallelize Applications • Early exit loops • Pointers and aliases • Complex branching behaviors • Small basic blocks • Small loops counts → Hard to parallelize with conventional techniques.

7. Introduce Maybe dependences • Data dependence? • Pointer aliasing? • Yes • No • Maybe • Maybe allows aggressive compiler optimizations • When in doubt, parallelize • Run-time check to correct wrong assumption.

8. CONTINUATION -Values needed to fork next thread Fork Fork TARGET STORE -Forward addresses of maybe dependences CONTINUATION -Values needed to fork next thread … … Fork Sync Sync TARGET STORE -Forward addresses of maybe dependences … COMPUTATION -Forward addresses and computed data as needed CONTINUATION -Values needed to fork next thread … … Sync COMPUTATION -Forward addresses and computed data as needed TARGET STORE -Forward addresses of maybe dependences … WRITE-BACK COMPUTATION -Forward addresses and computed data as needed Sync Sync Thread i WRITE-BACK Sync Thread i+1 WRITE-BACK Thread i+2 Thread Pipelining Execution Model

9. Instruction Cache Super-Scalar Core Super-Scalar Core Super-Scalar Core Super-Scalar Core Registers Registers Registers Registers PC Execution Unit PC Execution Unit PC Execution Unit PC Execution Unit Comm Dependence Buffer Comm Dependence Buffer Comm Dependence Buffer Comm Dependence Buffer Data Cache The Superthread Architecture

10. Predicted path Speculative execution Correct path Prediction result is wrong Wrong path Wrong path execution Not ready to be executed Wrong Path Execution Within Superscalar Core

11. Parallel region Parallel region Sequential region Kill all the wrong threads from the Previous parallel region Mark the successor threads as wrong threads Sequential region between two parallel regions Wrong thread kills itself Wrong Thread Execution

12. How Could Wrong Thread Execution Help Improve Performance? When i=4, j=0,1,2,3=>y[0], y[1], y[2], y[3],y[4]… When i=5, j=0,1,2,3,4 =>y[0],y[1],y[2],y[3],y[4],y[5]… for (i=0; i<10; i++) { …… for (j=0; j<i; j++) { …… x=y[j]; …… } …… } i=4 TU1 TU2 TU3 TU4 y[0] y[1] y[2] y[4] y[3] y[5] i=5 TU1 TU2 TU3 TU4 y[0] y[1] y[2] Parallelized y[4] y[3] y[5] y[6] wrong threads

13. Correct execution Wrong execution Operation of the WEC

14. Processor Configurations for Simulations SIMCA (the SIMulator for the Superthreaded Architecture) features configurations

15. Parameters for Each Thread Unit

16. Characteristics of the Parallelized SPEC2000 Benchmarks

17. Performance of the Superthreaded Architecture for the Parallelized Portions of the Benchmarks Baseline configuration

18. Performance of the wth-wp-wec Configuration on Top of the Parallel Execution

19. Performance Improvements Due to the WEC

20. Sensitivity to L1 Data Cache Size

21. Sensitivity to WEC Size Compared to a Victim Cache

22. Sensitivity to WEC Size Compared to Next-Line Prefetching (NLP)

23. Additional Loads and Reduction of Misses %

24. Conclusions for the WEC • Allow loads to continue executing even after they are known to be incorrectly issued • Do not let them change state • 45.5% average reduction in number of misses • 9.7% average improvement on top of parallel execution • 4% average improvement over victim cache • 5.6% average improvement over next-line prefetching • Cost • 14% additional loads • Minor hardware complexity

25. Typical Computer Architecture Study • Find an interesting problem/performance bottleneck • E.g. Memory delays • Invent a clever idea for solving it. • This is the hard part. • Implement the idea in a processor/system simulator • This is the part grad students usually like best • Run simulations on n “standard” benchmark programs • This is time-consuming and boring • Compare performance with and without your change • Execution time, clocks per instruction (CPI), etc.

26. Problem #2 – Simulation in Computer Architecture Research • Simulators are an important tool for computer architecture research and design • Low cost • Faster than building a new system • Very flexible

27. Performance EvaluationTechniques Used in ISCA Papers * Some papers used more than one evaluation technique.

28. Simulation is Very Popular, But … • Current simulation methodology is not • Formal • Rigorous • Statistically-based • Never enough simulations • Design a new processor based on a few seconds of actual execution time • What are benchmark programs really exercising?

29. An Example -- Sensitivity Analysis • Which parameters should be varied? Fixed? • What range of values should be used for each variable parameter? • What values should be used for the constant parameters? • Are there interactions between variable and fixed parameters? • What is the magnitude of those interactions?

30. Let’s Introduce Some Statistical Rigor • Decreases the number of errors • Modeling • Implementation • Set up • Analysis • Helps find errors more quickly • Provides greater insight • Into the processor • Effects of an enhancement • Provides objective confidence in results • Provides statistical support for conclusions

31. Outline (Part 2) • A statistical technique for • Examining the overall impact of an architectural change • Classifying benchmark programs • Ranking the importance of processor/simulation parameters • Reducing the total number of simulation runs [Yi, Lilja, Hawkins, HPCA, 2003]

32. A Technique to Limit the Number of Simulations • Plackett and Burman designs (1946) • Multifactorial designs • Originally proposed for mechanical assemblies • Effects of main factors only • Logically minimal number of experiments to estimate effects of m input parameters (factors) • Ignores interactions • Requires O(m) experiments • Instead of O(2m) or O(vm)

33. Plackett and Burman Designs • PB designs exist only in sizes that are multiples of 4 • Requires X experiments for m parameters • X = next multiple of 4 ≥ m • PB design matrix • Rows = configurations • Columns = parameters’ values in each config • High/low = +1/ -1 • First row = from P&B paper • Subsequent rows = circular right shift of preceding row • Last row = all (-1)

34. PB Design Matrix

35. PB Design Matrix

36. PB Design Matrix

37. PB Design Matrix

38. PB Design Matrix

39. PB Design Matrix

40. PB Design • Only magnitude of effect is important • Sign is meaningless • In example, most → least important effects: • [C, D, E] → F → G → A → B

41. Case Study #1 • Determine the most significant parameters in a processor simulator.

42. Determine the Most Significant Processor Parameters • Problem • So many parameters in a simulator • How to choose parameter values? • How to decide which parameters are most important? • Approach • Choose reasonable upper/lower bounds. • Rank parameters by impact on total execution time.

43. Simulation Environment • SimpleScalar simulator • sim-outorder 3.0 • Selected SPEC 2000 Benchmarks • gzip, vpr, gcc, mesa, art, mcf, equake, parser, vortex, bzip2, twolf • MinneSPEC Reduced Input Sets • Compiled with gcc (PISA) at O3

44. Functional Unit Values

45. Memory System Values, Part I

46. Memory System Values, Part II

47. Processor Core Values

48. Determining the Most Significant Parameters 1. Run simulations to find response • With input parameters at high/low, on/off values

49. Determining the Most Significant Parameters 2. Calculate the effect of each parameter • Across configurations

50. Determining the Most Significant Parameters 3. For each benchmark Rank the parameters in descending order of effect (1=most important, …)