Advanced Pipelining

1. Advanced Pipelining Out of Order Processors COMP25212

2. Classic 5-stage pipeline • A single execution flow Inst Cache Data Cache Fetch Logic Decode Logic Exec Logic Mem Logic Write Logic

3. Modern Pipelines • Many execution flows Ld1 Ld2 Write Back Inst Cache Add1 Write Back Functional Units (FU) Fetch Decode Mul1 Mul2 Mul3 Write Back Div1 Div2 Div3 Write Back

4. ARM Pipelines In-order processor Out of order processor

5. Out of Order Execution The original order in a program is not preserved Processors execute instructions as input data becomes available Pipeline stalls due to conflicted instructions are avoided by processing instructions which are able to run immediately Take advantage of ILP Instructions per cycle increases

6. Conflicted Instructions • Cache misses: long wait before finishing execution • Structural Hazard: the required resources are not available • Data hazard: dependencies between instructions

7. Structural Hazards • Functional Units are typically not pipelined • This means only one instruction can use them at once • If all suitable Functional Units for executing an instruction are busy, then the instruction can not be executed • This is known as an Structural Hazard

8. True dependency r1 <- r2 op r3 r4 <- r1 op r5 Anti-dependency r1 <- r2 op r3 r2 <- r4 op r5 Output dependency r1 <- r2 op r3 … r1 <- r4 op r5 Read-after-write RAW Write-after-read WAR Write-after-write WAW Data dependencies

9. Dynamic Scheduling • Key Idea: Allow instructions behind stall to proceed. => Instructions executing in parallel. There are multiple execution units, so use them DIVD F0, F2, F4 ADDD F10, F0, F8 SUBD F12, F8, F14 • Enables out-of-order execution => out-of-order completion Even though ADDD stalls, the SUBD has no dependencies and can run. Dynamic pipeline scheduling overcomes the limitations of in-order pipelined execution by allowing out-of-order instruction execution

10. Out of Order Execution with Scoreboard

11. Scoreboard • The scoreboard is a centralized hardware mechanism • Instruction are executed as soon as their operands are available and there are no hazard conditions • It dynamically constructs the dependency graph by hardware for a window of instructions as they are issued in program order • The scoreboard is a “data structure” that provides the information necessary for all pieces of the processor to work together CDC6600(1963) (In Appendix A.8)

12. The Key idea of Scoreboards • Out-of-order execution divides ID stage: 1. Issue—decode instructions, check for structural hazards 2. Read operands—wait until no data hazards, then read operands • Scoreboard allows instruction to execute whenever 1 & 2 hold, not waiting for prior instructions • We will use In order issue, out of order execution, out of order commit ( also called completion)

13. Typical Scoreboard Structure

14. Stages of a Scoreboard Pipeline Execute Integer Write Back Execute FP Multiplication Write Back Execute FP Multiplication Issue Read Operands Write Back Execute FP Division Execute FP Add Write Back Write Back

15. Stages of a Scoreboard Pipeline 1. Issue —decode instructions & check for structural & WAW hazards (ID) If a functional unit for the instruction is free (no structural hazards) and no other active instruction has the same destination register (no WAW), the scoreboard issues the instruction to the functional unit and updates its internal data structure. If a structural or WAW hazard exists, then the instruction issue stalls, and no further instructions will issue until these hazards are cleared. 2. Read operands —wait until no data hazards, then read operands (RO) A source operand is available if no earlier issued active instruction is going to write it, or if the register containing the operand is being written by a currently active functional unit (no RAW). When the source operands are available, the scoreboard tells the functional unit to proceed to read the operands from the registers and begin execution. The scoreboard resolves RAW hazards dynamically in this step, and instructions may be sent into execution out of order. Always done in program order Can be done out of program order

16. Stages of a Scoreboard Pipeline 3. Execution—operate on operands (EX) The functional unit begins execution upon receiving operands. When the result is ready, it notifies the scoreboard that it has completed execution. 4. Write result—finish execution (WB) Once the functional unit has completed execution, the scoreboard checks for WAR hazards. If none, it writes results. If WAR, then it stalls the instruction. Example: DIVD F0,F2,F4 ADDD F10,F0,F8 SUBD F8,F8,F14 Scoreboard would stall SUBD completion until ADDD reads operands

17. Information within the Scoreboard 1. Instruction status—which of 4 steps the instruction is in 2. Functional unit status—Indicates the state of the functional unit (FU). 9 fields for each functional unit Busy—Indicates whether the unit is being used or not Op—Operation to perform in the unit (e.g., + or –) Fi—Destination register Fj, Fk—Source-register numbers Qj, Qk—Functional units producing source registers Fj, Fk Rj, Rk—Flags indicating when Fj, Fk are ready. Set to No after operands are read. 3. Register result status—Indicates which functional unit will write each register, if one exists. Blank when no pending instructions will write that register

18. A Scoreboard Example Functional Unit (FU) # of FUs EX cycles Integer Mem 1 1 Floating Point Multiply 2 10 Floating Point Add 1 2 Floating point Divide 1 40 The following code is run on a scoreboard pipeline with: L.D F6, 34(R2) L.D F2, 45(R3) MUL.D F0, F2, F4 SUB.D F8, F6, F2 DIV.D F10, F0, F6 ADD.D F6, F8, F2 Functional units are not pipelined

19. Dependency Graph For Example Code 1 2 3 4 5 6 L.D F6, 34(R2) L.D F2, 45(R3) MUL.D F0, F2, F4 SUB.D F8, F6, F2 DIV.D F10, F0, F6 ADD.D F6, F8, F2 1 3 2 4 6 5 L.D F2, 45 (R3) L.D F6, 34 (R2) MUL.D F0, F2, F4 DIV.D F10, F0, F6 SUB.D F8, F6, F2 ADD.D F6, F8, F2 Real Data Dependence (RAW) Anti-dependence (WAR) Output Dependence (WAW) Example Code Date Dependence: (1, 4) (1, 5) (2, 3) (2, 4) (2, 6) (3, 5) (4, 6) Output Dependence: (1, 6) Anti-dependence: (5, 6)

20. Scoreboard Example FU count down Clock cycle counter Instruction stream Functional Units: 1 Integer 2 Multiplication 1 Addition 1 Division

21. Scoreboard Example Cycle 1

22. Scoreboard Example Cycle 2

23. Scoreboard Example Cycle 3

24. Scoreboard Example Cycle 4

25. Scoreboard Example Cycle 5

26. Scoreboard Example Cycle 6

27. Scoreboard Example Cycle 7

28. Scoreboard Example Cycle 8a

29. Scoreboard Example Cycle 8b

30. Scoreboard Example Cycle 9

31. Scoreboard Example Cycle 11

32. Scoreboard Example Cycle 12

33. Scoreboard Example Cycle 13

34. Scoreboard Example Cycle 14

35. Scoreboard Example Cycle 15

36. Scoreboard Example Cycle 16

37. Scoreboard Example Cycle 17

38. Scoreboard Example Cycle 18

39. Scoreboard Example Cycle 19

40. Scoreboard Example Cycle 20

41. Scoreboard Example Cycle 21

42. Scoreboard Example Cycle 22

43. 39 cycles later…

44. Scoreboard Example Cycle 61

45. Scoreboard Example Cycle 62 • In-order issue, out-of-order execution and out-of-order completion.

46. Summary • Scoreboard techniques to deal with hazards: • Result forwarding to reduce or eliminate RAW hazards • Hazard detection hardware to stall the pipeline during hazards • Uses a hardware-based mechanism to rearrange instruction execution order to reduce stalls dynamically at runtime (dynamic scheduling) • Better dynamic exploitation of instruction-level parallelism (ILP)

47. Limitations of Scoreboard • The amount of parallelism available among a block of instructions • The number of score entries determines the window size (typically small ~5 instr.) • The number and types of functional units (Structural hazards increase with out of order) • The presence of antidependence and output dependences lead to WAR and WAW stalls.

48. Out of Order Execution with Tomasulo

49. Tomasulo’s Algorithm • Control logic for out-of-order execution is decentralized • Reservation Stations (RS) in the functional units keep instruction information • In addition RS seamlessly rename registers • A Common Data Bus (CDB) broadcasts data and results to the different devices • A single instruction can finish each cycle • Distributed control allows for a larger window of instructions – Dynamic scheduling

50. Tomasulo’s Algorithm • Structural hazards stall the pipeline • RS tracks when operands are available and buffers them as soon as they are • No need for a register bank (store values or sources) • Impact of RAW dependencies are limited • Execute an instruction when its operands are available • WAW and WAR dependencies are avoided • Register renaming