Superscalar Operation (Week 9)

1. Superscalar Operation(Week 9)

2. Superscalar Operation The maximum throughput of a pipelined processor is one instruction per clock cycle. Amore aggressive approach is to equip the processor with multiple execution units, each ofwhich may be pipelined, to increase the processor’s ability to handle several instructionsin parallel. With this arrangement, several instructions start execution in the same clockcycle, but in different execution units, and the processor is said to use multiple-issue. Suchprocessors can achieve an instruction execution throughput of more than one instructionper cycle. They are known as superscalar processors. Many modern high-performanceprocessors use this approach. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

3. Superscalar Operation To enable multiple-issue execution, a superscalar processor has a more elaborate fetchunit that fetches two or more instructions per cycle before they are needed and places them inan instruction queue. Aseparate unit, called the dispatch unit, takes two or more instructionsfrom the front of the queue, decodes them, and sends them to the appropriate execution units.At the end of the pipeline, another unit is responsible for writing results into the registerfile. Figure shows a superscalar processor with this organization. It incorporates twoexecution units, one for arithmetic instructions and another for Load and Store instructions.Arithmetic operations normally require only one cycle, hence the first execution unit issimple. Because Load and Store instructions involve an address calculation for the Indexmode before each memory access, the Load/Store unit has a two-stage pipeline. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

4. Superscalar Operation CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

5. Superscalar Operation This organization raises some important implications for the register file.An arithmetic instruction and a Load or Store instruction must obtain all their operandsfrom the register file when they are dispatched in the same cycle to the two execution units.The register file must now have four output ports instead of the two output ports needed inthe simple pipeline. Similarly, an arithmetic instruction and a Load instruction must writetheir results into the register file when they complete in the same cycle. Thus, the registerfile must now have two input ports instead of the single input port for the simple pipeline.There is also the potential complication of two instructions completing at the same timewith the same destination register for their results. This complication is avoided, if possible,by dispatching the instructions in a manner that prevents its occurrence. Otherwise, oneinstruction is stalled to ensure that results are written into the destination register in thesame order as in the original instruction sequence of the program. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

6. Superscalar Operation To illustrate superscalar execution in the processor, consider the followingsequence of instructions: Add R2, R3, #100 Load R5, 16(R6) Subtract R7, R8, R9 Store R10, 24(R11) Figure shows how these instructions would be executed. The fetch unit fetches twoinstructions every cycle. The instructions are decoded and their source registers are read inthe next cycle. Then, they are dispatched to the arithmetic and Load/Store units. Arithmeticoperations can be initiated every cycle. A Load or Store instruction can also be initiatedevery cycle, because the two-stage pipeline overlaps the address calculation for one Loador Store instruction with the memory access for the preceding Load or Store instruction. As instructions complete execution in each unit, the register file allows two results to bewritten in the same cycle because the destination registers are different. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

7. Superscalar Operation An example of instruction flow in the processor. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

8. Branches and Data Dependencies In the absence of any branch instructions and any data dependencies between instructions, throughput is maximized by interleaving instructions that can be dispatched simultaneously to different execution units. However, programs contain branch instructions that change the execution flow, and data dependencies between instructions that impose sequential ordering constraints. A superscalar processor must ensure that instructions are executed in the proper sequence. Furthermore, memory delays due to cache misses may occasionally stall the fetching and dispatching of instructions. As a result, actual throughput is typically below the maximum that is possible. The challenges presented by branch instructions and data dependencies can be addressed with additional hardware. We first consider branch instructions and then consider the issues stemming from data dependencies. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

9. Branches and Data Dependencies The fetch unit handles branch instructions as it determines which instructions to place in the queue for dispatching. It must determine both the branch decision and the target for each branch instruction. The branch decision may depend on the result of an earlier instruction that is either still queued or newly dispatched. Stalling the fetch unit until the result is available can significantly reduce the throughput and is therefore not a desirable approach. Instead, it is better to employ branch prediction. Since the aim is to achieve high throughput, prediction is also combined with a technique called speculative execution. In this technique, subsequent instructions based on an unconfirmed prediction are fetched, dispatched, and possibly executed, but are labeled as being speculative so that they and their results may be discarded if the prediction is incorrect. Additional hardware is required to maintain information about speculatively executed instructions and to ensure that registers or memory locations are not modified until the validity of the prediction is confirmed. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

10. Branches and Data Dependencies Data dependencies between instructions impose ordering constraints. A simple approachis to dispatch dependent instructions in sequence to the same execution unit, wheretheir order would be preserved. However, dependent instructions may be dispatched todifferent execution units. For example, the result of a Load instruction dispatched to theLoad/Storeunit may be needed by an Add instruction dispatched to the arithmeticunit. Because the units operate independently and because other instructions mayhave already been dispatched to them, there is no guarantee as to when the result neededby the Add instruction is generated by the Load instruction. A mechanism is needed toensure that a dependent instruction waits for its operands to become available. When aninstruction is dispatched to an execution unit, it is buffered until all necessary results fromother instructions have been generated. Such buffers are called reservation stations, andthey are used to hold information and operands relevant to each dispatched instruction. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

11. Branches and Data Dependencies Results from each execution unit are broadcast to all reservation stations with each resulttagged with a register identifier. This enables the reservation stations to recognize a resulton which a buffered instruction depends. When there is a matching tag, the hardware copiesthe result into the reservation station containing the instruction. The control circuit beginsthe execution of a buffered instruction only when it has all of its operands.In a superscalar processor using multiple-issue, the detrimental effect of stalls becomeseven more pronounced than in a single-issue pipelined processor. The compiler can avoidmany stalls through judicious selection and ordering of instructions. For example, for theprocessor of our example, the compiler should strive to interleave arithmetic and memoryinstructions. This enables the dispatch unit to keep both units busy most of the time. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

12. Out-of-Order Execution The instructions ın example are dispatched in the same order as they appear in theprogram. Add R2, R3, #100 Load R5, 16(R6) Subtract R7, R8, R9 Store R10, 24(R11) However, their execution may be completed out of order. For example, theSubtract instruction writes to register R7 in the same cycle as the Load instruction that wasfetched earlier writes to register R5. If the memory access for the Load instruction requiresmore than one cycle to complete, execution of the Subtract instruction would be completedbefore the Load instruction. Does this type of situation lead to problems? CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

13. Out-of-Order Execution We have already discussed the issues arising from dependencies among instructions. For example, if an instruction Ij+1 depends on the result of instruction Ij, the execution of Ij+1 will be delayed if the result is not available when it is needed. As long as such dependencies are handled correctly, there is no reason to delay the execution of an unrelated instruction. If there is no dependency between a pair of instructions, the order in which execution is completed does not matter. However, a new complication arises when we consider the possibility of an instructioncausing an exception. For example, the Load instruction may attempt anillegal unaligned memory access for a data operand. By the time this illegal operationis recognized, the Subtract instruction that is fetched after the Load instruction may havealready modified its destination register. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

14. Out-of-Order Execution Program execution is now in an inconsistentstate. The instruction that caused the exception in the original sequence is identified, but asucceeding instruction in that sequence has been executed to completion. If such a situationis permitted, the processor is said to have imprecise exceptions.The alternative of precise exceptions requires additional hardware. To guarantee aconsistent state when exceptions occur, the results of the execution of instructions mustbe written into the destination locations strictly in program order. This means that wemust delay writing into register R7 for the Subtract instruction until afterregister R5 for the Load instruction has been updated. Either the arithmetic unit must retain the result of the Subtract instruction, or the result must be buffered in atemporary register until preceding instructions have written their results. If an exceptionoccurs during the execution of an instruction, all subsequent instructions and their buffered results are discarded. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

15. Out-of-Order Execution It is easier to provide precise exceptions in the case of external interrupts. Whenan external interrupt is received, the dispatch unit stops reading new instructions from theinstruction queue, and the instructions remaining in the queue are discarded. All instructionswhose execution is pending continue to completion. At this point, the processor and all itsregisters are in a consistent state, and interrupt processing can begin. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

16. Execution Completion To improve performance, an execution unit should be allowed to execute any instructionswhose operands are ready in its reservation station. This may lead to out-of-order executionof instructions. However, instructions must be completed in program order to allowprecise exceptions. These seemingly conflicting requirements can be resolved if executionis allowed to proceed out of order, but the results are written into temporary registers. Thecontents of these registers are later transferred to the permanent registers in correct programorder. This last step is often called the commitment step, because the effect of an instructioncannot be reversed after that point. If an instruction causes an exception, the results of anysubsequent instructions that have been executed would still be in temporary registers andcan be safely discarded. Results that would normally be written to memory would also bebuffered temporarily, and they can be safely discarded as well. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

17. Execution Completion To improve performance, an execution unit should be allowed to execute any instructionswhose operands are ready in its reservation station. This may lead to out-of-order executionof instructions. However, instructions must be completed in program order to allowprecise exceptions. These seemingly conflicting requirements can be resolved if executionis allowed to proceed out of order, but the results are written into temporary registers. Thecontents of these registers are later transferred to the permanent registers in correct programorder. This last step is often called the commitment step, because the effect of an instructioncannot be reversed after that point. If an instruction causes an exception, the results of anysubsequent instructions that have been executed would still be in temporary registers andcan be safely discarded. Results that would normally be written to memory would also bebuffered temporarily, and they can be safely discarded as well. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

18. Execution Completion Whenan instruction reaches the head of this queue and the execution of that instruction has beencompleted, the corresponding results are transferred from the temporary registers to thepermanent registers and the instruction is removed from the queue. All resources that wereassigned to the instruction, including the temporary registers, are released. The instructionis said to have been retired at this point. Because an instruction is retired only when it isat the head of the queue, all instructions that were dispatched before it must also have beenretired. Hence, instructions may complete execution out of order, but they are retired inprogram order. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

19. Dispatch Operation We now return to the dispatch operation. When dispatching decisions are made, the dispatchunit must ensure that all the resources needed for the execution of an instruction are available.For example, since the results of an instruction may have to be written in a temporaryregister, there should be one available, and it is reserved for use by that instruction as a partof the dispatch operation. There must be space available in the reservation station of anappropriate execution unit. Finally, a location in the reorder buffer for later commitmentof results must also be available for the instruction. When all the resources needed areassigned, the instruction is dispatched. Should instructions be dispatched out of order? For example, the dispatch of the Loadinstruction in may be delayed because there is no space in the reservation stationof the Load/Store unit as a result of a cache miss in a previously dispatched instruction.Should the Subtract instruction be dispatched instead? In principle this is possible, providedthat all the resources needed by the Load instruction, including a place in the reorder buffer,are reserved for it. This is essential to ensure that all instructions are ultimately retired inthe correct order and that no deadlocks occur. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

20. Dispatch Operation A deadlock is a situation that can arise when two units, A and B, use a shared resource. Suppose that unit B cannot complete its operation until unit A completes its operation. At the same time, unit B has been assigned a resource that unit A needs. If this happens, neither unit can complete its operation. Unit A is waiting for the resource it needs, which is being held by unit B. At the same time, unit B is waiting for unit A to finish before it can complete its operation and release that resource. To prevent deadlocks, the dispatch unit must take many factors into account. Hence, issuing instructions out of order is likely to increase the complexity of the dispatch unit significantly. It may also mean that more time is required to make dispatching decisions. Dispatching instructions in order avoids this complexity. In this case, the program order of instructions is enforced at the time instructions are dispatched and again at the time they are retired. Between these two events, the execution of several instructions across multiple execution units can proceed out of order, subject only to interdependencies among them. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

21. Dispatch Operation A final comment on superscalar processors concerns the number of execution units. The processor analyzed has one arithmetic unit and one Load/Store unit. For higher performance, modern superscalar processors often have two arithmetic units for integer operations, as well as a separate arithmetic unit for floating-point operations. The floating-point unit has its own register file. Many processors also include a vector unit for integer or floating-point arithmetic, which typically performs two to eight operations in parallel. Such a unit may also have a dedicated register file. A single Load/Store unit typically supports all memory accesses to or from the register files for integer, floating-point, or vector units. To keep many execution units busy, modern processors may fetch four or more instructions at the same time to place at the tail of the instruction queue, and similarly four or more instructions may be dispatched to the execution units from the head of the instruction queue. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

22. Concluding Remarks Two important features for performance enhancement have been introduced, pipelining and multiple-issue. Pipelining enables processors to have instruction throughput approaching one instruction per clock cycle. Multiple-issue combined with pipelining makes possible superscalar operation, with instruction throughput of several instructions per clock cycle. The potential gain in performance can only be realized by careful attention to three aspects: • The instruction set of the processor • The design of the pipeline hardware • The design of the associated compiler It is important to appreciate that there are strong interactions among all three aspects. High performance is critically dependent on the extent to which these interactions are taken into account in the design of a processor. Instruction sets that are particularly well-suited for pipelined execution are key features of modern processors. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

23. Input/Output Organization

24. Bus Structure Asimple structure that implements the interconnectionnetwork isshown. Only one source/destination pair of units can use this bus to transferdata at any one time. A single-bus structure. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

25. Bus Structure I/O interface for an input device. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

26. Bus Structure The bus consists of three sets of lines used to carry address, data, and control signals.I/O device interfaces are connected to these lines, as shown, for an input device. Each I/O device is assigned a unique set of addresses for the registers in its interface. When the processor places a particular address on the address lines, it is examined by the address decoders of all devices on the bus. The device that recognizes this address responds to the commands issued on the control lines. The processor uses the control lines to request either a Read or a Writeoperation, and the requested data are transferred over the data lines. When I/O devices and the memory share the same address space, the arrangement is called memory-mapped I/O, as described before. Any machine instruction that can access memory can be used to transfer data to or from an I/O device. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

27. A bus requires a set of rules, often called a bus protocol, that govern how the bus is used by various devices. The bus protocol determines when a device may place information on the bus, when it may load the data on the bus into one of its registers, and so on. These rules are implemented by control signals that indicate what and when actions are to be taken. One control line, usually labeled R/W, specifies whether a Read or a Writeoperation is to be performed. As the label suggests, it specifies Read when set to 1and Writewhen set to 0. When several data sizes are possible, such as byte, half word, or word, the required size is indicated by other control lines. The bus control lines also carry timing information. They specify the times at which the processor and the I/O devices may place data on or receive data from the data lines. A variety of schemes have been devised for the timing of data transfers over a bus. These can be broadly classified as either synchronous or asynchronous schemes. In any data transfer operation, one device plays the role of a master. This is the device that initiates data transfers by issuing Read or Write commands on the bus. Normally, the processor acts as the master, but other devices may also become masters as we will see later. The device addressed by the master is referred to as a slave. Bus Operation CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

28. Synchronous Bus Timing of an input transfer on a synchronous bus. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

29. Synchronous Bus On a synchronous bus, all devices derive timing information from a control line called the bus clock, shown at the top. The signal on this line has two phases: a high level followed by a low level. The two phases constitute a clock cycle. The first half of the cycle between the low-to-high and high-to-low transitions is often referred to as a clock pulse. The address and data lines in figure are shown as if they are carrying both high and low signal levels at the same time. This is a common convention for indicating that some lines are high and some low, depending on the particular address or data values being transmitted. The crossing points indicate the times at which these patterns change. A signal line at a level half-way between the low and high signal levels indicates periods during which the signal is unreliable, and must be ignored by all devices. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

30. Synchronous Bus Let us consider the sequence of signal events during an input (Read) operation. At time t0, the master places the device address on the address lines and sends a command on the control lines indicating a Read operation. The command may also specify the length of the operand to be read. Information travels over the bus at a speed determined by its physical and electrical characteristics. The clock pulse width, t1 − t0, must be longer than the maximum propagation delay over the bus. Also, it must be long enough to allow all devices to decode the address and control signals, so that the addressed device (the slave) can respond at time t1 by placing the requested input data on the data lines. At the end of the clock cycle, at time t2, the master loads the data on the data lines into one of its registers. To be loaded correctly into a register, data must be available for a period greater than the setup time of the register. Hence, the period t2 − t1must be greater than the maximum propagation time on the bus plus the setup time of the master’s register. A similar procedure is followed for a Write operation. The master places the output data on the data lines when it transmits the address and command information. At time t2, the addressed device loads the data into its data register. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

31. Synchronous Bus A detailed timing diagram for the input transfer CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

32. Synchronous Bus A more realistic picture of what actually happens. It shows two views of each signal, except the clock. Because signals take time to travel from one device to another, a given signal transition is seen by different devices at different times. The top view shows the signals as seen by the master and the bottom view as seen by the slave. We assume that the clock changes are seen at the same time by all devices connected to the bus. System designers spend considerable effort to ensure that the clock signal satisfies this requirement. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

33. Synchronous Bus Multiple-Cycle Data Transfer The scheme described above results in a simple design for the device interface. However, it has some limitations. Because a transfer has to be completed within one clock cycle, the clock period, t2 − t0, must be chosen to accommodate the longest delays on the bus and the slowest device interface. This forces all devices to operate at the speed of the slowest device. Also, the processor has no way of determining whether the addressed device has actually responded. At t2, it simply assumes that the input data are available on the data lines in a Read operation, or that the output data have been received by the I/O device in a Write operation. If, because of a malfunction, a device does not operate correctly, the error will not be detected. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

34. Synchronous Bus An input transfer using multiple clock cycles. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV

35. Synchronous Bus To overcome these limitations, most buses incorporate control signals that represent a response from the device. These signals inform the master that the slave has recognized its address and that it is ready to participate in a data transfer operation. They also make it possible to adjust the duration of the data transfer period to match the response speeds of different devices. This is often accomplished by allowing a complete data transfer operation to span several clock cycles. Then, the number of clock cycles involved can vary from one device to another. CENG 222 - Spring 2012-2013 Dr. Yuriy ALYEKSYEYENKOV