FPGA Architecture, timing, Software

1. FPGA Architecture, timing, Software Mose Wahlstrom Lattice Research & Development Team May 16, 2014

2. Overview • Mose Wahlstrom, BSEE OSU 1992 • At Lattice for last 22 years • Excited to enhance partnership between Lattice and OSU • Will continue to give to OSU (hardware, software, time) • Will continue to hire (interns and permanent positions) • Will entertain other guest lectures • Here to focus on FPGA architecture, software, and timing • Not here to pitch Lattice or recruit. (That comes later!) • Ask questions • (And yell if I use acronyms or unknown terms) • This will go fast and I don’t expect 100% of it to stick. • Just a general background you can build on.

3. topics • Field Programmable Gate Array (FPGA) Architecture (~25 min) • Top level block diagrams • Logic building blocks • Signal routing • Timing and Power (~15 min) • Clocking and control signals • Typical path • Operating frequency and timing constraints • Static timing analysis (setup/hold and clock-to-out) • Power consumption • Software Flow (~10 min) • Design capture in Verilog, IPexpress • Synthesis into ‘standard’ cells • Cell placement and routing in the FPGA • Bitstream generation and device configuration

4. Block diagram • FPGAs: An array of PLCs IO Ring ProgrammableLogicCell (PLC) • Field Programmable Gate Arrays are built from a grid of tiled, simple cells. • Cell counts from ~256 to 100k • PLCs align by abutment; can build any reasonably sized array. • Includes both ROUTING and LOGIC • ROUTING tracks cross boundaries • LOGIC is self-contained in the PLC • The IO ring contains the Input/Outputs • The IO ring also contains other dedicated ‘value added’ logic LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC LOGIC NOTE: PLC drawn to approximate scale • ROUTING comprises ~70% of area • LOGIC comprises ~30% of area

5. MACHXO2 block diagram (XO2-1200) sysIO Buffers Support LVCMOS/LVTTL, LVDS Outputs. I/O Logic Supports 7:1 Output Gearing On-chip Flash Memory Provides Instant-on, High Security & Single Chip Solution Embedded Function Block Hardened SPI, I2C,Timer/Counter sysCLOCK PLLs Frequency Synthesis & Clock Alignment sysIO Buffers Support LVCMOS/LVTTL, and DDR Memory Interfaces sysMEM Block RAM 9Kbit Dual Port Programmable Function Units (PFUs) with RAM sysIO Buffers Support LVCMOS/LVTTL, LVDS inputs and PCI. I/O Logic Supports 7:1 Input Gearing • ‘Value added’ features in and around the core User Flash Memory Flexible Routing Optimized for Speed, Low-Cost and Routability

6. Brief Overview of the PLC The Programmable Logic Cell (PLC) is the fundamental building block of the FPGA Fabric. The PLC consists of 2 components: PFU – Programmable Function Unit (Very simple logic!) Programmable Routing Block or Big Switch Box (BSB) (Lots of Muxes) The routing wires are an integral part of the BSB, but pass over the PLC from tile to tile.

7. PFU – The look-up-table (lut) Building the PFU from the inside out… Nearly all FPGAs are based on a Look-Up-Table plus Register. Most are a LUT4. Aka LUT4+REG. A 4-input LUT is just a 16-bit ROM, with 4 ‘address’ bits (ABCD) and a ‘data’ bit (F). By programming the ROM, any 4 input logic functions can be formed. Or it can be a simple ROM. A B q0 C q1 q2 LUT Memory q3 q4 D q5 q6 q7 F q8 q9 q10 q11 q12 (‘q’ values are programmable SRAM memory bits that are determined through the design synthesis process) q13 q14 q15

8. PFU – LUT+register The Register (Flip-Flop) A custom configurable register with clock (CLK) and Data (D) input and a Q output. The typical register also contains other options such as: Clock Enable (CE), Set/Reset (LSR), Latch mode, and selectable input polarities. The LUT and register can be used independently, or the output of the LUT can feed the D input of the register. Early FPGAs were just LUTs and registers! Nearly any logic can be built from just these two blocks. All other PFU logic supports logic expansion and special enhancements to the LUT+REG. F LUT4 REG A B C D LUT inputs F DQCE Q FROMROUTING TO ROUTING SR ‘M’ input Clock Enable Set/Reset Clock

9. PFU – enhancements Over time, the PFU has evolved to include other logic to increase performance and logic density. Wider LUTs In order to perform wider logic functions with minimal performance hit (no general routing), special muxes are added to allow wider LUT functions. The OFX signal is a muxed output from a pair of LUTs. It is controlled by the Miscellaneous (M) input. This creates a LUT5 from inputs ABCDM. Five inputs, 32 possible logical combinations. Additional OFX muxes support LUT6, LUT7, and LUT8. A LUT7 consumes 8 LUT4s (128 memory bits) . A B C D LUT inputs F OFX (LUT5) LUT0 A B C D F ( . . . ) LUT1 ‘M’ input

10. PFU – enhancements (cont) RAM mode RAM mode can be implemented by adding some ‘write’ logic to the LUT. This allows the LUT to implement a small 16 bit RAM. The ‘read’ just functions similar to ROM/LUT mode. Implementing the write operation requires ‘borrowed’ signals and logic. Ripple mode Ripple Mode is implemented by adding some additional logic and re-purposing the LUT bits to function as a carry-look-ahead adder. Ripple mode supports configurable options for implementing adders, subtractors, and comparators. Now we can put it all together to construct a typical PFU. Wide LUT, RAM, and Ripple mode don’t really have any practical uses with a single LUT. Only when LUTs are grouped together can we build wide logic functions and multi-bit adders and RAMs. Thus the minimal building blocks need to be more than just a single LUT+REG.

11. PFU – Building ‘Slices’ • Pairs of LUT+REG are grouped together with extra RAM/Ripple logic to form SLICEs. • Each Slice consists of: • Two 4-Input LUTs • Two Registers • Arithmetic Logic circuits • Circuitry to support simple RAM mode • Slice Inputs: • LUT Inputs: A, B, C, D • Multi-Purpose Inputs: M • Fast Carry Input: FCI • Register Control Inputs: CLK, CE, LSR • Slice Outputs: • LUT Outputs: F • Register Outputs: Q • Wide Function Outputs: OFX • Fast Carry Output: FCO

12. PFU – Building the PLC The ‘SLICE’ Multiple slices are bolted together to form the PLC. Ripple and RAM mode are superset slice options. RAM mode requires ‘stealing’ logic and ports from adjacent slices. All slices can be used as a LUT and/or a REG. A Typical PFU contains 4 slices. This would contain/support the following: 8 LUT4s or up to a single LUT7, up to a 16x8bit pseudo dual port RAM, an 8 bit register or shift register, a full 8bit adder/subtractor/comparator, or other combinations.

13. PLC – TOP Level View A2 B2 C2 D2 M2 A0 B0 C0 D0 M0 A4 B4 C4 D4 M4 A6 B6 C6 D6 M6 A1 B1 C1 D1 M1 A5 B5 C5 D5 M5 A3 B3 C3 D3 M3 A7 B7 C7 D7 M7 CLK0/1 LSR0/1 CE2 CLK0/1 LSR0/1 CE1 CLK0/1 LSR0/1 CE0 CLK0/1 LSR0/1 CE3 Arithmetic Logic Arithmetic Logic Arithmetic Logic Arithmetic Logic FF/ Latch FF/ Latch FF/ Latch FF/ Latch FF/ Latch FF/ Latch FF/ Latch FF/ Latch LUT4 LUT4 LUT4 LUT4 LUT4 LUT4 LUT4 LUT4 FCI FCO Slice 0 Slice 3 Slice 2 Slice 1 F0 Q0 OFX0 F2 Q2 OFX2 F6 Q6 OFX6 F4 Q4 OFX4 F7 Q7 OFX7 F3 Q3 OFX3 F5 Q5 OFX5 F1 Q1 OFX1

14. PFU – Slice Level View w/control FCO FXB OFX1 FXA LSR M1 Q1 A1 D B1 FS1/GEN1 LUT4 F/SUM1 C1 FF_1/ Latch_1 D1 Arithmetic & Carry Logic F1 F0 F/SUM0 A0 FS0/GEN0 B0 LUT4 Q0 C0 D D0 FF_0/ Latch_0 M0 1 0 0 0 1 LSR 1 1 0 OFX0 CE 0 0 1 1 LSR0 0 1 1 0 LSR LSR1 CLK_DEL CLK to FF_1 CLK0 CLK to FF_0 CLK1 FCI • The configuration of the Slice is controlled by programmable SRAM bits. • The bits are used to: • Set LUT ‘q’ bits • Set Slice to RAM mode • Set Slice to Ripple mode • Set Register modes • Regor latch • Set or Reset • Set Slice options • LSR/CE/CLK polarities • Clock and LSR selection • Wide LUT modes

15. Additional pLC detail • Wide LUTs revisited • The OFX muxes and the FX/OFX IO signals are dedicated to wider LUT functions. OFX0 is always a LUT5. OFX1 performs a different function in each slice. Two of the slices use the OFX1 mux to generate LUT6s from a pair of LUT5s. One slice can then use OFX1 to generate a LUT7 from the pair of LUT6s. The final OFX1 can be used to generate a LUT8 from its own LUT7 and that of a neighboring PFU. • The neighboring LUT7 output/input is one of the few special directly connected signals that span PFUs. • RAM mode revisited • Implementing RAM mode requires more signals than are present in a single slice. At a minimum, a Write Enable (WRE) and a Data Input (DI) are needed. However in order to support Pseudo Dual Port (PDP) mode, an additional Write Address (WAD) bus is needed as well as a Write Clock (WCK) for the write port. • In a typical implementation, these signals come from one of the other slices. For example, slice 0 and 1 may be used to implement a 16x4 PDP RAM. Slice 2 is ‘burned’ to generate the control signals, which are sent to slice 0/1. And slice 3 doesn’t support RAM mode. • Thus there are special RAM mode signals sent between slices. These signals are dedicated for RAM mode and do not leave the PFU.

16. Additional pfu detail (cont.) • Ripple mode revisited • Ripple mode repurposes a pair of LUTs to implement a full 2-bit Carry Look-Ahead (CLA) adder. The LUT is programmed as a ROM and slightly modified to produce Propagate and Generate signals from the (4) inputs and the carry input signal. • By adding dedicated Fast Carry Input/Output (FCI/FCO) signals, performance is greatly improved when compared to arithmetic functions using LUT logic and general routing resources. • Ripple mode is not a full CLA, but rather a chain of 2-bit CLAs with a dedicated, rippling fast carry chain. • Every slice has a FCI and FCO port that connect adjoining slices. At the PFU boundary, the FCI and FCO connect between neighboring PFUs (typically L to R).

17. Programmable Routing Block ROUTING: (The other three-quarters of the PLC) The general purpose routing is not deterministic. It is a collection of pseudo-random paths. The design software, user preferences, design congestion, and random seeding all affect the signal routing. The routing portion of the PLC can be divided into the signal wires and the switch boxes that feed them. The wires carry signals from one PLC to another. The switch boxes are the programmable source connections. The wires are the roads, the switch boxes are the intersections. Once again, the programmable connections are controlled by SRAM bits. The muxes are typically two-stage, one-hot pass-gate muxes.

18. A typical switch-box mux Routing In general, the routing wire segments are unidirectional, buffered segments that span either 2, 3, or 7 PLC blocks (Seg-1, Seg-2, Seg-6 or X1, X2, X6). The wires feed both horizontally and vertically in all four directions. Example 6:1 routing mux: I5 I4 I3 I2 I1 I0 M1 M0 M1 M2 M0 M3 M2 M4 Mux Output Driver Pre-Driver To Connect I0 to Mux Output: Turn ON progammable SRAM memory cells M0 and M3

19. Programmable Routing Block There are two distinct categories of switch boxes. Input Switch Boxes (ISBs) are muxes that feed the inputs to the PFU. Output Switch Boxes (OSBs), are muxes that feed the routing wires out of the PLC. The source for each ISB and OSB is a programmable mux. The inputs to the mux determine what subset of signal sources can drive that particular wire or PFU input. A typical routing mux contains about 20 inputs. Using these ISB and OSB mux connections, routing paths can be found to connects any PLC output to any other PLC input.

20. Programmable Routing Block The BSB contains hundreds of programmable muxes. The ISBs feed into the PFU to drive: LUT and M inputs Clock inputs CE and Local Set/Reset inputs The OSBs feed out of the PLC: OSBs drive all the segment wires (X1, X2, X6) The sources for OSB muxes are a pseudo-random selection of both PFU outputs and other routing wires. The following horribly confusing diagram illustrates a high level view of the routing structure. In general, the routing muxes are sparsely populated. Only a small fraction of the possible wires feeds each ISB or OSB. The software must search many possible options to find a route. This includes swapping LUT inputs, moving logic drivers, and duplicating logic.

21. PLC Routing (confusing diagram)

22. PLC Routing connectivity Example X2 wires: There are X2 wires that travel in all 4 directions Typically there are 8-16 in each direction At each driving end there is an OSB mux At each destination, the wires feed many potential ISB muxes X1 and X6 wires are similar: X1 wires feed neighboring PLCs X6 wires feed 6 PLCs away Similar to a system of roads: Routing is Heirarchical Expressways with limited off ramps Local highways Neighborhood streets

23. PLC Routing Connectivity Directs are local: F/Q of local PFU F for wider logic Q for Counters, State machines Others inputs are from general routing. The mux controls are set static by SRAM programming bits • Example: LUT ISB Mux connectivity • CE, LSR, CLK are similar but include global clock resources To PFU LUT Inputs (A,B,C,D) and M

24. PLC Routing Connectivity Example: X1 ‘Output’ Mux connectivity Mux controls are static SRAM (Similar for X2, X6, however not limited to just PFU outputs. X2/X6 primarily source other routing resources)

25. Global routing (CLOCK timing) In addition to the general purpose routing, there are some global control signals that feed the entire PLC array and the peripheral logic in the IO ring. The most important global signals are the clock signals. (Synchronous logic) All FPGA devices contain clock ‘trees’ to guarantee all registers receive the clock at essentially the same time to provide deterministic setup and hold times between registers. This is known as the primary clock tree, which has many branches and programmable options for clock gating, switching, and power control. All clock signal trees have special routing mux resources to choose their sources. These muxes are located in the central clock switch. The sources for the clocking resources are a mix of general purpose routing, dedicated IO pins, Phase Lock Loop (PLL) outputs, and other clock resources. This allows the implementation of complex clock systems with both internal and external clocks, multiple clock frequencies, and various clock phases. All devices have a Global Set Reset (GSR). This signal fans out to every register in the device, both in the PLC array and in the IO ring. This provides a system reset. From a user’s perspective, these are the only significant global signals. However there are lots of other global signals dedicated to programming, power control, etc.

26. Clock tree Clock trees guarantee identical clock delay to all registers. This includes PLC registers and IO ring IP blocks such as IO cells and embedded RAM. There are many parallel clock trees to support multiple clock domains. The software controls all timing PFU to PFU PFU to IO ring PFU to IP block IP block to PFU IO Ring REG CENTRAL CLOCK SWITCH REG … … Clock Input Source

27. Typical signal Path Typical Path: A 4-logic-level deep register-to-register path Path Delay = (CLK2Q + LUT4 delays + FF_Setup) + (OSB Routing + ISB) The routing delays could represent local intra-PLC delays, or they could be inter-PLC delays crossing the entire PLC array. (Not shown is the clock signal, which also has a delay.) Routing delay Routing delay Routing delay Routing delay Q D A D A Routing + ISB Routing + ISB Routing + ISB Routing + ISB FF FF LUT LUT LUT LUT PFU PFU PFU PFU PFU LUT4 delay LUT4 delay CLK2Q LUT4 delay LUT4_delay + FF Setup

28. Software timing control • All routing structures, PFU logic elements and peripheral IP blocks require specific timings in order to function correctly. There are three basic timing constructs: FMAX, routing delays, and Port Timings. • FMAX (Maximum Operating Frequency) • IP blocks, PFU modes, and clock trees all have an explicit FMAX • A Block RAM may have an FMAX of 300MHZ, beyond which read or write functional will fail to execute correctly. • An LVCMOS IO standard may have an FMAX of 250MHZ, beyond which the high and low output level would be violated. • These FMAX values are determined by simulation of the design (Lattice Spice sims), by characterization, or by ‘binning’ at final test. • The FMAX values are provided to the software to limit a users allowed operating frequency. • The sum of the routing delays between registers can also determine a ‘design specific’ maximum operating frequency. • Every signal path (route) in a user’s design has a specific delay. • There is usually a very small fraction of routes (or even a single route) that may also limit the maximum operating frequency. • Users enter their desired operating frequency and the software checks to make sure that all blocks and resources meet the FMAX constraints.

29. Software timing control (cont.) • Routing Delays and Port Timings • All synchronous blocks require specific Setup/Hold time (TSU/TH) on IN ports and they provide specific Clock To Out (TCO) on OUT ports. • These TSU/TH/TCO values are determined by simulation of the device, by characterization, or by ‘binning’ at final test. • The routing delays of each wire and mux type (X2, X6, ISB, OSB) are also simulated and characterized. • All these port and routing timings are integrated into the software so that it can determine timing closure on every signal path between register. • For the software to be effective, users must enter timing ‘constraints’. • This sets FMAX and prioritizes internal requirements (general goals). • External IO requirements must also be provided (TSU/TH/TCO). • These are determined by off-chip clock and data alignments, such as from an external memory, interface chip, or processor. • With all of the above information, the software attempts to meet all static timing requirements. It has many options such as: • Moving logic (registers) closer together to fix a Setup time violation. • Intentionally adding delay to a route to fix a Hold time violation. • Duplicating logic closer to a destination to fix a setup time violation. • Giving up.

30. FPGA Power CONSUMPTION The software can also estimate power consumption. Similar to the timing, the power data comes from many sources such as: Spice simulations, characterization, and final test ‘binning’. Power data is integrated into the software for each routing element, PFU mode, and IP block. There are two basic components of power, DC and AC power. DC power consumption: DC power is comprised of static bias currents and leakage. The leakage currents are calculated from a curve fit equation based on supply voltage (VCC), Temperature, and process variation (typical or worst-case). The software provides users the means to enter VCC and temperature as well as select the process. Every LUT, Block RAM, routing wire, etc. has different bias and leakage. There are also different calculations based on if an element is used or unused in a particular user’s design. The software will also generate statistical averages when logic values affect DC bias or leakage current (a ‘1’ and ‘0’ may be different).

31. FPGA Power CONSUMPTION (cont.) AC power consumption: For routing, only a function of: Capacitance (extracted from layout, technology, and simulation) Voltage (Provided by customer’s design) Frequency (average switching frequency of each ‘wire’) Every ‘wire’ will switch at different rates depending on the design, activity, and time slice. Activity Factor (AF) is provided by the customer as an average indicator of switching rates (0%-100%) For the PFU logic and other IP blocks The same CVF method can be used. ‘Capacitance’ values are provided in the software for each IP block. Users must enter the clock frequency for each clock that feeds the PFUs and IP blocks. Total Power: Total power is the summation of ‘used’ DC, ‘unused’ DC, and AC CVF. Every routing, PFU, and IP element is separately calculated and summed together to produce the total power for the entire FPGA. Every supply is analyzed separately.

32. Software Flow • ‘ispLEVER’ is the old name for the Diamond software. • Users capture their design, usually with Verilog and other GUI and file inputs. • Synthesis turns Verilog into ‘standard cells’ that are the fundamental building blocks of logic: • Logic equations, registers, adders, memory, etc. • This step is pretty much voodoo to me. • Lattice provides architectural info to Synopsys.

33. Software Flow (Cont.) • The logical building blocks are then ‘mapped’ to FPGA specific resources such as: • LUTs, Block RAMs, ripple adders, PLLs, etc. • All of the logic elements are then ‘placed’ into appropriate sites in the FPGA. • LUT equations are placed in Slices. • RAMs in Block RAM locations. • If the device runs out of resources, it will try to remap some of the elements, or it will ultimately fail. • All of the logic elements are then ‘placed’ into appropriate sites in the FPGA. • LUTs are placed in specific Slices, etc. • Once placed, all signals are then routed. • The software can rearrange placement if it encounters routing limitations. • Some designs will fail to route. • The final step is ‘bitstream’ generation. • This process sets all of the programmable SRAM bits to implement the logic and routing in the design • The bitstream is then loaded into the device SRAM bits to implement the user’s function. (Or into Flash Memory)

34. OTHER SW RESOURCES • I believe all of these tools are available in the free software. • EPIC (don’t know this acronym) • EPIC is a GUI tool included in the software that can be used to view, analyze and design nearly every aspect of the FPGA. • It shows every routing segment, LUT, IP block, etc. • It is very cumbersome and difficult to learn. • NCL • The software can also generate an NCL (NeoCad Listing?), which is a text version of the design files that the Map, Place, and Route tools operate on. • Once again, very cryptic and difficult to learn. But it is a text representation of an entire user’s design with all routing and logic. • Power Calculator • Can be used to estimate FPGA power consumption.