1 / 29

Beam Secondary Shower Acquisition System: Igloo2 GBT Implementation tests at 5Gbps

Beam Secondary Shower Acquisition System: Igloo2 GBT Implementation tests at 5Gbps. Student Meeting Jose Luis Sirvent PhD. Student 09/06/2014. 0. The Set-up 0.1 Material used and system status:.

loc
Télécharger la présentation

Beam Secondary Shower Acquisition System: Igloo2 GBT Implementation tests at 5Gbps

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Beam Secondary Shower Acquisition System:Igloo2 GBT Implementation tests at 5Gbps Student Meeting Jose Luis Sirvent PhD. Student 09/06/2014

  2. 0. The Set-up0.1 Material used and system status: • The modified version of GBT_FPGA for Igloo2 was finally implemented correctly @ 2.5Gbps & 5Gbps. • Some constraints have to be optimized for correct operation of 5Gbps after every compilation. • The GBT Firmware was finally organized & commented properly, including an error counter and Boards auto-detection. • The Console Application was modified and re-structured to include the error counter. • Tullio already offered a very valuable information from his measurements. • Needed to verify timing details to check if we can recover the LHC clock on the front-end system. (Study the recovered Clk phase, link latency and ref frequency tolerance TX  RX).

  3. 0.2 GBT-FPGA Overview in Igloo2:(Clock Management as in Virtex 6) GBT_BANK (Very simplified view) SERDES_INIT_MASTER APB_BUS Data_In (84 bits) @40Mhz Tx_Word (20 bits) @ 240MHz Tx_CLK (240MHz) GBT_TX Scrambler Encoder Gearbox GBT_MGT SERDES_0 Vendor Specific IP TX_Data_P (4.8Gbps) TX_Data_N (4.8Gbps) TX_Word_CLK (240MHz) TX_Frame_CLK (40MHz) RefCLK1_P (120MHz) RefCLK1_N (120MHz) Rx_CLK (240MHz) RX_Data_P (4.8Gbps) RX_Data_N (4.8Gbps) Rx_Word (20 bits) @ 240 MHz Data_Out (84 bits) @ 40Mhz GBT_RX Gearbox Decoder Descrambler RX_Word_CLK (240MHz) RX_Frame_CLK (40MHz) RX_PLL TX_PLL

  4. 0.3 Clock relationships:Understanding some details of the clock recovery

  5. 1. Clock Recovery (Phase)1.1 Word_CLK (SERDES TX_CLK & RX_CLK 250Mhz @ 5Gbps) • Yellow EPCS_TX_CLK (Board 1) • Blue  EPCS_RX_CLK (Board 2) • Test performed resetting every time • the boards: • Observed random phase difference • There are 20 possible lock steps • Possible to know RX_WORD_CLK phase based on RX_BITSLIP_NUMBER. • Study the possibility to tune in the SERDES the EPCS_RX_CLK internally as done in Virtex 6 • Similar results obtained when doing manual RX Reset and CDR Lock steps.

  6. 1. Clock Recovery (Phase)3.2 Checking that the Word_CLK’s phase difference depends of RX_BITSLIP • The test were done resetting the boards: • As spected there is a high correlation between RX_BITSLIP_NUMBER and the phase difference between TX_WORD_CLK (Board #1) and RX_WORD_CLK (Board #2) • The Phase 0 - 2 pi corresponds to 0 - 19 bit displacement. • The measurements were taken using the scope cursors, so there are some measurement error, but the tendency is clear.

  7. 1. Clock Recovery (Phase)1.2 Frame_CLK (41.6Mhz @ 5Gbps) • Yellow FRAME_TX_CLK (Board 1) • Blue  FRAME_RX_CLK (Board 2) • These are the clocks after the TX & RX • PLLs, based on EPCS_ TX & RX _CLK • The test were done resetting the boards: • As previously the phase variation is random. • This would be our CLK for acquisition electronics. • The PLL can lock on any of the 6 rising edges of RX_WORD_CLK. • Possible to optimize phase variations adjusting the PLL CLK phase on based on RX_HEADER_FLAG. • To be seen if only adjusting Frame_Clkthe link provides deterministic latency.

  8. 2. Link latency (5Gbps)2.1 Observing the Match_Flag signal (Memory based Gearbox) • Yellow TX_MATCH_FLAG (Board 1) • Blue  RX_MATCH_FLAG (Board 2) • These are flags that goes to 1 when certain frame is detected • The test were done resetting the boards: • As expected the latency is not deterministic, with a pseudorandom behaviour. • The Phase differences on the Word & Frame Clocks are the responsible of this delay variation. • If we are able to align the clocks properly the link delay would be “more stable”.

  9. 2. Link latency (5Gbps)2.1 Observing the Match_Flagsignal (Register based Gearbox) • Yellow TX_MATCH_FLAG (Board 1) • Blue  RX_MATCH_FLAG (Board 2) • These are flags that goes to 1 when certain frame is detected • The test were done resetting the boards: • As expected the latency is not deterministic, with a pseudorandom behaviour. • The Phase differences on the Word & Frame Clocks are the responsible of this delay variation. • If we are able to align the clocks properly the link delay would be “more stable”.

  10. 3. RX_Header Flag (5Gbps)3.1 Comparing phase respect Board1 TX_Word_CLK(Register based Gearbox) • Yellow EPCS_TX_CLK (Board 1) • Blue  RX_HEADER_FLAG (Board 2) • RX_HEADER_FLAG goes to 1 when the word received contains the GBT Header. • The test were done resetting the boards: • As expected the latency is not deterministic, with a pseudorandom behaviour. • The Phase difference on the Word clocks is the responsible of this delay variation (20 possible phases). • If we are able to align the word clocks properly the phase would be constant.

  11. 3. RX_Header Flag (5Gbps)3.2 Checking that the RX_HEADER phase depends of RX_BITSLIP • The test were done resetting the boards: • As done previously with the WORD_CLK’s. • RX_HEADER_CLK status it’s updated on the RX_WORD_CLK rising edges. • RX_HEADER_CLK is used as a sync signal in Virtex 6 version to align RX_FRAME_CLK (in that example the RX_WORD_CLK has been already tunned based on BITSLIP_Number) • Measurements taken more carefully, clear tendency.

  12. 3. Ref. Frequency tolerance3.0 How to transmit LHC Clock to the Front-End Electronics Surface FPGA Tunnel FPGA SERDES Ref_Clock 120.237Mhz LHC Clock 40.079Mhz Local SERDES REF_CLK 120Mhz TX_PLL x3 Optial Link GBT @ 4.809Gbps TX_SERDES RX_SERDES LHC Clock 40.079Mhz (RX_Frame_Clock) RX_PLL /6 SERDES EPCS_RX_CLK 240.474Mhz (Word_Clock) • It will be possible to work with such difference on the REF_CLKs? • Do we recover the correct EPCS_RX_CLK when the REF_CLK are different? • Which are the difference limits on REF_CLKs? • If the TX REF_CLK varies during transmission the link will suffer errors?

  13. 3. Ref. Frequency tolerance3.1 Link performance over Ref_Clk differences on TX & RX • LHC operations require a clock tolerance of 50 ppm (the LHC clock varies during operation). • We’ll play with Ref_CLK’s around 125Mhz (the local osc of the Dev.Kit) • A safe region was found for GBT @ 2.5 & 5Gbps • Board 1  125.0 Mhz • Board 2  125.0 ± 0.5 Mhz • Difference  0.8%  8000 ppm >> 50 ppm (we met the specs) • Tullio’s measures • EPCS_DEMO example @ 3.125Gbps • Difference  2.3Mhz  1.8%  18400ppm Board 2 Variable Ref_CLK 123.5 – 126.5 Mhz Board 1 Local Ref_CLK 125Mhz

  14. 3. Ref. Frequency tolerance3.1 Link performance over Ref_Clk differences on TX & RX • LHC operations require a clock tolerance of 50 ppm (the LHC clock varies during operation). • We’ll play with Ref_CLK’s around 125Mhz (the local osc of the Dev.Kit) • A safe region was found for GBT @ 2.5 & 5Gbps • Board 1  125.0 Mhz • Board 2  125.0 ± 0.5 Mhz • Difference  0.8%  8000 ppm >> 50 ppm (we met the specs) • Tullio’s measures • EPCS_DEMO example @ 3.125Gbps • Difference  2.3Mhz  1.8%  18400ppm Board 2 Variable Ref_CLK 123.5 – 126.5 Mhz Board 1 Local Ref_CLK 125Mhz SAME RESULTS FOR 2.5 AND 5Gbps

  15. 4. Clocks Frequency Stability4.1 Freq Histogram of the TX & RX Word CLKs (250Mhz) EPCS_TX_CLK (Board 1) • Clocks Stability: • For Both clocks (TX_WORD_CLK & RX_WORD_CLK), independently of their Ref_CLK source it was observed certain frequency dispersion. • EPCS_TX_CLK  STDEV ~ 6.1MHz (24400ppm) • Frame_TX_CLK  STDEV ~ 41kHz (985ppm) • EPCS_RX_CLK  STDEV ~ 3.7MHz (14800ppm) • Frame_RX_CLK STDEV ~ 45kHz (1081ppm) • Possible Reasons: • Not really well understood • Part of the SERDES Specifications? • More care needed on routing/ Implementation? • Some questions to think about: • Impact on system performance? • Could we tolerate this variations? EPCS_RX_CLK (Board 2) EPCS_TX_CLK (Board 2) EPCS_RX_CLK (Board 1)

  16. 4. Clocks Frequency Stability4.2 Freq Histogram of the TX & RX Frame CLKs ( After PLLs 41.6Mhz) Frame_TX_CLK (Board 1) • Clocks Stability: • For Both clocks (TX_WORD_CLK & RX_WORD_CLK), independently of their Ref_CLK source it was observed certain frequency dispersion. • EPCS_TX_CLK  STDEV ~ 6.1MHz (24400ppm) • Frame_TX_CLK  STDEV ~ 41kHz (985ppm) • EPCS_RX_CLK  STDEV ~ 3.7MHz (14800ppm) • Frame_RX_CLK STDEV ~ 45kHz (1081ppm) • Possible Reasons: • Not really well understood • Part of the SERDES Specifications? • More care needed on routing/ Implementation? • Some questions to think about: • Impact on system performance? • Could we tolerate this variations? Frame_RX_CLK (Board 2) Frame_TX_CLK(Board 2) Frame_RX_CLK (Board 1)

  17. 5.Compilation Resources5.1 Comparison using Memory & Register Based Gearbox Register Gearbox Memory Gearbox • The GBT_FPGA code was compiled successfully on Igloo2 with the following resource usage: • No Clock phase alignment has been yet performed for the on Igloo2 version. • Implemented using the two gearbox possibilities for clock domain crossing available on Xilinx V6 example. • Memory Gearbox : Two port memories • Register Gearbox : Through registers. • Still TODO  Aligh the FRAME_CLK • (We’ll use the information of BITSLIPNUMBER for and RX_HEADER_FLAG for this)

  18. 6.Constraining the Design6.1 On Synplify Pro & on Smart-Time editor. Constraint Clocks on Synplify Constraint Clocks on Smart-Time  Allow multicycle path on Smart-Time  (Between TX_FIFO & SERDES_TX)

  19. 7. Synchronizing SERDES, GBT_TX & GBT_TX7.1 Maintaining CLK/DATA phase relationships • Buffered EPC_TX_CLK & EPCS_RX_CLK: • Helps to reduce the CLK Skew through global resources (and therefore reach the clkcontraints for GBT_TX & GBT_RX) • However the buffered VS non buffered versions have different phases. • Synchronization CLK / TX & RX Data needed (done through Registers & Delay lines)  File: Igloo2_gtx_std.vhdl Adjustable Delay

  20. 7. Synchronizing SERDES, GBT_TX & GBT_TX7.1 Maintaining CLK/DATA phase relationships • Buffered EPC_TX_CLK & EPCS_RX_CLK: • Helps to reduce the CLK Skew through global resources (and therefore reach the clkcontraints for GBT_TX & GBT_RX) • However the buffered VS non buffered versions have different phases. • Synchronization CLK / TX & RX Data needed (done through Registers & Delay lines)  File: Igloo2_gtx_std.vhdl Adjustable Delay Adjustable Delay MultiCycle path (2)

  21. 7. Synchronizing SERDES, GBT_TX & GBT_TX7.1 Maintaining CLK/DATA phase relationships • Buffered EPC_TX_CLK & EPCS_RX_CLK: • Helps to reduce the CLK Skew through global resources (and therefore reach the clkcontraints for GBT_TX & GBT_RX) • However the buffered VS non buffered versions have different phases. • Synchronization CLK / TX & RX Data needed (done through Registers & Delay lines)  File: Igloo2_gtx_std.vhdl Adjustable Delay Adjustable Delay MultiCycle path (2) EPCS_TX_CLK EPCS_TX_CLK_BUFF GBT_TX_DATA GBT_TX_DATA_DELAYED

  22. 7. Synchronizing SERDES, GBT_TX & GBT_TX7.1 Maintaining CLK/DATA phase relationships • Buffered EPC_TX_CLK & EPCS_RX_CLK: • Helps to reduce the CLK Skew through global resources (and therefore reach the clkcontraints for GBT_TX & GBT_RX) • However the buffered VS non buffered versions have different phases. • Synchronization CLK / TX & RX Data needed (done through Registers & Delay lines)  File: Igloo2_gtx_std.vhdl Adjustable Delay Adjustable Delay MultiCycle path (2) MultiCycle path (2) EPCS_RX_CLK EPCS_RX_CLK_BUFF EPCS_RX_DATA EPCS_RX_DATA_DELAYED

  23. 8.Post-Layout Implementation Report8.1 Reaching constraints • We have to play a bit with the TX & RX delays in order to respect the Max & Min Delay analysis on Smart-Time. • NOTE!!: Every time the design is changed and a new compilation (and place&route) is performed the reports change, and under certain circumstances the constraints are not met, therefore more efforts on design constraining have to be done (or we leave this as it is now and we’ll take care about it later when the final design is done). • Currently Experimenting some difficulties to reach the constraints… new ideas are more than welcome!! • Maybe bigger Igloo2 FPGAS with more resources can reach easier the specifications (or at least with bigger margin). • In spite of the limitations on WORD_CLK_RX & WORD_CLK_TX at 241Mhz / 246Mhz, the test were done at 250Mhz with satisfactory result.

  24. 9. The beauty is inside: GBT_On_Igloo2 9.1 All the implementation PLLs

  25. 9. The beauty is inside: GBT_On_Igloo29.2 GBT_BANK

  26. 9. The beauty is inside: GBT_On_Igloo2 9.3 GBT_BANK/GBT_TX

  27. 9. The beauty is inside: GBT_On_Igloo2 9.4 GBT_BANK/GBT_RX

  28. 9. The beauty is inside: GBT_On_Igloo29.5 GBT_BANK/MGT Global Clock Network

  29. 10. Code Availability10.1 First release on Dropbox/SVN • First release STD version available for download (09/06/2014) : • In DropBox: • https://www.dropbox.com/sh/5sjvdkp9wwykpz4/AADrIrhM5BKj_zcraAYGxnURa • In SVN (Use Tortoise or other SVN client): • https://svn.cern.ch/reps/be-bi-bl/electronics/bwsdev/studies/BWS_pCVD Diamond detector Readout Electronics/GBT_On_Igloo2/Firmware/GBT_FPGA_Igloo2/STD • What is provided: • Libero 11.3 Project with GBT_on_Igloo2 code: (2014_06_09_GBT_On_Igloo2_M2GL_EVAL_KIT.rar) • Features: GBT Protocol (STD Version) on Igloo2 with UART communication through USB port. Constraints are not always met, so care must be taken when new changes are performed analysing timing reports. • All the necessary VHDL files are in : GBT_On_Igloo2_M2GL_EVAL_KIT\hdl • Programming file (stp) available in : GBT_On_Igloo2_M2GL_EVAL_KIT\designer\GBT_On_Igloo2_M2GL_EVAL_KIT\export • In case of trouble, just let me know!! jsirvent@cern.ch there are many things to improve. • Console Application UART_APP_V3.0: (2014_06_09_UART_APP_V3.0.rar) • Features: Controls the workflow of the GBT implementation on Igloo2 and checks different signal values and parameters of the link (RX_BITSLIP_NUMBER, Error number…), Boards Auto-Detection. • Microsoft Visual Studio 2008 Project: UART_APP_V3.0\UART_APP.sln • Executable File: UART_APP_V3.0\Release\UART_APP.exe • Readme File: (2014_06_09_Readme.pdf) • Features: Short guide to implement the design on the Dev.Kit and run the application

More Related