Clock Distribution for IceCube

1. http://www.phys-astro.sonoma.edu/people/students/baker/SouthPoleFoucault.htmlhttp://www.phys-astro.sonoma.edu/people/students/baker/SouthPoleFoucault.html Clock Distribution for IceCube Gerald. Przybylski Lawrence Berkeley National Laboratory, Design Review, September 16, 2005

2. History • string 18 implementation: o Rubidium module slaved to GPS Clocko fan-out port for each domcom card. • string 21 implementation: + OCXO GPS clock, Symmetricom ET-6000+ Passive fan-out to each DOM Hub (DSB Card)+ Sub-nanosecond skew and jitter demonstrated+ Simple and Reliable, but not scalable

3. MCU Requirements • Straightforward; conceptually very simple • 5ns absolute accuracy (skew and jitter), within the IceCube counting house > Across all DOM Hubs (at DOR cards) > Fixed and stable offset from Universal Time, Coordinated (UTC) > Based on Scattering Length in the ice • Distribute 10 MHz, 1Hz and TimeValueString • Free from Metastable states/events; no glitches • Measurable and Verifiable • Single driver per output-port; no shared drivers • Robustness requirements in ERD x Mainly dealing with satellite drop-out and loss of availability x Also dealing with tracking multiple GPS clocks • Phase accuracy: 0.4ns at fan-out, 0.7ns at DSB, and 1.0 ns at DOR

4. IceCube + IceTop + AMANDA NOT COVERED IN DETAIL IN THIS TALK • IceTop same as IceCube • IceCube to AMANDA *** Zeuthen/Wuppertal; Install 05/06; Holger et. al. ***Fiberoptic transmitter driven by GPS clockFiberoptic receiver drives TWR (GPS4TWR)Autonomous from the MCU Clock Fan-Out subsystem 10 MHz BNC, 1Hz BNC, IRIG-B BNC10 ns precision with respect to IceCube time • AMANDA to IceCube*** Same playersTrigger system signals over Fiberoptic link to IceCube counting houseDepends on DOM Main Board(s) on a “String 81” Justification: “We realized for Amanda that any artifial jitter below 15 ns in the MC has no influence on reco accuracy or background rejection.We then said 10 ns just for safety. Then others came and said that for nearly vertical tracks close to a string scattering may be negligible and a 5 nsec request makes sense.” -- Christian

5. Our GPS clock ET-6000: • Time to first fix: < two minutes • Outputs operational: < five minutes • Timing accuracy better than 2 µS & frequency accuracy better than 1E-8 • Full system accuracy (100nS) within one hour. 10 MHz output, 1Hz output, Time burst output GPS from US Naval Observatory clock ~2x10-15 Accuracy

6. MCU Brick-Walls ET-6000 Specifications: • 1Hz output is:positive (rising) edge on time, within ±100 nanoseconds relative to either UTC or GPS with six or more satellite averaging with 95% confidence.(± 150ns peak) • 40ns RMS accuracy (jitter not specified) • Cannot “Vote” multiple clocks; Neighboring clocks don’t track!x Tracking Algorithms in GPS clock “PLLs”x Variations in path, and multipathx Constellations and satellite switch-over

7. Unlock Behavior • Power Up:+ Sync to Satellites within an hour> Elapsed Time Format until “Tracking”- Hic-ups while searching; every 2 hours… • Potential Loss of lock:x Clock Firmware (a-la TrueTime 2000 problem)x Power Outagex Misadventure (Murphy…)x Wind/Weather damage to Antenna (speculative)x National Security outage (speculative)

8. Flywheeling/Freewheeling • TCXO Clock “continues” for hours • Optional OCXO “continues” > 1 week Aging ±5 x 10-10 per day, ±5 x 10-8 per yearPhase Noise -115 dbc/Hz @ 10Hz -94 dBc/Hz @1Hz • Optional Rubidium Oscillator clock ~100ns per day slew WRTAging < ± 5 x 10-8 over 20 years,Phase Noise -90 dBc/Hz @10Hz ~ -80 dBc/Hz @ 1 Hz Ru: Good short term stability, best hang timeOCXO: best short term stability, good hang time Based on Symmetricom ET6000 series product specifications/experiences11

9. 2004-2005 Implementation • One GPS clock, an ET6000-OCXO • Simple Passive fan-out (resistive splitters) • All 9 DOM Hubs driven; All clock BNCs used! • 0.35ns Jitter and Skew measured in situ at NPX • DOR Firmware Improvements fixed NPX GPS “glitches” • Not scalable to 90 Hubs

10. 2005-2006 Implementation • 2U chassis with 24 port fan-out (2 cards)o Modulates 1Hz signalo RJ-45 distribution cables carry 10 MHz , modulated 1Hz, & TVSo All balanced signaling • < 0.6ns skew and jitter measured on the bench • Passed MOAT: sps-ichub04, sps-ichub05, domhubjr, domhub51 • Good Noise Immunity • Stepping stone; scalable to needs of IceCube

11. Design Goals/Drivers • > 90 ports plus Spares • Meet accuracy, jitter and skew requirements • Convenience: Single distribution cable per DOM Hub • Measurability/Verifiability: Easy to confirm phase across all ports • Reliability/Robustness: Quality components. No electrolytics. • Noise Immunity: Balanced signals to DSB cards • Minimize hard connections between racks: Magnetic coupling • Modularity/Extensibility/Maintainability • Hot-Swappable Port Cards • Independent Port Drivers for each signal • Low power • Off the Shelf Components • No heroic solutions • Simplicity! (no programmable logic in clock distribution)

12. Signaling Details • Balanced 10 MHz 500mV P-P through ‘ethernet magnetics’- High common-mode immunity- Suppress EMI emission & RFI pick-up- Avoid ‘Ground Bounce’ pick-up in the counting house- Commodity components: compact, inexpensive… • Modulate* the 1Hz signal (180deg Phase Modulation)* required to pass through ethernet magnetics… • RS-422/RS-485 differential serial+12/-7V CM range 10 MHz 1Hz/1PPS Registered 1Hz U47-4 Modulated 1Hz

13. Rev 0 Fan-Out Card • 12 Port • 0.6ns port to port skew, worst pair • <200 ps jitter • Symmetry matters! • Passes MOAT • Revised DOR Firmware now supports Modulated 1Hz • Inputs directly from GPS clock in Stand-Alone configuration • Passes Fluorescent Lamp noise immunity test • Status LEDs & Test Header

14. 2006 Implementation and Beyond • VME form factor conditioner card (1) in 2006 • VME backplane fanout mezzanine card (1) in 2006 • VME form factor 12 channel fan-out cards (9) proto now • VME form factor Monitor card (1) 2006 goal (Least well defined) DOM Hub Coax Cable (2) GPS Clock … DSB RJ-45 Cable (90+) Serial Cable

15. Port Card Features, Rev 1 • 2-wideVME form factor; 12 RJ-45 port • Common 10 MHz from Backplane • Common modulated 1Hz from Backplane • Common Serial from Backplane • LVDS inputs, instead of direct GPS clock signals • One Point Signal conditioning • 2 minor schematic corrections • Skew tweaks • Additional Monitoring points: P2 and Header (e.g. demodulated 1Hz)

16. Backplane Fan-Out • Active Piggy-back card mated to “Back-of-Crate” style VME backplane over Socket “2” • Independently drive each EVEN socket • Match phase at each driven socket • 1-to-10 LVDS-to-LVDS fan-out chips • Match skew to each driven socket Piggy-back Card Industry Standard “Back of Crate” style Backplane

17. Conditioner card • Driven by GPS clock 10 MHz, 1Hz, and Serial • Modulate 1Hz signal • Control skew by design • Ensure symmetry of port signals • Drive LVDS backplane fan-out • Status indicators/LEDs Prototyped on the Fan-Out card

18. Monitor Card • Occupies an Odd Slot in VME card cage (e.g. #1) • Implement GPS clock Monitoring specified by PDR Document • Could contain FPGA, & SOPC, or SBC in a DIMM form factor • Could report via ethernet • Scope TBD

19. Watch List/Wish list • “Lock” status from GPS clocks • “Tracking” status from GPS clock • “Power” status from GPS clock • Parse time strings for error conditions • Parse GPS Clock console port output Rich set of Status bits/words Satellite constellation • Monitor phase offset between multiple GPS clocks

20. Verification • Reference signals on header10 pin header on DSB10 pin header on Fan-Out card4 pin header on DOR card (inside DOM Hub)

21. The Bottom Line • Fan-Out card comfortably meets the 5ns Requirement • Our GPS clock is a good choice, for the money • No speed bumps this year • On track for the final assaultSingle unit, flexible, modular designBuilds on previous successesAvoids heroics and death marches. Fin

22. Verifications in situ IceCube AMANDA Opt 1: Measure Fiber Round-Trip time (2 fibers) Opt 2: Use Portable Atomic Clock- packaged PRS-10- battery power Steps: 1. Measure/calibrate WRT IceCube clock 2. Transport clock to MAPO 3. Measure fiber distribution signals in MAPO against Atomic Clock 4. Transport clock back to IceCube Clock 5. Check CalibrationRepeat 1 – to – 5 until satisfied • Measure/Verify within 1ns should be achievable. 4”