ASKAP Signal Processing Overview DIFX Users and Developers Meeting

1. ASKAP Signal Processing OverviewDIFX Users and Developers Meeting Dr. Grant Hampson | ADE Team Leader 4 September 2012 CSIRO Astronomy and Space Science

2. ASKAP Overview • 36x12-metre antennas • 630 baselines • 188-port PAF receiver • “Tuneable” over 0.7-1.8GHz • Digitisation • Provides at least 300MHz processed BW • 30deg2FoV • 36-dual pol. Beams (varies with frequency) • Carries through to correlator • Tied array outputs also DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

3. ASKAP Differences • BETA (Eng. Test Array) • Dual heterodyne receiver • ADC at antenna • Dragonfly-2 digitiser (IF sampling) • Redback-2 DSP board inside ATCA chassis • Xilinx Virtex-6 FPGAs • ADE (ASKAP Design Enhancements) • RFOF (RF over fibre) from PAF to central site • All digital electronics in central site • Dragonfly-3 digitiser (direct sampling) • Redback-3 DSP board in 1U chassis • Xilinx Kintex-7 FPGAs DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

4. Digital Back-end Blocks 36-antennas 192-ports 2x16-bits 300 MHz =66Tbps =6600 x 10G Digital Receiver PAF Cross Connect Coarse Filter Bank ADC PAF Phase Switch Channel Select 192 Beamformer Array Covariance Matrix 36-antennas 2x36-beams 2x16-bits 300 MHz =25Tbps =2500 x 10G Fine Filter Bank 1 Beamformer Coarse Delay Fringe Correlator 1 Antenna Cross Connect Long Term Accumlator Correlator 36 Tied Array DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

5. PAF Electronics – The Domino • The “Domino” combines: • Low noise amplifier • RF filtering – defines 3 bands • RF over SM fibre transmitter • Each Domino contains electronics for 2 PAF ports: • 1.8W power per port • 250grams per port • Backplanes provide power,control and monitoring • Duplex LC/APC connections to MTP elite, connect RF to central site DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

6. ADE Backend – Digital Receiver • Upgrade to Xilinx Series-7 FPGAs • Kintex-325 FPGAs (840-DSP/RAM) • Contains first stage coarse filter bank • Direct sampling ADCs • NatSemi 12-bit 1600MSPS ADCs • Avagooptical MiniPODs • Multimode 120Gbps modules • FPGA connect directly at 10Gbps • 1U chassis packaging • No front or rear to boards • 16 ports per 1U • ~10W per PAF port DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

7. ASKAP (BETA) Processed Spectrum Sampled Bandwidth = 384MHz Processed Bandwidth = 304MHz Stage 1 filterbank 384MHz 768MHz Beamformer Second Nyquist zone First-stage filterbank output: 304 x 1MHz channels oversampled by 32/27 Stage 2 filterbank 54 x 18.52kHz fine channels (= 1MHz) sent to the correlator Second-stage filterbank output: 64 x 18.52kHz channels per 1MHz critically sampled Critically sampled filterbank DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

8. Frequencies? • Can’t have a single 768MHz channel – need to channelise • Beamformer requires ~1MHz resolution to apply weights • RFI – can remove frequencies that are troublesome – few MHz • Allows looking at frequencies that are well separated • Allows different simultaneous zooms on the same frequencies • Zooms • Fine delays/fringe stopping require ~100kHz to avoid smearing at band edges • To avoid aliasing near edges of 1MHz channels, and frequency rolloff • The 1MHz channels are oversampled by 32/27=18.5% • Tied Array / VLBI • Fine channels are stitched back together to form larger BWs (e.g., 16MHz) • To avoid aliasing and frequency rolloff, the fine channels are planned to also be oversampled • Don’t require sample rates to be a power of 2 DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

9. ADE DSP: Beamformer and Correlator • Common hardware platform • Xilinx Kintex-420 Series-7 FPGAs • ~1680 DSP/RAM per FPGA • 2 DDR3 modules @ 1066MTS • 120Gbps optical transceiver • Direct 10Gbps from FPGA to optics transceivers simplifies design • 1U chassis packaging • No front or rear to boards • 6 FPGAs per Redback board • Use of optical backplanes • Low capital cost – no power • 1.5Tbps optical backplane • Acts like “fixed” network switch • Scalable and modular DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

10. Why use FPGAs for Beamforming? • Why use FPGAs? • 10k multipliers per Redback-3 board (@300MHz = 3Tera ops/sec) • We are using 36x6=216 Redback-3 boards in beamformer • Beamforming, ACM and Fringe • Large processed BW – 300MHz with 36 dual pol beams is significant • Large bitwidths – linear up to beamformer output – preserving dynamic range • Lots of IO – guaranteed IO and lots of it • Power efficient • Why not use a GPU? • Network switch is a killer: 6600 input ports + 6600 output ports • size, price, power, cabling, ??? • Are GPU’s IO constrained for such an application? • How many GPUs per PC? DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

11. Why use FPGAs for Correlator? • We are using 75 Redback-3 boards in correlator (225TOPS) • Why not use a GPU? • Network switch: 2500 input ports + 2500 output ports • size, price, power, cabling, ??? • Processing still very regular • Correlation cells are not that different to beamformers (CMAC with DDR3) • Have to also manage Tied Array outputs, inverse filterbank • The balance of on-chip resources: logic, RAM, DSP units along with the high IO bandwidth and relatively low power consumption still makes the FPGA a compelling choice in obtaining a cost effective solution for these types of processing applications. DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

12. ASKAP ~1 Peta Op • Data rates going up • 10G to 25G • 10GbE pretty slow • 100GbE needsto become standard • Density going up • Better cooling My two bits ... • Issues with FPGAs??? • Communications • Programming • Software tools • Fixed interconnections? • Compilation times • Test benches • For single pixel feeds • No beamformer • Correlator is so much smaller • Pretty simple – GPU? • However, single pixel feeds can be wider BW • Maybe a factor of 10 larger? How many more dishes? • Issues with GPUs??? • Network switch ports and BW • Watching bit widths carefully • Conversion fixed-floating point • Data rates into GPU cards? • Density of processing? • Software tools DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

13. We acknowledge the WajarriYamatji people as the traditional owners of the Observatory site. Thank you CSIRO Astronomy and Space Science Dr. Grant HampsonADE Team Leader t +61 2 9372 4647 e grant.hampson@csiro.au w www.csiro.au/cass CSIRO Astronomy and Space Science