RFI Mitigation Techniques at the ATA Garrett “Karto” Keating RFI2010 – Groningen, Netherlands

1. RFI Mitigation Techniquesat the ATA Garrett “Karto” Keating RFI2010 – Groningen, Netherlands March 31st, 2010

2. A Quick (Re)Introduction 20 ft/6.1 m Primary Wideband (0.5 to 11 GHz) Currently 42 Dishes Planned to move to 350 dishes, producing 61,425 unique baselines Currently 4 independent IFs and 3 independent FX 64-input correlators Data volume is destined to become HUGE

3. Welcome to RAPID Rapid Automated Processing and Imaging of Data. Flag, Calibrate, Image, Repeat.

4. RAPID and ARTIS • “Offline” data reduction package for ATA data • Development began in January 2008 • “Online” data reduction package for ATA data • Offshoot of the RAPID project • First light in March 2008 • In regular use since Feb 2009

5. The truth will set you free… I am not a software/computer engineer A “real” engineer would program in a “real” language I “program” using shell scripts (awk/grep/sed) wrapped around MIRIAD “Effective” troubleshooting

6. MacGyver would be proud “Like building a radio telescope from a Swiss army knife and duct tape” –Anonymous ATA Engineer • Limitations arise from programming choices: Limited math support Slow processing Messy code • The impact of these limitationscan be minimized!

7. Overall Processing Philosophy • Assume Nothing • Nothing is every truly “static” – LNAs, antennas, RFI, correlators and even objects in the sky are all dynamic • Pragmatic Processing • Processing choices must maximize time utility • “Fallback” Processing • Assume tasks will fail to do everything correctly • Allow for recovery at each processing stage (RFI/Calibration/Imaging)

8. System Overview Observation Parameters Images USER Master Obs ARTIS Antennas Correlator Catcher Flag Calibrate Image Archive

9. The RFI “cube” Time • “Axes of RFI” • Most RFI cases can be adequately described by its length, bandwidth and intensity • Each case requires a unique strategy and multiple layers of mitigation • Not to be confusedwith Rubik’s cube Long Medium Short Intensity Narrow Multichannel Bandwidth Broadband Moderate Weak Strong

10. Spectral Occupancy Power “Counts” Frequency Frequency

11. Spectral Occupancy

12. Spectral Occupancy 0.2s 0.1s 0.1s x100 Scale

13. Spectral Occupancy Data is broken up into temporal windows - the more spectra processed, the better/deeper the RFI processing can go With enough spectra, extremely weak RFI can be identified and excised. Bad channels and surrounding channels are flagged.

14. Benefits • No gains solutions required • Most spectra can be processing without having solved for the gains solution first • Results can be easily combined • Spectral occupancy results can be combined across antennas and correlator dumps, or “windowed” to limit processing to only select antennas, baselines, etc. • Processing is fast/computationally inexpensive • Achieves an 8:1 observing to processing ratio at the ATA

15. Drawbacks • SNR Limitations • Results dependant on noise-to-bandpass feature ratio • Normally becomes an issue with bright objects • Can be corrected by normalizing datasets • Can also be corrected with a good gains solution and sky model

16. Drawbacks • Is it really RFI? • Results can confuse wanted and unwanted RF emissions • Can be solved by creating “good frequency range” masks when flagging • Can also be corrected for during the imaging stage

17. Drawbacks • Results require a large number of samples • The smaller the array, the higher the required dump rate or the lower the sigma threshold • Multiple iterations sometimes required • More powerful RFI may need to be culled first before reaching weaker RFI • Trouble with “burst” RFI • RFI with an extremely short duration not likely to be picked up as consistently

18. Threshold RFI Removal Imaginary Real

19. Benefits • Fast with calibrated data • Hours of data can be processed with this method in a matter of minutes • Effective against “burst” RFI • Good at catching RFI left behind by spectral occupancy flagging • No iterative processing generally required • With calibrated data, generally one pass is adequate

20. Drawbacks • Slower with uncalibrated data • When used in conjunction with gains-solving processes, normally a 1:2 observing to processing ratio is achieved • Iterative processing with uncalibrated data • Process must be repeated after a new gains solution is built • Vulnerable against weaker RFI • RFI below the noise threshold difficult to catch

21. WRATH RFI Removal Continuum Image Channel by channel

22. WRATH RFI Removal WRATH flagging stands as the “last guard” against RFI Dynamic range and fidelity in images can easily double following use of the WRATH mechanism Pre-WRATH Post-WRATH

23. Benefits • Excellent “last guard” • Can generally catch the weakest of RFI • Fast processing, not dependent on data size • Processing scales with number of spectral channels, relatively invariant to the size of the dataset • Robust against calibration errors • Gains-solution errors usually affect all channels, thus don’t bias results

24. Who needs HI?

25. Who needs HI? • Spectral line problems • Spectral line features can (again) be misrecognized as RFI • Not a lost cause! • Spectral line channels can be masked a priori • Spectral line features should still image with “better” deconvolution, RFI should not

26. Drawbacks • Relatively slow for small datasets • Again, processing time is primarily proportional to number of channels • Vulnerable against “burst” RFI • RFI with an extremely short duration not likely to be picked up • “Baby and bathwater” dilemma • Highest risk for throwing away good data, since imaging artifacts may only be due to a single datapoint

27. Summary • Current system is robust against most RFI • Still some trouble with transient/”burst” RFI, but future upgrades to fix that • Processing able to keep up with observing • As processing speed increases, more processing tasks and choices can be made to render better images • Processing is highly scalable • Good astronomer acceptance • Most astronomers using ATA data use the RAPID package, particularly the RFI flagging routines

28. Source code available at svn.hcro.org/mmm/karto/RAPIDBeta karto@hcro.org Office number: 1-530-335-2364 Allen Telescope Array/Hat Creek Radio Observatory