Statistics of the Spectral Kurtosis Estimator

1. Statistics of the Spectral Kurtosis Estimator Gelu M. Nita and Dale E. Gary New Jersey Institute of Technology RFI Mitigation Workshop, Groningen The Netherlands

2. Population Spectral Kurtosis RFI Mitigation Workshop, Groningen The Netherlands

3. Spectral Kurtosis Estimator RFI Mitigation Workshop, Groningen The Netherlands

4. Time domain signal Contaminated spectrum FFT Block S1 S2 RFI filter Add and Scale ||||||||||||||| RFI flags SK The SK Spectrometer • Key features: • Conceptual simplicity • Frequency channel independence • Straightforward FPGA implementation RFI Mitigation Workshop, Groningen The Netherlands

5. Statistical thresholds for the rejection of RFI outliers (M>>1)(Nita et. al, 2007 PASP, 119, 805:827) • Hardware implementation of the SK excision algorithm and initial tests revealed that, although it performs generally well, • the lower threshold level is set too low, failing to reject some RFI contaminated channels • the upper threshold level is set too low, rejecting more RFI free channels than statistically expected • CONCLUSION: More accurate RFI rejection levels are needed for improved and reliable performance. RFI Mitigation Workshop, Groningen The Netherlands

6. Characteristics of the SK Estimator(Monte Carlo Simulations) • The probability distribution of the SK estimator remains asymmetric even for a fairly large accumulation number M, approaching normality at a slower pace than needed for practical applications. • Main goals of this study: • Redefine the SK estimator to remove bias • Find an analytical expression for probability distribution of the SK estimator to allow accurate calculation of its tail probabilities RFI Mitigation Workshop, Groningen The Netherlands

7. Starting Point RFI Mitigation Workshop, Groningen The Netherlands

8. Joint Distribution (Monte Carlo): Mean of Squares-Square of Mean • S2 and S12 are strongly correlated random variables • Their (unknown) joint distribution would be needed to derive the distribution of their ratio • Is there any work-around approach? RFI Mitigation Workshop, Groningen The Netherlands

9. Joint Distribution (Monte Carlo): Mean of Squares to Square of Mean Ratio - Square of Mean Monte Carlo simulations suggest: • S2 /S12 and S12 are uncorrelated random variables • Their individual distributions are independent This is the fundamental property that makes the whole SK concept work by allowing SK to have a unity value independently of the power level This property was analytically proven for the exponential distribution based on first principles (Nita & Gary, PASP, in press) RFI Mitigation Workshop, Groningen The Netherlands

10. Raw statistical moments of the Mean of Squares to Square of Mean Ratio RFI Mitigation Workshop, Groningen The Netherlands

11. Analytical Results (Nita & Gary, PASP, in press) RFI Mitigation Workshop, Groningen The Netherlands

12. Redefinition of the SK Estimator RFI Mitigation Workshop, Groningen The Netherlands

13. Standard Moments of the SK Estimator RFI Mitigation Workshop, Groningen The Netherlands

14. Moment based approximation of the SK estimator distribution using Pearson Probability Curves Region of interest M > 23 RFI Mitigation Workshop, Groningen The Netherlands

15. Pearson Type IV PDF RFI Mitigation Workshop, Groningen The Netherlands

16. Pearson IV CF and CCF To compute the tail probabilities of the SK estimator, one needs to evaluate the cumulative function (CF) and complementary cumulative function (CCF) of the Pearson IV probability curve Knowing the analytical expression for the Pearson IV PDF, the CF and CCF can be computed analytically, by using the closed form expressions involving Hypergeometric series provided Heinrich(2004) or Willink(2008). Alternatively, CF and CCF can be computed by a simple numerical integration. The asymmetrical RFI thresholds are then chosen so as to provide symmetric tails probabilities of rejecting true Gaussian signals of 0.13499%, which are equivalent to the ±3 thresholds of a normal distribution. RFI Mitigation Workshop, Groningen The Netherlands

17. Willink Heinrich Numerical Integration RFI Threshold Computation Example 0.13499% 0.13499% RFI Mitigation Workshop, Groningen The Netherlands

18. Pearson IV PDF vs. Monte Carlo Simulations RFI Mitigation Workshop, Groningen The Netherlands

19. Time Domain Kurtosis Estimator RFI Mitigation Workshop, Groningen The Netherlands

20. Standard Moments of the time domain Kurtosis Estimator RFI Mitigation Workshop, Groningen The Netherlands

21. Kurtosis estimator Pearson IV PDF and RFI thresholds (M>45) RFI Mitigation Workshop, Groningen The Netherlands

22. Time domain signal Contaminated spectrum FFT Block S1 S2 RFI filter Add and Scale ||||||||||||||| RFI flags SK Summary The Spectral Kurtosis RFI excision algorithm recommends itself by • Conceptual simplicity • Frequency channel independence • Straightforward FPGA implementation (See Dale Gary’s following presentation) • Theoretically determined RFI thresholds for arbitrary integration time RFI Mitigation Workshop, Groningen The Netherlands

23. Main References • Gelu M. Nita and dale E. Gary, “Statistics of The Spectral Kurtosis Estimator”, 2010, PASP, in press • Gelu M. Nita, Dale E. Gary, Zhiwei Liu, Gordon J. Hurford, & Stephen M. White, 2007, "Radio Frequency Interference Excision Using Spectral Domain Statistics," Publications Of The Astronomical Society Of The Pacific, 119, 805. • Yuichi Nagahara, 1999, "The PDF and CF of Pearson type IV distributions and the ML estimation of the parameters," Statistics & Probability Letters, 43, (1999), page 251. • Joel Heinrich, 2004, "A Guide to the Pearson Type IV Distribution," Collider Detector at Fermilab internal note 6820, 2004, http://www-cdf.fnal.gov/publications/cdf6820 pearson4.pdf • Willink, R., 2008, "A Closed-Form Expression For The Pearson Type IV Distribution Function,“ Aust. N. Z. J. Stat. 50(2), 199, 205 RFI Mitigation Workshop, Groningen The Netherlands