Data quality tool: frequency domain lines identification by the PSS library (Rome 1 group)

1. Frequency Lines Identification: cmt package for On-line (by Fd ) and Off-line Virgo data qualitySabrina D’Antonio Roma2 Tor VergataRoma 1 Pulsar group:Pia Astone, Sergio Frasca, Cristiano Palomba, Federica Antonucci

2. Data quality tool: frequency domain lines identification by the PSS library(Rome 1 group) SUMMARY: Time domain disturbances: event’s identification and removal (to obtain clean data) Estimation of the average AR spectrum Frequency domain lines identification The log file and the spectra files

3. Time domain disturbances: event’s identification and cleaning We want to identify and remove “high frequency shorts events”, which will increase the level of the noise. Events identification: after high-pass bilateral filtering (not phase shifts) with fhp=100 Hz. Once high frequency events have been found and their parameters registered, we “subtract” them from the original time series. Hence we produce the “cleaned data sets”.

4. Time domain disturbances: event’s identification • We evaluate AR mean and standard deviation; • The threshold is set on the critical ratio CR defined as • The memory time depends on the apparatus. We set it to 600 s and the CRthr to 6. • The “dead time”, minimum time between two events, depends on the noise and the expected signal. We are using 1s.

5. Time domain disturbances: cleaning • The cleaning requires the set up of another parameter, “the edge width”: it indicates how many seconds before and after the event are used in the cleaning of the data. We have used 0.15 s • From the “beginning time” up to the “beginning time + duration” we subtract the high frequency component to the data. • Data from the “beginning time – edge width“ to “beginning time” and data up to “beginning time+duraton+edge width” are linearly interpolated.

6. The procedure to estimate the average spectrum A good estimator should have the following properties: If peaks in the frequency domain are present, the estimator should not be affected by peaks. This should be as much as possible independent on the SNR of the peak; If the noise level varies, either slowly or rapidly, the estimator should be able to follow the noise variations. We refined the procedure, with the use of an autoregressive estimation (AR) of the average of the spectrum, with the basic idea of a “clean estimator” .

7. The procedure to estimate the average spectrum AR estimation. Already described applied from higher frequencies toward lower frequencies. To deal with the increasing of the noise level toward lower frequencies. • FFT length: 4194314 ->T=1048.6s • FFT mode: overlapped (or not) by the half, flat top- cosine window

8. Lines frequency identification: peak map The frequency lines search starts with the ratio R of the spectrum to its AR estimation. On this function, we set a threshold at the level of SNRthr =(2.5)0.5. In the log files we record frequency lines with SNR≥GEN_FAC*SNRthr(GEN_FACT used =2). All the data which cross the threshold are local maxima are then registered into the log file.

9. Output Files • Log file Date_of_creation.log: * information about time domain and frequency domain events *one file for all the processed time period (?) *24 Mb (10 days). • Spectra files: PS with high frequency resolution df=9.5367e-004 Hz A new file every 100 FFT (?) Dimension ~ 850 Mb PS with lower frequency resolution df*128 Hz Dimension ~ 6.3 Mb

10. Log file info • PSS crea_sfdb job log file • started at Wed Jan 23 09:51:32 2008 • INPUT : VIR_h_4000Hz_869983200.GWF First data time in the first file of the run • OUTPUT : VIR_h_4000Hz_869983200.SBL The first SBL file opened • ! even NEW: a new FFT has started • ! PAR1: Beginning time of the new FFT • ! PAR2: FFT number in the run • ! even EVT: time domain events • ! PAR1: Beginning time, in mjd • ! PAR2: Duration [s] • ! PAR3: Max hp data amplitude*EINSTEIN • ! PAR4: Max CR • ! PAR5: Energy (sum of squared amp)

11. Log file info ! even EVF: frequency domain events, with high threshold • ! PAR1: Beginning frequency of EVF • ! PAR2: Duration [Hz] • ! PAR3: Ratio, in amplitude, max/average • ! PAR4: Power*EINSTEIN**2 or average*EINSTEIN (average if duration=0, when age>maxage) • ! par GEN: parameters of the AR spectrum estimation • (PAR) GEN_THR = 2.500000000 • (PAR) GEN_TAU = 0.020000000 • (PAR) GEN_MAXAGE = 0.020000000 • (PAR) GEN_FAC = 2.000000000 • ! GEN_THR is the threshold in amplitude • ! GEN_TAU the memory frequency of the AR estimation • ! GEN_MAXAGE [Hz] the max age of the process. If age>maxage the AR is re-evaluated • ! GEN_FAC is the factor for which the threshold is multiplied, to write less EVF in the log file

12. Log file info ! par GEN: general parameters of the run (PAR) GEN_BEG = 54313.249837963 (PAR) GEN_NSAM = 2097152.000 (PAR) GEN_DELTANU = 0.000953674 (PAR) GEN_FRINIT = 0 ! GEN_BEG is the beginning time (mjd) ! GEN_NSAM the number of samples in 1/2 FFT ! GEN_DELTANU the frequency resolution ! GEN_FRINIT the beginning frequency of the FFT (PAR) EVT_CR = 6 (PAR) EVT_TAU = 600 (PAR) EVT_DEADT = 1 (PAR) EVT_EDGE = 0.15 ! EVT_CR is the threshold ! EVT_TAU the memory time of the AR estimation ! EVT_DEADT the dead time [s] ! EVT_EDGE seconds purged around the event

13. Log file info • (PAR) EVF_THR = 2.5 • (PAR) EVF_TAU = 0.02 • (PAR) EVF_MAXAGE = 0.02 • (PAR) EVF_FAC = 2 • ! EVF_THR is the threshold in amplitude • ! EVF_TAU the memory frequency of the AR estimation • ! EVF_MAXAGE [Hz] the max age of the process. If age>maxage the AR is re-evaluated • ! EVF_FAC is the factor for which the threshold is multiplied, to write less EVF in the

14. Log file info • --> NEW > 54313.249837963 1 • --> EVT > 54313.249837963 0.0055 3629.99 181.981 5.03206e+07 • --> EVT > 54313.250697575 1.0575 241.162 11.2496 3.04115e+07 • ……. • --> EVF > 1999.696731567 0.0038147 6.29069 292307 • --> EVF > 1999.495506287 0.0038147 9.73132 714481 • --> EVF > 1998.849868774 0.00286102 5.82867 452195 • >>> TOT > 22629 • --> NEW > 54313.255906111 2 • --> EVT > 54313.255906111 0.006 11499.6 248.579 4.91038e+08

15. Virgo data from T0=869983200 (2007-08-01-05:59:45)up toT=870537636 • 818 FFT (4.9637 days) • EVT veto=21303/2 (total time vetoed 2919.2/2 s) • EVF=245996

16. From the log files: Time-events veto <A>=8.895 10-17 =5.95 10-15

17. From the log files: Time-events veto 4 events family one with double duration respect to the other.

18. From the log files: Time-events veto

19. From the log files: Time-frequency plot

20. From the log files: Time-frequency plot

21. From the log files: Time-frequency plot

22. From the log files: Time-frequency plot

23. From the log files: Time-frequency plot Frequency lines detected from: (from Gabriele LineMonitor) (mag) 874444952 to 874589542 (green) 863680494 to 863831687

24. From the log files: Time-frequency plot Frequency lines detected from: (from Gabriele LineMonitor) (mag) 874444952 to 874589542 (green) 863680494 to 863831687

25. From the log files: Time-frequency plot

26. From the log files: frequency lines hist.

27. From the log files: frequency lines hist.

28. From the log files: Amplitude-frequency.

29. From the log files: CR-frequency.

30. From the log files: CR

31. From the log files: CR

32. From the log files: Duration vs frequency red dot : Duration> EVF_MAXAGE = 0.02 Hz

33. From the short PSTime-frequency plot

35. TO BE DONE: To write the documentation To define with interested people : files dimension (open a new file after N FFT) Writing of the output files optional Lower the SNRthr …. Suggestions…