1 / 59

Improved Thresholds for Outlier Detection in L2OS Processor

Enhancing L2OS thresholds on TB measurements to eliminate outliers independent of L1c products. Explore outlier detection methods like dwell test and nsig test.

nhung
Télécharger la présentation

Improved Thresholds for Outlier Detection in L2OS Processor

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. L2OS threshold optimisation 20 June 2014, PM26 JL Vergely, J. Boutin, P. Spurgeon ACRI-ST,LOCEAN, ARGANS

  2. RFI/outlier detection Aim : • To improve the thresholds of the L2OS processor to be applied on TB measurements in order to remove outliers. About thresholds : • should be independent on L1c quality products

  3. Thresholds to be tested test for outlier detection (dwell test) nsig test for out of range TB detection (FOV test) Tm_out_of_range_affov Tm_out_of_range_eaffov Tm_out_of_range_stokes3_affov Tm_out_of_range_stokes3_eaffov Tm_out_of_range_stokes4_affov Tm_out_of_range_stokes4_eaffov test for oscillation TB detection (FOV test) Ts_std Ts_std_stokes3 Ts_std_stokes4 Other tests : max of iteration

  4. Tests conditions |X_swath| < 400 km Coast : 1000 km PCT_var < 80 -40° < lat < 40° SSS ref: Coriolis global NRTOA (MyOcean) Day : 1,2,3,4,5/5/2013, L1C v550 L2OS proc : v600 (CATDS processing chain)

  5. Indicators / SSS quality filter • chi2P : good TB fit if chi2P high. Chi2P > 0.05 in current processor. Warning : Dg_chi2P in L2OS processor = 1-Chi2P • SSS error < 1.4 psu • mean(SSS SMOS – SSS Coriolis) and std(SSS SMOS – SSS Coriolis) • X = (SSS SMOS – SSS Coriolis)/SSS_error. X should be close to a Gaussian law with mean(X)=0 and std(X)=1. Does not depend on SSS accuracy (close to the ratio between empirical error and theoretical error).

  6. Chi2P and RFI Percentage of RFI contamination : january 2012, asc Chi2P, 5/5/2013, asc

  7. nsig : full ocean Outlier detection. Dwell test. TB removed if : |TBsmos – TBmodel – DA| > nsig.rad_noise DA = mean dwell correction Current value : 5 Tested values : 2, 3, 4, 5

  8. nsig full ocean

  9. nsig full ocean 4 sigmas test Queue distribution : outliers Expected distribution Centred reduced variable

  10. nsig full ocean Mean and std(X) nsig = 2 No specific filter Chi2p > 0.05 & sigSSS < 1.35

  11. nsig full ocean Mean and std(X) nsig = 3 No specific filter Chi2p > 0.05 & sigSSS < 1.35

  12. nsig full ocean Mean and std(X) nsig = 4 No specific filter Chi2p > 0.05 & sigSSS < 1.35

  13. nsig full ocean Mean and std(X) nsig = 5 No specific filter Chi2p > 0.05 & sigSSS < 1.35

  14. nsig full ocean nsig=2: Many outliers at 4 sigmas

  15. nsig : coast (1000km)

  16. nsig coast Expected distribution

  17. nsig coast Mean and std(X) nsig = 2 No specific filter Chi2p > 0.05 & sigSSS < 1.35

  18. nsig coast Mean and std(X) nsig = 3 No specific filter Chi2p > 0.05 & sigSSS < 1.35

  19. nsig coast Mean and std(X) nsig = 4 No specific filter Chi2p > 0.05 & sigSSS < 1.35

  20. nsig coast Mean and std(X) nsig = 5 No specific filter Chi2p > 0.05 & sigSSS < 1.35

  21. nsig coast nsig=2: Many outliers at 4 sigmas nsig=2 : very biased !!

  22. Tm_out_of_range_affov or eaffov (polar X,Y,3,4) Snapshot removed if at least one TB is an outlier : |TB smos – TB model| > Tm_out_of_range Problem because the test is applied directly on the TBs and not on the TBs normalised by the radiometric noise X and Y from short and long integration time Current value : 50 K for affov and 100 K for eaffov Tested value : 10, 20, 30, 40 K

  23. Tm_out_of_range_affov full ocean

  24. Tm_out_of_range_affov full ocean Tm = 10K No specific filter Chi2p > 0.05 & sigSSS < 1.35

  25. Tm_out_of_range_affov full ocean Tm = 40K No specific filter Chi2p > 0.05 & sigSSS < 1.35

  26. Tm_out_of_range_affov full ocean

  27. Tm_out_of_range_eaffov full ocean Tm=10K : Little bit better but lost of accuracy No significative change (with Tm_out_of_range_affov = 12)

  28. Tm_out_of_range_stokes3_affov full ocean Tm=6K : Little bit better -> try to work in dual pol mode ? No significative change (with Tm_out_of_range_affov/eaffov = 12/18)

  29. Tm_out_of_range_stokes3_eaffov full ocean No significative change (with Tm_out_of_range_stokes3_affov = 8)

  30. Tm_out_of_range_stokes4_affov full ocean No significative change (with Tm_out_of_range_stokes3_affov/eaffov = 8/16)

  31. Tm_out_of_range_stokes4_eaffov full ocean No significative change (with Tm_out_of_range_stokes3_affov/eaffov = 8/16 & Tm_out_of_range_stokes4_affov = 10)

  32. Ts_std thresholds Snapshot is removed if : rms((TB smos –TB model)/ra) > Ts_std rms((TB smos –TB model)/ra) is expected to be close to 1 (if OTT well applied) Current value = 2.5

  33. Ts_std full ocean Ts_std=1.5 : Little bit better

  34. Ts_std_stokes3 full ocean Ts_std=0.5 or 1 : Too low Too biased No significative improvement

  35. Ts_std_stokes4 full ocean Ts_std=1 : Too low Too inaccurate No significative improvement

  36. Comparison current configuration and configuration without thresholds

  37. Comp current/without thres. full ocean current conf Chi2p > 0.05 & sigSSS < 1.35

  38. Comp current/without thres. full ocean without filter No signicative change Chi2p > 0.05 & sigSSS < 1.35

  39. Comp current/without thres. full ocean A little bit better without thresholds

  40. Comp current/without thres. coast Without thresholds : better for 4 sigmas SSS

  41. 5 day processing 1,2,3,4,5/05/2013; nsig = 2, 3, 4, 5 Dwell test. TB removed if : |TBsmos – TBmodel – DA| > nsig.noise DA = mean dwell correction Current config : nsig = 5

  42. Without LS mask. chi2P > 0.05 nsig=2

  43. nsig=3

  44. nsig=4

  45. nsig=5

  46. Iteration number 2 modes !!

  47. Iteration number Signature TEC ? RFI or island ? Hot spot

  48. Global improvement using iterMax Small global effect. What about specific area ?

  49. 4 zones with RFI/coast contamination Pacific + coast Pacific Atlantic Indian ocean

  50. 4 zones with RFI/coast contamination Pacific + coast Pacific Atlantic Indian ocean

More Related