1 / 37

Observation Screening

Observation Screening. Kertész Sándor 3D-VAR/ODB Training Budapest, 7 June, 2006. OUTLINE. ODB primer Observation data flow Screening QC and decisions in ODB Screening inputs How is screening working? Output listings Exercises. ODB primer. ODB is Observational Database

louie
Télécharger la présentation

Observation Screening

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Observation Screening Kertész Sándor 3D-VAR/ODB Training Budapest, 7 June, 2006

  2. OUTLINE • ODB primer • Observation data flow • Screening QC and decisions in ODB • Screening inputs • How is screening working? • Output listings • Exercises

  3. ODB primer • ODB is Observational Database • Basic data structure is called table. Tables is made up of data columns • Set of tables (with relations) forms a database type. E.g. ECMA that contains all the observations to be used in a DA system • Information on tables, columns etc. • http://www.cnrm.meteo.fr/gmapdoc/meshtml/DOC_odb/odb.html • In „Doc” directory on regatta: Assimilation_Chapter9_v2.3.pdf • ODB training!

  4. ODB primer • An ODB database for a given analyis date is a set of files organized into a directory structure (check your BATOR output!) • ODB data is divided into pools for parallel runs (to ensure load balance) • To look into ODB you need to run the viewer with your view! • The view define the retrieval!

  5. ODB primer • View definition (e.g. to retrieve AIREP data) • CREATE VIEW myview AS • SELECT • obstype, lat, lon, varno, obsvalue, status.blacklist@body • FROM hdr, body • WHERE obstype == 2 View name Retrieved columns Involved tables Selection criteria

  6. ODB primer • The concept of sub-bases was invented to have the choice for handling observation types separately. E.g.: ECMA.conv, ECMA.amsua, ECMA.amsub etc… • These ODB sub-bases must be merged into a new ODB • Many applications are working on this kind of ODB that is called virtual basis

  7. Observation Screening • The observation screening is the last step in the pre-processing chain to produce observation inputs for the variational analysis • Screening performs various quality control checks on data • ALADIN screening is very similar to the global (ARPEGE) one • Screening uses an ECMA ODB as input • The production of this ODB is rather complicated, we have to step back to BATOR to see the complete data flow …

  8. Obs dataflow: BATOR • BATOR is producing separate ECMA ODB sub-bases containing the required number of ODB pools • We need: #pools = # CPUs for further applications! • These ODB sub-bases must be merged into a new ECMA ODB using programmeSHUFFLE • Further applications are working on this ODB virtual basis

  9. Obs dataflow: LAMFLAG • The resulting ODB is not suitable forALADIN screening since it is not working with observations outside the C+I zoneAbort in GEFGER! • Solution: the input ODB must be restricted to the C+I zoneprogrammeLAMFLAG E LAMFLAG writes status flags into ODB C+I zone@hdr=0 zone@hdr=1

  10. Obs dataflow: LAMFLAG namelists Observation filter If .TRUE.: selection is based only on observation filter! &NAMFCNT LOBSONLY=.FALSE. /END &NAMFGEOM ELAT0=46.2447006399999978, ELON0=17.0000000000000000, ELATC=46.2447005948822678, ELONC=16.9999999982550491, EDELX=12176.1563281414165, EDELY=12176.1563281414165, NDLUN=1, NDGUN=1, NDLUX=229, NDGUX=205, Z_CANZONE=1500., LVAR=.T., LNEWGEOM=.F., /END &NAMFOBS LSYNOP=.T., LAIREP=.T., LSATOB=.T., LDRIBU=.F., LTEMP=.T., LPILOT=.T., LSATEM=.T. LPAOB=.F., LSCATT=.F., /END Model geometry settings Used for screening

  11. Obs dataflow: SHUFFLE reduction • LAMFLAG puts flags into ODB but does not reduce it! • SHUFFLE must run with setting TO_ODB_REDUC=1 for each sub-bases in the ODB separately to produce a reduced ODB • Then another SHUFFLE run is needed to put sub-bases together again (merging) • This results in a virtual base ECMA ODB again that is now ready for used as screening input

  12. Obs dataflow: Screening • Screening puts its quality control flags and decisions into the input ECMA ODB • QC flags, decisions for reports (in table hdr): • statusstatus_t • event1report_event1_t • blacklistreport_blacklist_t report_event1_t     no_data                  bit1     all_rejected             bit1     bad_practice           bit1     rdb_rejected            bit1     rdb_activated          bit1     whitelist_activated   bit1     horipos_outrange    bit1     vertpos_outrange    bit1     time_outrange         bit1     redundant                bit1     land                     bit1     sea                      bit1….. report_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1     stalt        bit1     scanpos      bit1     retrtype     bit1 ….. status_t activebit1 passivebit1rejectedbit1blacklistedbit1 …. Passive and blacklisted are a subset of rejected!

  13. Obs dataflow: Screening • QC flags, decisions and additional infromation for observed values (in table body): • status status_t • event1 report_event1_t • blacklist datum_blacklist_t • fg_depar = obs-guess datum_event1_t vertco_missing        bit1      obsvalue_missing    bit1      fg_missing              bit1      rdb_rejected            bit1      rdb_activated           bit1      whitelist_activated    bit1      bad_practice            bit1      vertpos_outrange     bit1      reflevel_outrange      bit1      fg2big                  bit1      depar2big              bit1      obs_error2big           bit1      datum_redundant      bit1      level_redundant         bit1 …. datum_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1 …. status_t activebit1 passivebit1rejectedbit1blacklistedbit1 ….

  14. Obs dataflow: Compression • After Screening another SHUFFLE is running, this time to perform an ECMA CCMA conversion • A CCMA ODB consists of only those tables and column that are needed for the analysis • The resulting CCMA ODB after SHUFFLE contains only the active reports and observations! • This CCMA ODB is the input of the minimization (e131) • The observation data flow is finished!

  15. ECMA (Full) LAMFLAG BATOR SHUFFLE (Reduc) ECMA.sb1 ECMA.sb2 … ECMA.sb1 ECMA.sb2 … SHUFFLE (Merge) SHUFFLE (Merge) ECMA (C+I) Screening SHUFFLE (ECMA->CCMA) CCMA Obs dataflow summary

  16. How to run screening? • It is configuration 002: • MASTER –c002 –maladin –vmeteo –t001 –aeul –eMIN1 • Inputs: • ECMA ODB (observation, sigma-o, blacklisting) • First guess file under 5 different names: • ICMSHMIN1INIT,ICMSHMIN1IMIN,ICMRFMIN10000,ELSCFMIN1ALBC000,ELSCFMIN1ALBC • Constants, statistics • Namelists

  17. Constants • errgrib: GRIB file containing grid-point space sigma-B values (lat-lon grid, several levels recieved from MF) • rt_coef_atovs_neepred_ieee.dat: coefficients for the RTM (received from MF), binary file • rszcoef_fmt.dat: the same the previous file but in ASCII format • bcor_noaa.dat: bias correction file • chanspec_noaa: obsolete? • rmtberr_noaa.dat: obsolete? • cstlim_noaa.dat: obsolete?

  18. Screening specific namelists • These parameters should be set like this for 3D-VAR: • NUPTRA = -1 , number of updates of the trajectory (NAMVAR) • NOBSHOR = 201, bilinear horizontal interpolation in H (NAMVAR) • LOBS = .TRUE., real obs (NAMCT0) • LOBSC1 = .FALSE., (NAMCT0), traj. is not computed • LSIMOB = .FALSE., no simelated obs (NAMCT0) • LSCREEN = .TRUE., (NAMCT0), screening is on! • LSCRE4D = .FALSE. , 3D screening (NAMSCC) • Tunable parameters are in NAMSCC and NAMOBS (discussed later)!

  19. How is screening working? Decisions are taken in subroutine: Independent decisions: performed separately for each report, RDB flag is assigned to reports DECIS rdbflag@hdr rdbflag@body Update of flags (FLGTST). Flags are converted into status: active, passive, blacklisted, rejected Dependent decisions: dependency on other reports or obs, or the previous decisions

  20. Independent decisions • Preliminary checks (PRECH): Completeness of report missing station altitude for SYNOP, TEMP and PILOT… • Blacklisting (BLACKCLN, BALCKSAT) • Background quality control (FIRST)

  21. Blacklisting • Blacklisting information is placed into ODB by BATOR • In the screening a separate selection is applied for satellite data (BLACKSAT) • Blacklisting is also performed in BLACKCLN: • Blacklisted obs will be rejected • Various blacklisting for SATOB (e.g. QI < 85) • Land-sea rejection (SYNOP, TEMP, PILOT). Controlled via NAMOBS LSLREJ (default is .T.) • LSLRW10: Wind 10m, LSLRT2: T 2m, LSLRRH2: THU 2m (default is .T. meaning rejection over land. Will get a passive status!)

  22. Blacklisting • Also in BLACKCLN: • Height-based rejection. Controlled via NAMOBS LHDLREJ (default is .T., rejected obs will get passive status!) • Orographic rejection limit is applied: • LHDRW10: Wind 10m, LHDRT2: T 2m, LHDRRH2: RHU 2m (default is .T.) • AIREP data and significant levels for TEMP and PILOT are not used below the model surface! • RHU and Q is rejected above 300 hPa • AIREP is not used below a certain pressure

  23. Background QC (FIRST) • From the BLUE theory we know: • For a given location we expect: Values are read from file errgrib and interpolated to obs locations Values are read from ODB

  24. Background QC (FIRST) • Flags are assigned to observations by checking: Based on predefined flaglimits 4 flag values can be assigned to obs 0: correct 1: probably correct 2: probably incorrect 3: incorrect Flaglimits are defined in DEFRUN

  25. Dependent decisions • Remove redundant surface levels from TEMP/PILOT (REDSL) • Vertical consistency of profiles (VERCO) • Removal of duplicated reports (DUPLI): two reports with the same ID, time an location … • Redundancy check (REDUN) • Thinning for AIREP (THIAIR) • Specific scatterometer quality control • Thinning of satellite data (THINN)

  26. Redundancy check (REDUN) • Applied for all active reports which are co-located and originate from the same station. Separate treatment for: • Land SYNOP: more reports for the same place but with different time, the one with closest to the analysis date with the most active obs are selected • Ship SYNOP, DRIBU: redundant if moving platforms are within a circle of 1° radius • TEMP and PILOT • SYNOP mass observation is redundant if TEMP obs is available and the diff < 50 hPa

  27. Thinning of AIREP data (THIAIR) • One flight consist of a set of reports • Thinning is performed for each flight separately • 3D boxes are constructed around model levels • In each box the report closest to the analysis date with the most active observations are selected • Box size is set in metres via RFIND_AIREP in NAMSCC, the default is ~70 km! • By default box size less than 50 km cannot be used in ARPEGE, code modification is needed

  28. Thinning of satellite data (THINNER) • For radiances a horizontal thinning is performed in two steps: • First a minimum distance is enforced (default ~ 70 km, can be changed in NAMSCC via RMIND_RAD1C(sensor)) • Then a repeated thinning is performed to reach the final separation (default ~ 140 km, can be changed in NAMSCC via RFIND_RAD1C(sensor)) • Where sensor is: 0=HIRS, 1=MSU, 2=SSU, 3=AMSU-A, 4=AMSU-B, 6=SSM/I, 20=METEOSAT, 21=MSG_HR, … • Selection criterion: • Sea is preferred over land • Clear sky pixel is preferred over a cloudy one • Obs closer to the analysis date is preferred

  29. Thinning of satellite data (THINNER) • For SATOB a the thinning is performed: • Horizontally in two steps similarly to radiances: controlled in NAMSCC via RMIND_SATOB, RFIND_SATOB (default is ~70 and ~140 km) • Vertically around model levels • In each box the report closest to the analysis date with the most active observations are selected • QI values are also used in the selection

  30. Sreening output • Valuable summary about screening decisions can be found in the NODE file: • Try „grep SCREENING STATISTICS” to get: • STATUS summary • EVENT summary • Number of variables, departures and missing departures • Diagnostic JO-table

  31. Sreening output • Summary at „grep SCREENING STATISTICS” • Report and data status summary OB.TYP REPORTS ACTIVE PASSIVE REJECTED BLACKLISTED 1 725 44 0 681 0 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 61 60 0 1 0 6 61 59 0 2 0 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ---------------------------------------------------------------------------------------- TOT 847 163 0 684 0 OB.TYP DATA ACTIVE PASSIVE REJECTED BLACKLISTED 1 4576 164 1846 4412 3862 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 9186 7271 1218 1915 705 6 3438 2050 64 1388 72 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ------------------------------------------------------------------------- TOT 17200 9485 3128 7715 4639

  32. Sreening output • Summary at „grepSCREENING STATISTICS” • Diagnostic Jo table Obstype 1 === SYNOP, Land stations and ships -------------------------------------------------- Codetype 11 === SYNOP Land Manual Report Variable DataCount Jo_Costfunction JO/n ObsErr BgErr U10 1380 2250.937197524 1.63 0.200E+01 0.000E+00 H2 721 834.1196058351 1.16 0.100E+00 0.000E+00 Z 567 892.5077244762 1.57 0.785E+02 0.000E+00 T2 721 2138.950482967 2.97 0.140E+01 0.000E+00 Q 721 1577.382611689 2.19 0.998E-03 0.000E+00---------- --------------------------- -------- ObsType 1 Total: 4110 7693.897622492 1.87

  33. Sreening output • Observation monitoring available at: • http://pc2088.met.hu/monitor/start.php

  34. Thank you for your attention! Any question?

  35. Exercises • Run screening for all the prepared observations with script ~/workdir/3dvar/Assim and study the output listing! (You have to put an exit after screening in the script).

  36. Exercises • Examine in detail the SYNOP and TEMP report for Budapest, see if surface obs (except Z) are really rejected, and find the reason for it. Try this view first: • DEFINE VIEW bp AS • SELECT • obstype, status.active@hdr,varno, • obsvalue, fg_depar, press, status.active@body, • status.passive@body, status.blacklisted@body, status.rejected@body, • event1.land@body, event1.datum_redundant@body • FROM hdr, body • WHERE (obstype == 1 OR obstype == 5) && statid = ‘12843 ‘ • Guidance: A prepared viewer par file and sql file can be found in ~wshop01/workdir/OdbViewer as bp.par bp.sql. Copy them to your directory, set the sql path fo your file and type „viewer –p bp.par” to run viewer. • If you are brave enough try to write the same view for AIREP (obstype == 2) and explore bit fields of event@body!

  37. Exercises • Re-run screenig with a 10 km thinning distance for AIREP. To do this you have to use the exe you created yesterday (by modifying arp/obs_preporc/thiair.F90to reduce ISCALE to 1000!) Modify PROC_LAM_ODB and d_RUN in include.inc! Compare the default and the new airep thinning in the output listings or in the ODB!

More Related