1 / 108

SeaDataNet tools NEMO, MIKADO

SeaDataNet tools NEMO, MIKADO. M. Fichaut , IFREMER, France. Overview. SeaDataNet tools in their context NEMO : Reformatting tool Practical session : reformatting a set of stations NEMO and SDN Download Manager – coupling table, NEMO and MIKADO – CDI summary file

gwidon
Télécharger la présentation

SeaDataNet tools NEMO, MIKADO

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SeaDataNet toolsNEMO, MIKADO M. Fichaut, IFREMER, France

  2. Overview • SeaDataNet tools in their context • NEMO : Reformatting tool • Practical session : reformatting a set of stations • NEMO and SDN Download Manager – coupling table, NEMO and MIKADO – CDI summary file • Practical session : generation of a CDI summary file and of a coupling table • MIKADO : XML files generator, Web validation service, Link between MIKADOand SDN Download Manager – coupling table • Practical session : • Automaticcreation of XML files for CDI, Web validation of these files, creation of the corresponding coupling file • Manual input for EDIOS program, series and platform • Automatic creation of XML files for EDIOS

  3. CSR Metadata In database Meta data at XML ISO19155 New metadata Metadata in Excel files EDIOS Coupling table Data in database EDMED SEADATANET PORTAL Collection of ASCII files at SDN Format New data Data in ASCII file Format X NEMO EDMERP Download Manager CDI Local copy of data for downloading AAA Data request RSM Downloading European portal Data centres SeaDataNet Vocabulary MIKADO MIKADO

  4. ODV DIVA Data Centre CSR Local ID+ File address or SQL Select Coupling table Data in database EDIOS Collection of ASCII files at SDN format Data User Data Central ID EDMED SEADATANET PORTAL Zip file EDMERP Download manager Local ID Data request CDI Local ID + File address Local ID + SQL Select AAA Central ID + ID local + EDMO code Central ID + Local ID+ EDMO code Local copy of data for downloading RequestStatus Manager (RSM) Zip file Downloading

  5. NEMO – Reformating software

  6. SeaDataNet Vocabulary CSR Meta data at XML ISO19155 Metadata In database New metadata MIKADO Metadata in Excel files EDIOS MIKADO Coupling table Data in database EDMED SEADATANET PORTAL Collection of ASCII files at SDN Format New data Data in ASCII file Format X NEMO EDMERP Download Manager CDI Local copy of data for downloading AAA Data request Downloading RSM European portal Data centres : SeaDataNet nodes

  7. Objectives • NEMOis a reformatting software used to generate ASCII files at MEDATLAS or ODV formats which are defined as SeaDataNet formats for data exchange between SeaDataNet partners. Can be downloaded from SeaDataNet Web site : http://www.seadatanet.org/standards_software/software/nemo • It was created in IFREMER for internal purposes, and adapted to take into account the SeaDataNet formats and vocabularies.

  8. Technical characteristics • Release : 1.4.4 • Written in Java Language (Version >= 1.6) • Bilingual (French, English) • Available under multiple environments : • Microsoft : Windows 2000, WINDOWS 7, XP, VISTA , APPLE • Unix - Solaris • Linux • Interactive and batch modes available • Use of SeaDataNet common vocabularies web services • to update lists of values of the SeaDataNet common vocabularies • need network connections in order to have up to date lists of values. • But NEMO works offline once the lists are up-to-date

  9. NEMO main features • Reformat ASCII text file of vertical profiles, time-series or trajectories to a SeaDataNet ASCII format (ODV, MEDATLAS). • The input ASCII files can be: • one file per station for vertical profiles or time series • one file for one cruise for vertical profiles, time series or trajectories • Interact with Mikado, to be able to generate ISO-19115 XML descriptions of the data.

  10. NEMO principles • NEMO must be able to read almost any ASCII format to translate it to MEDATLAS, ODV • Users of NEMO describe the entry files format so that NEMO is able to find the information which is necessary in the SeaDataNet formats • One mandatory pre-requirement is that in the set of entry files the information about the stations must • be located at the same position : same line in the file, same position on the line or same column if CSV format • be in the same format • For example : for all the stations the latitude is • on line 3 on the station header, • from character 21 to character 27, or 3rd column in CSV • the format is +DD.ddd

  11. Running NEMO • 4 or 5 steps to convert the entry files to a SeaDataNet format • Describe the type of file(s) • [Describe the cruise, if the files are related to 1 cruise] • NEMO is able to read XML CSR generated with MIKADO • Describe the station information • Describe the measured parameters • Convert the file • One more step to • Save the description of the format (Model)

  12. NEMO - Description of entry files (1) • Where are the files? • Is it a cruise? • Is it one file per cruise or one file per station ? • Is it a collection of station files ? • grouped by cruises or not? • Are the files with separators? • Tabulations? Semicolon? Comma? • Are they vertical profiles, time series or trajectories data?

  13. NEMO - Description of entry files (2) File to convert is displayed in NEMO window The principle is to describe the input file by selecting the information in the file window

  14. Station header : constant number of lines

  15. Station header : number of lines not constant

  16. Cruise description Can be: Manually input Imported from a SeaDataNet XML CSR Imported from a database (IFREMER only)

  17. Station description

  18. Station description Select the date in the file, in the left part of the screen

  19. Station description Input the dateformat

  20. Station description Click on the Set button to tell NEMO the start and end position of the date

  21. Station description Click on the Test button to verify your input

  22. Station description

  23. Data description Choose the parameter list

  24. Data description Choose the parameter list Give the code, unit, position , format and input default value of each parameter

  25. Data description Select « Parameter list » in the context menu of the measured parameters table (right click to open the menu)

  26. Data description Add all the measured parameters and their unit (P061) using the context menu

  27. Data description Select the value of the measured parameter in the file, in the upper left part of the screen

  28. Data description Select « Set start /end » in the context menu of the measured parameters table

  29. Data description Start and End position of depth are added in the table

  30. Data description Describe the format and default value of the parameters (output default value only if output format is MEDATLAS)

  31. Data description Check your input : Select « Update test » in the context menu of the measured parameters table

  32. Data description The values read by NEMO are displayed in the Test column of the table. User is able to check that his inputs are correct

  33. Data description Tell NEMO which parameter is the vertical reference

  34. Output formats • MEDATLAS • Multi-station • ODV • Mono-station • Multi-station File conversion

  35. For sediment description in Sediment Core Compatible with ODV Non numeric parameters

  36. NEMO in batch mode • MIKADO can be run in batch mode, from NEMO install directory, using existing models • Several arguments can be added on the command line • Log file in NEMO install directory

  37. Files which can’t be reformatted by NEMO • Binary format : like EXEL, WORD … • File have to be first reformatted to text file • Files which do not respect NEMO pre-requirements • be located at the same position : same line in the file, same position on the line or same column if CSV format • be in the same format • Files which are not : • Vertical profiles (with depth or pressure as vertical reference) • Time series (with time as reference) • Trajectories (with latitude, longitude and time as reference)

  38. 1201 310219590650370-029-330100 2 19660719140058999000000000000080161 013 30029800000258+ + + + 201214 42010 28 0000 6000000036043261501461904 6000090030943320801460604 6000240039043409401465404 6000470054343452101472604 6000710047843449101470404 6000940052043472301472704 6001180059643491301476404 6001410065743509201479404 6001650060343506601477704 6001880051943491601474504 6002120052343496101475104 6002340046543491801473104 1201 31025246064008000060530000 2 19660806190058999000000000000170161 013 30030900000298+ + + + 201214 40213 0000 6000000147043307601503504 6000100133643417301500604 6000250119043467401496504 Like for example old historical tape formats

  39. NEMO – User manual • User manual is provided : File :sdn_Nemo_UserManual_V1.4.4.pdf • Detailed explanation for NEMO installation and use, lots of snapshots

  40. NEMO next release: • V1.5.0 (scheduled in autumn) • BODC vocabulary, version 2 • Pre-requirement • Importation tools must manage BODC V2 vocabulary • V1.5.1 (not scheduled yet) • Generation of files at NetCDF format • Pre-requirement • NetCDF format final definition

  41. Practical work on NEMOConverting files M. Fichaut Training Workshop – Ostende – 1-4 March 2010

  42. ODV files Collection of ASCII files XML CDI files SeaDataNet CDI summary_CDI_NEMO.xml Delivered with NEMO CDI summary CSV file MIKADO From ASCII data to XML files Coupling table Export Coupling table

  43. NEMO installation • Open the install_nemo_1.4.4_windows.zip file (on the desktop) • Unzip it • Double click on launcher_nemo.bat • Follow the installation procedure • Don’t forget to add the NEMO shortcut on your desktop

  44. Collection of ASCII files • Training data set • 6 CTDs raw files under SDN tools/Practical work/data/input files • ASCII files with measured parameters in column • Measured parameters = Depth, Temperature, Salinity, Fluorescence and Dissolved oxygen

  45. First step : conversion to ODV with NEMO • Run NEMO to convert the files to ODV format. • Modify the settings of NEMO • Default directories for data set to SDN tools/Practical work/data • Default directories for data set to SDN tools/Practical work/models • Enter your EDMO ID in the field below using the search function (right click) • Proceed the 4 steps of NEMO

  46. NEMO – File description • Choose the type of file (cruise directory, cruise file, …) • Browse your PC to input the data files directory • Once the file is open in NEMO, have a look to its content • Choose file type : Profile • Choose the output format : ODV, one unique file for all stations or One file per station • Enter the Station header information • there is a specific character string at the end of each Station header : *END* • The Data termination indicator is EOF (End of file) • Validate the step when description is OK

  47. NEMO – Station description • Station number is given on line “**Station” • Data type is ‘CTD stations’ • Time is U.T. • Latitude format is NDD MM.mm • Longitude format is NDDD.MM.mm • Bottom Depth is not available • Input all the mandatory fields : station number, time, date, latitude, longitude • Use the Test function to verify your input • Validate the step when description is OK

  48. NEMO – Data description (1) • Select the parameter list to use : P011 via P021 • Add the parameters : P021 codes are • Depth : AHGT, Temperature : TEMP, Salinity : PSAL, Fluorescence : FVLT, Dissolved oxygen : DOXY, Input the units of the parameters (for salinity, use Dimensionless) • Add a text label to each parameter • Set Start and End positions for the parameters • Tell NEMO which code is the vertical reference

  49. NEMO – Data description (2) • Format is mandatory • Describe the output format that you want • %6.1f : means 6 digits total (decimal point included), 1 digit for decimal, f for float (decimal number) • Use the contextual menu ‘Select the format’ for predefined format • Use the test fields to validate your input • Validate the step when description is OK • Save you model in the model directory

More Related