1 / 84

Virtual Observatories and Data Interfaces for Atmospheric Science

Virtual Observatories and Data Interfaces for Atmospheric Science. 12th EISCAT International Workshop Incoherent Scatter Radar School Swedish Institute of Space Physics, Space Campus, Kiruna, Sweden Bill Rideout MIT Haystack Route 40, Westford, MA, USA 1-781-981-5624

content
Télécharger la présentation

Virtual Observatories and Data Interfaces for Atmospheric Science

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Virtual Observatories and Data Interfaces for Atmospheric Science 12th EISCAT International Workshop Incoherent Scatter Radar School Swedish Institute of Space Physics, Space Campus, Kiruna, Sweden Bill Rideout MIT Haystack Route 40, Westford, MA, USA 1-781-981-5624 brideout@haystack.mit.edu 26 August 2005

  2. Outline • Virtual Observatories • Madrigal • Web interface • Remote API • Extending/Contributing • Madrigal 2.4 • Cedar Database • Other data sources

  3. A day in the life of an Atmospheric Scientist I have done an experiment with my instrument, but now I need to … • Search numerous websites for data • Figure out their parameters, units • Figure out their coordinate system, date format • Figure out how to determine data quality • Write code to download data, or (worse) manually download • Write code to convert to your format • Finally, do science

  4. How can Virtual observatories help? Virtual Observatories – one stop data shopping!

  5. Virtual Observatories • Ideally… • Provide a single interface to access all data • Knows about all data sources • Allows simple, powerful searches to discover unknown data sources • Always gets the most up-to-date data • Uses a single set of well-defined parameters • Provides data in consistent format(s) • Provides data in consistent coordinates • Informs user of contact information and rules-of-the-road for all data

  6. Two approaches • Top down • Bottom up Build an interface Build a standard data source

  7. How do they work? • Top-down approach: • Accept that all data sources will be forever incompatible • Build a data model so metadata can be shared • Build a unique interface to interface to each new data source. • Scales linearly with number of data sources. • Works best with more uniform data (i.e., astronomical images) • Bottom-up approach: • Standardize data format and semantics • Standardize data provider API • Approach taken by Madrigal/Cedar • Try for community acceptance

  8. Outline • Virtual Observatories • Madrigal • Web interface • Remote API • Extending/Contributing • Madrigal 2.4 • Cedar Database • Other data sources

  9. What is the Madrigal database? • An open-source, web-based database designed to hold one group’s data • www.openmadrigal.org has all code and downloads • Built upon the Cedar database format established over 20 years ago • Fundamentally a data source – allows local owners to improve/correct their data • Designed to be used for a wide variety of instruments • New installations always welcome!

  10. Madrigal Data Model Madrigal site (typically a facility with scientists and a Madrigal installation) ↓ Instruments (ground-based, typically with a set location) ↓ Experiments (typically of limited duration, with a single contact) ↓ Experiment Files (represents data from one analysis of the experiment) ↓ Records (measurement over one period of time) ↓ Data shared among all Madrigal sites Data unique to one Madrigal site

  11. Madrigal Records Records (measurement over one period of time) Three types: • Catalog record • descriptive information about entire experiment • Header record • descriptive information about one section of experiment • Data record • Stores values • All parameters defined by Cedar Database standard • Contains 3 parts • Prolog • 1D records • 2D records

  12. Madrigal Data Records • Prolog • Start and end time • Instrument id • Kind of data id • 1D records (scalar) • Single value parameters • 2D records (vector) • Multiple value parameters • All parameters must have same number of rows • Meant to allow multiple spatial measurements • Not meant for time variation – conflicts with Prolog! Prolog Data record ID (scalar) – S/N=2.5 2D (vector) – Altitudes = 100,150, 200,250,300,350

  13. Cedar/Madrigal Database • All parameters in file defined • http://cedarweb.hao.ucar.edu/documents/parameters_list.txt • Ranges of parameters for each instrument • Data stored in one or two 16 bit ints • Additional increment parameters • Error parameters • Mnemonics start with D • Code is negative of parameter

  14. Cedar/Madrigal Database, continued • Special values • missing • assumed (error value only) • knownbad (error value only) • Defined in • http://cedarweb.hao.ucar.edu/cgi-bin/cedar_file_access.pl?filename=documents/cedar_fmt.pdf

  15. Cedar Database parameters Example additional increment parameter

  16. Cedar parameters - continued • Madrigal contains many “derived only” parameters • Not included in Cedar standard • Cannot be stored in Cedar file • New python API hides the existence of additional increment parameters • All values are doubles • Exceptions occur on overflow • More later…

  17. Madrigal Derivation Engine • Derived parameters appear to be in file • Assumes information can be derived from records • Time from prolog • Position either as 1D or 2D • Other parameters • Engine determines all parameters that can be derived

  18. Classes of derived parameters • Space, time • Examples: Local time, shadow height • Geophysical • Examples: Kp, Dst, Imf, F10.7 • Magnetic • Examples: Bmag, Mag conjugate lat and long, Tsyganenko magnetic equatorial plane intercept • MSIS • Examples: Tn, Nol

  19. Outline • Virtual Observatories • Madrigal • Web interface • Remote API • Extending/Contributing • Madrigal 2.4 • Cedar Database • Other data sources

  20. Madrigal web interface - homepage Access Data All Madrigal sites

  21. Three ways to access Madrigal data Data in individual experiments Data across experiments Plot data across experiments

  22. Searching for experiments Choose one or more instruments By default, view only most recent files By default, all Madrigal sites are searched Find any experiments with any overlap with these dates

  23. Madrigal experiment listing These links could be to experiments at any site

  24. Madrigal experiment files – part 1 These two files have no catalog or header records, otherwise there would be a link Data browser (isprint) allows viewing both measured and derived parameters with filtering

  25. Madrigal experiment files – part 2 Madrigal allows any additional web-compatible files to be added to the experiment Image-conversion feature written at Eiscat Notes can be added by users – also written at Eiscat

  26. Data browser (isprint) – part 1 Users can define filters to select certain filters and parameters with one click Filters to reduce data • Time • Altitude • Azimuth • Elevation …

  27. Data browser (isprint) – part 2 Filters, continued • Filter data using any • parameter, or the sum, • difference, product or • quotient of two • parameters. Example: Nel –DNel > 1

  28. Data browser (isprint) – part 3 See a full description of all parameters Choose parameters to display • Measured in bold • Derived in normal font • Listed by category • Click on any parameter for a full description

  29. Data browser (isprint) – part 4 User clicked on CHISQ Some parameters have a more complete description

  30. Data browser (isprint) – part 5 Longer description of CHISQ

  31. Data browser (isprint) – part 6 String to indicate missing data Show header for each record option

  32. Data output Display only text Save text version to file Summary of selected filters Headers were on in this example

  33. Second approach – Global search Global search for data

  34. Global search – part 1 Choose kinds of data Choose one or more instruments Choose seasonal filter (optional) Choose date range (optional)

  35. Global search – part 2 Filter by experiment name (optional) Select parameters to display Filter using any parameter, just like isprint

  36. Global search – selecting parameters Parameters with categories and pop-up definitions as on isprint page

  37. Global search – review search Review all aspects of the global search before submitting

  38. Global search – returned message Message returns number of files being searched, along with rough estimate of time required. Since reports may take a long time to generate, a email with a link is sent when done

  39. Third approach – Plotting across experiments Plotting data from various instruments across experiments

  40. Creating plots Select one or more instruments. In this example Svalbard and Millstone (not visible) selected. Click here to see a list of all experiments Select date range (can cross experiment boundaries) Select a scatter plot or pcolor of altitude versus time

  41. Choose single parameter to plot Same pop-up listing of parameters as in isprint Radio buttons, since only one parameter can be selected

  42. Set up limits and filters Set limits on the parameter you selected Data can also be filtered using another parameter If a pcolor plot, can set altitude limits

  43. Pcolor plot output Rules of road for each site shown Single request generates Millstone and Eiscat plots Plot are requested from each site simultaneously to improve performance

  44. Pcolor plot output – part 2 Can add more stacked plots with different parameters, or start over

  45. Adding additional plots Now add a scatter plot of DST with same time scale

  46. Adding additional plots – part 2 Time scales align with stacked plots if times not changed

  47. Outline • Virtual Observatories • Madrigal • Web interface • Remote API • Extending/Contributing • Madrigal 2.4 • Cedar Database • Other data sources

  48. Remote Access to Madrigal Data • Built on web services • Like the web, available from anywhere on any platform • Complete Matlab and Python API written • More APIs available on request or via contribution

  49. Madrigal Web Services • Simple delimited output via CGI scripts • Not based on SOAP or XmlRpc since no support in languages such as Matlab • CGI arguments and output fully documented at http://www.haystack.edu/madrigal/remoteAPIs.html

  50. Madrigal Web Services – part 2 • To write a new API, each method must • Take input arguments and generate the correct CGI URL • Parse the delimited text • Return data to user

More Related