Searching for Data with the Virtual ITM Observatory

1. Searching for Data with the Virtual ITM Observatory D. Morrison, M. Weiss, R. Daley, L. Immer, C. Colclough, R. Holder, J. Jen, D. Patrone, M. Hashemian, P. Meckel, M. Potter, R. Barnes, S. Nylund, J.-H. Yee, and E. Talaat (Johns Hopkins University, Applied Physics Laboratory 11100 Johns Hopkins Rd. Laurel MD, 20723: 240-228-4172) J. Russell (Hampton University) R. Heelis (Univ. Texas at Dallas) J. Kozyra (Univ. of Michigan) D. Bilitza, R. McGuire, and R. Candey (NASA Goddard Spaceflight Center) P. Fox (HAO/NCAR) Virtual Observatories in Geosciences (VOIG) Conference 2007

2. Primary Tasks for Mission Data Center or Virtual Observatory • (1) describe their data holdings and resources • (2) discover what data sets are available for solving the user’s particular science problem • (3) provide a means to access this data or resource • (4) provide tools to help personnel use the data that they have found. Virtual Observatories in Geosciences (VOIG) Conference 2007

3. What is a VxO? • “A Virtual Observatory (VO) is a suite of software applications on a set of computers that allows users to uniformly find, access, and use resources (data, software, document, and image products and services using these) from a collection of distributed product repositories and service providers. A VO is a service that unites services and/or multiple repositories.” • An observatory is a location used for observing terrestrial and/or celestial events. • A virtual observatory is a system that allows an observer to work as if they were at the observatory. • The VO does this typically by allowing the observer to access multiple data files (“observations”) from multiple sources of the object of interest. • The VO allows access to “different instruments” to study a given system or phenomena. • VitmO ties together multiple data sets from different satellites and ground stations to create the appearance of having many different types of instruments observing the same phenomena from different locations. Virtual Observatories in Geosciences (VOIG) Conference 2007

4. Virtual ITM Observatory • The Virtual ITM Observatory (VITMO) is one of the recently awarded domain specific NASA virtual observatories (VxO). • VITMO will focus on data covering the ITM region as well as providing connections to the energy drivers for this region (principally solar and magnetospheric). • VITMO is focused on the “interdisciplinary” ITM user rather than the “instrument team” user. Virtual Observatories in Geosciences (VOIG) Conference 2007

5. ITM is Diverse in Observables An ITM VO must facilitate correlative study between multiple parameters from multiple sources and cross-discipline studies. Virtual Observatories in Geosciences (VOIG) Conference 2007

6. ITM Domain Differences • Data covering the ITM region has a number of differences from data in other domains. • The ITM region is observed by ground based remote sensing instruments, satellite based remote sensing instruments, and in-situ satellite instruments. • There are external drivers in solar radiation and the solar wind and magnetospheric particle inputs. • A Virtual Observatory that covers the ITM region needs to deal with the large diversity of data types in the study of this region. • A heterogeneous data format environment also exists here. • Solar imagery is typically in FITS files, ground based data are typically ASCII or NetCDF, in-situ satellite data are commonly in CDF format, remote sensing satellite data are typically HDF, NetCDF, or CDF. • Non-NASA data sources (i.e. NSF, NOAA, DoD) are important contributors to the ITM domain. Virtual Observatories in Geosciences (VOIG) Conference 2007

7. Discovery of Data • With the introduction of imaging instruments in ITM physics the data volume has exploded. There are now terabytes of data in the ITM domain. • This talk will focus on (2) discover what data sets are available for solving the user’s particular science problem. • VITMO uses an approach where external databases, calculated events, or other externally generated event lists can be used to restrict data searches to precise periods of relevance to the user study. Virtual Observatories in Geosciences (VOIG) Conference 2007

8. The Ability to Find the Appropriate Data Will Be a Key Requirement of a VITMO Increasingly satellites in the ITM community will rely on remote sensing instruments – TIMED, DMSP, NPOESS Virtual Observatories in Geosciences (VOIG) Conference 2007

9. Typical Researcher Workflow http://swdcwww.kugi.kyoto-u.ac.jp/index.html • Workflow • Identify periods of interest (onset of storm, etc) • Go to various websites to download data from different instruments • Plot data to identify overlaps in time and space of data sets • If appropriate data not found repeat loop with different time period http://www.srl.caltech.edu/ACE/ASC/ Virtual Observatories in Geosciences (VOIG) Conference 2007

10. Download Data from Different Websites Virtual Observatories in Geosciences (VOIG) Conference 2007

11. Check Data from Different Satellites for Coincidences April 2 UT Dec 18 UT Virtual Observatories in Geosciences (VOIG) Conference 2007

12. Metadata Limits the Searches • Typical metadata does not allow the level of search that our typical researcher would like to make. • Typical metadata includes: • descriptive metadata includes program, satellite and instrument information, as well as data details like observed phenomena or physical parameters measured, units of measure, time and spatial coverage (at a broad level only), processing level • structural metadata defines the details for data access and retrieval if selected by the use • Typical metadata does not include: • Time and location information at the pixel level Virtual Observatories in Geosciences (VOIG) Conference 2007

13. SGI Database VITMO Search Architecture VITMO VITMO Metadata Catalog ACE SuperDARN TIMED CDAWeb Virtual Observatories in Geosciences (VOIG) Conference 2007 VSTO

14. Solar Geophysical Indices Database • The TIMED program maintains a database of Solar Geophysical Indices that the ITM community decided were relevant. • VITMO queries the TIMED Oracle database directly pulling the SGI parameters of interest. • We then build windows when all conditions are met. Virtual Observatories in Geosciences (VOIG) Conference 2007

15. Coverlet Windows Event A 3<Kp<4 Event B Solar Wind Speed > 400 km/s Event C Bz<0 Resultant Timeline Virtual Observatories in Geosciences (VOIG) Conference 2007

16. What will VITMO search capability mean to users? • By taking on a more global view of data, VITMO can help the researcher understand the connectedness between the different data sets. • Example - storm studies - taking the Halloween 2003 superstorm as a case study. • During this event the following things happened: (1) 3 active regions (ARs) produced a total of at least 124 soft X-ray flares and more than 60 coronal mass ejections (CMEs) between Oct. 18 and Nov. 5, 2003. (2) 2 CMEs impacted Earth head-on producing huge geomagnetic storms. • Today – without VITMO: • A researcher may start with: LASCO images and ACE solar wind data over the broad time range in October and November 2003. If the researcher was interested in auroral impacts from the storm they may have looked for time periods when Kp was between 3 and 9 and Bz was negative. • They might have looked for Polar and GUVI imagery or auroral conductivity maps during those time periods. • All of this work would have involved finding and navigating numerous web sites that provide data, indices, and other information for the specific time period. • Now – with VITMO: • The researcher will be able to select the needed LASCO images and ACE solar wind data. • They will also be able to select auroral data during the time periods when Kp is between 3 and 9 and Bz was negative. • The VITMO, knowing that the researcher was interested in auroral data, would have automatically presented GUVI and Polar instrument products as appropriate. • It would then suggest data readers, appropriate for the data format, and tools that the researcher could use to work with or analyze that data. Virtual Observatories in Geosciences (VOIG) Conference 2007

17. Data Search Before VITMO Virtual Observatories in Geosciences (VOIG) Conference 2007

18. Select Coverage Virtual Observatories in Geosciences (VOIG) Conference 2007

19. Select Physical Parameters Virtual Observatories in Geosciences (VOIG) Conference 2007

20. Restrict Coverage Virtual Observatories in Geosciences (VOIG) Conference 2007

21. Select Products for Download Virtual Observatories in Geosciences (VOIG) Conference 2007

22. Coverage Windows Virtual Observatories in Geosciences (VOIG) Conference 2007

23. Files Are Packaged Up For Download Virtual Observatories in Geosciences (VOIG) Conference 2007

24. Applicability to ITM Research • Example – Using TIMED GUVI data the researcher has identified ring current aurora (dayside detached arcs, nightside detached aurora, proton spots, etc). • It is noted that in the cases examined certain geophysical conditions occur. • Search for all data (from other instruments as well) under those conditions to identify occurrence frequency. Virtual Observatories in Geosciences (VOIG) Conference 2007

25. Overcoming the Limits of Our Metadata GUVI 1356 Å Plasma bubbles Limb Scan 74o Inclination Cross-track scan perpendicular to orbit Disk Scan Equatorial arcs Virtual Observatories in Geosciences (VOIG) Conference 2007

26. Calculated Databases • VITMO will employ special coincidence calculators that know the viewing geometries of the various instruments. • The TIMED coincidence calculator is an example of a tool which when converted into a web service can be used to augment existing data querying. • The TIMED coincidence calculator has recently been imbedded into the TIMED query system. • These calculators will operate as web services inside the VITMO query system (or by outside query systems). • These calculators generate metadata for the coincidence of interest which is used by the catalog system to answer your query. • The metadata generated by this service is discarded after it is used – hence “virtual metadata”. Virtual Observatories in Geosciences (VOIG) Conference 2007

27. Not Plotting Tools – Imbedded Into Query Virtual Observatories in Geosciences (VOIG) Conference 2007

28. Coincidence Calculator Virtual Observatories in Geosciences (VOIG) Conference 2007

29. SGI Database Virtual Metadata Catalog (calculational services) VITMO Search Architecture VITMO VITMO Metadata Catalog SuperDARN TIMED CDAWeb Virtual Observatories in Geosciences (VOIG) Conference 2007 VSTO ACE

30. Enhanced Query • With the inclusion of virtual metadata sources a user can go to VITMO to ask “During the last 6 months where are GUVI EDP products over Millstone Hill radar when Kp > 3”. • The coincidence calculator will calculate when the GUVI product overlaps spatially and temporally with Millstone Hill. • This is combined with when Kp > 3. • Query is sent to VSTO or MADRIGAL to find appropriate radar or other products. • All data products returned for download. Virtual Observatories in Geosciences (VOIG) Conference 2007

31. Enhanced Search Results • Addition of virtual metadata capabilities allows pre-selection of data for analysis. • VITMO search will employ calculator services that allow us to deliver results from searches such as “What GUVI data that contains 23-01 MLT during active geomagnetic conditions?” Virtual Observatories in Geosciences (VOIG) Conference 2007

32. Products of Searches Are Windows • Products of searches are windows of time. • Future products of searches may be products of time and space. Virtual Observatories in Geosciences (VOIG) Conference 2007

33. Find Data During Events • Capability exists to search for data from a library of events • Solar flares • Geomagnetic storms • User generated conditions • Model – data comparisons • Identify where GUVI electron density profiles deviate from International Reference Ionosphere calculations in latitudes between ±30° mlat by more than a factor of two. • Now find all DMSP SESS ionospheric data at the same time and location. Virtual Observatories in Geosciences (VOIG) Conference 2007

34. Event Databases SGI Database Virtual Metadata Catalog (calculational services) VITMO Search Architecture VITMO VITMO Metadata Catalog SuperDARN TIMED CDAWeb Virtual Observatories in Geosciences (VOIG) Conference 2007 VSTO ACE

35. Conclusion • Addition of SGI databases, on-the-fly calculated databases (virtual metadata generators), and event listings greatly enhance the search capabilities far beyond the limitations of the present metadata available to describe products in our domain. • The VITMO prototype is available at HTTP://vitmo.jhuapl.edu. • VITMO will be operational at the end of 2007. Virtual Observatories in Geosciences (VOIG) Conference 2007