Building a VxO for Heliophysics

1. Building a VxO for Heliophysics Bob Bentley (UCL-MSSL) 21 June 2007 IHY General Assembly, Torino, Italy

2. Heliophysics • Heliophysics explores the Sun-Solar System Connection • An generalization of the study of space weather (SWx++) • Heliophysics sits in the boundary between two communities • Astrophysics • Study of solar phenomena helps understanding of stellar observations • Planetary sciences (including Earth sciences) • Solar activity can influence the environment on/around the planets • The discipline must be aware of the need to support the interests of both communities • A virtual observatory that supports Heliophysics must facilitate access to data from a number of communities • As such, it is in essence a next generation VxO • Proposal submitted under EC’s FP7 • Heliophysics Integrated Observatory (HELIO)

3. Heliophysics • The communities that constitute heliophysics have evolved independently over decades and even centuries • Communities include solar, heliospheric, magnetospheric and (possibly) ionospheric physics • Sharing ionospheric physics with planetary sciences gives overlap • Little or no coordination of how each has evolved • Each has very different ways of describing, storing and exploiting the data from their observations, varying use of standards • The desire to solve science problems that span disciplinary boundaries is driving the need to provide combined access to these data • To do this, we need to find ways to: • Tie the data together through searches across all the domains • Present any results in a form that does not require a detailed understanding of each discipline • This requires the re-evaluation of the capabilities provided within each community and some corrective action

4. Types of metadata • To facilitate a VxO for heliophysics we need to examine the metadata that is required. • There are many ways metadata can be grouped, one is: • Search metadata • Metadata used to identify time intervals and sets of data of interest • Observational metadata • Metadata used to describes the observations, e.g. FITS headers • Storage metadata • Metadata that describes how the data are stored and accessed • Administrative metadata • Metadata that allows the system to exploit the available resources • The rest of the talk will discuss issues related to these

5. Searches • In heliophysics, we are interested in how an event on or near the solar surface can propagate through the heliosphere and affect a planetary environment • May also want to work backwards and look for the cause of an effect – what solar event caused this ionospheric activity… • Searches should identify interesting time intervals based on a combination of event, features, etc. metadata • Light curves and images my also be used to augment the search • Each community of some combination of these metadata • There are concerns about the quality and integrity of these metadata and whether it is adequate to support the searches we would like to undertake

6. Solar search metadata • Searches in Heliophysics are mainly event driven • Phenomena occur on or near the solar surface • Event data gives time and location of phenomena • Feature data provides details of location and size of structures that may be relevant • Time information can be expressed in many ways • Essentially these are the same, with simple transformations • Spatial information can be expressed in terms of: • Coordinates in the observing frame – e.g. arcsecs from disc centre • Coordinates on the rotating body of the Sun – Carrington coords. • The position of the observer was ignored for the most part • Helio-seismology is an exception • In the bigger picture of heliophysics, also need to include the viewing perspective (c.f. STEREO)

7. Other search data • Observation of phenomena in the heliosphere and near/on planets are more complex • For in-situ observations in the heliosphere • Time is when a phenomena affected (passed) the observer • Position of the observer relative to the Sun is key to understanding • When the in-situ observations are made on/near a planet • Position of the observer relative to the planet is also important • Relating events that are defined from in-situ data to those on/near the Sun requires an understanding of how events propagate • Details of the velocity structure of CMEs and the solar wind are not easy to determine… • HELIO plans to develop a tool that will use even/feature data to refine a model to trace effects forwards from causes, etc.

8. Simple, but not so simple • In principle this all seems fairly obvious, but lets look in detail at some common solar event data • On 20 January 2005 there was an X7.1 flare that was intensely geo-effective. • The flare was associated with particle event and a CME; it was also observed by ground-level neutron monitors – a GLE. • Many superlatives were used to describe the event • "The solar energetic particle event of January 20 2005 has been called, by some measures, the most intense in 15 years..." (Mewaldt et al., 2005) • ”The fastest rising SEP event of current cycle [cycle 23]" (Rawat et al., 2006) • ”The most spectacular [solar event] of the Space Age" (Tylka et al., 2006) • ”The largest GLE [GLE 69] in half a century" (Bartol Research Institute) • But event is absent from the NOAA SEC list of "Solar Proton Events Affecting the Earth Environment" • When you look at the data and how lists are created, you realize that the lists are deficient in several ways • Humans and SmFCACs can understand what happened, but • It is harder for machines...

9. X7.1 of 20 Jan 2005 • The event was one of several from AR 10720 • Two other X class flares and several M class flares occurred in previous 3 days; others before this

10. X7.1 of 20 Jan 2005

11. X7.1 of 20 Jan 2005 • At the time of the event, the proton levels had not returned to normal after previous events • The criteria fails to recognize a new event • NOAA lists event on 16 Jan • The X-ray data also suffers from problems • The end of an event is defined by when the counts drop to 50% • New events can “interrupt” existing events • The shape and true duration of the decay phase are lost • NOAA gives start 0636; end 0726 • Not all locations are tagged!! • Significant brightenings seen on images not declared as flares

12. Some of the problems • That major events can be “missed” is worrying and makes automated searches difficult • A search for long duration events would yield spurious results • Since the locations of all flares are not known, it is impossible to know if they will be geo-effective • Instrument flare lists have gaps – nights, off times, etc. – but the reason for a null result is not included

13. Improving event data • Existing flare lists can give a distorted picture what has occurred • Such deficiencies make it difficult for non-experts to use them • The community “knowledge” is not written down • Need to re-evaluate and regenerate the event data in all domains with the idea that they will be used in a joint search across the domains • Ensure events more accurately described • Include information that might explain null results

14. Observation metadata • Metadata related to observations gives information about how the observation was made, etc. • Often quality issues related to the metadata that is provided • Parameters sometimes missing, or wrong • Inconsistent use of information, “synonyms” for keywords • In solar data, space-based observations much better described than their ground-based counterparts • Ground-based observations are only source in some wavelengths and need access to as many observatories as possible • If ionospheric included, good coverage of ground stations key • Researchers often used to deficiencies in their own domain • Difficult for machines to handle if it is not quantified properly • VxOs can sometimes develop ways of patching the gaps • What do we do with this information? How is it shared?

15. Observational metadata • In solar physics we have developed suggested standards • SOHO made good start with their keyword document • EGSO enhanced concepts with its data model • Situation better than it was but adoption still not universal • Even some problems within SOHO… • At the Virtual Observatories in Geoscience (VOiG) Conference we discussed whether a standard for observational metadata that span wide set of domains is possible • Core part that is required – (time, location, etc.) • Additional information that may be specific to a domain • Other information that the instrument team wants to add • Not practical to have metadata same for all domains • HELIO proposes to use semantics to develop a data model that bridges the data models of the domains

16. Storage Metadata • How data are stored in a data source can make a lot of difference to their accessibility • EGSO has the concept of resource-rich and resource-poor providers • Resource-rich providers – e.g. SDAC – should provide what is needed in response to simple query • Resource-poor providers may only be able to made data accessible over Internet • At VOiG we discussed whether we should provide guidelines on ways that data could be organized for resource-poor providers • Existing VxOs have opinions on what is easy to use • Providing data following simple naming conventions in an ordered directory structure would make them simpler to access • Simple textual catalogue might provide additional information

17. Sources available through EGSO

18. Ground based observatories need to make their data more available and in more useful formats (e.g. FITS) and better metadata

19. Standards • Need to extend the use of standards to all domains so that all future data are more compatible with the VxOs • The situation has changed with enhancements to technology; providers need to ensure they are more compliant • VxO will need to handle problems with the older data; providers cannot be expected to do it • Standards need to be developed in collaboration with the community and funding agencies • In developing standards need to draw on the experience gained within the general (VO) community and adopt the best ideas available • eGY is a good forum to engender discussion on these topics!!

20. Conclusions • Developing a virtual observatory to support heliophysics will not be simple, but we have the technology • Cooperation of the community essential if we are to succeed • Some of the possible problems have been highlighted • To address them, we need to engage the communities in all the domains that constitute heliophysics and develop standards that will facilitate the process • In principle no reason why all ground-based observatories should not be included – if they want to be…

22. Outline • Examine some of the items needed to facilitate a virtual observatory for heliophysics • Identify deficiencies and suggest solutions • Talk evolved as I was preparing it… • Started looking at proton events and GLEs because of space weather effects related to aviation • Found problems with the metadata