Data Management at SOLEIL: A NeXus Approach

1. @ With TANGO S. Poirier – Data management group

2. Short introduction • Devices servers • GUIs

3. Introduction • SOLEIL requirements for data recording • A system able to record any kind of data • Generic (independent from beamlines and experiments types) • That can easely configurated (even by users!) • The SOLEIL choice for data format: NeXus • Based on HDF (Hierarchical Data Format) and, recently, XML • Why ? • Logical organisation rather than physical organisation • Ability to store any kind of data • Self-describing format • Efficiency (binary, compression…) • Used in other facilities: DIAMOND, APS, LLB, ISIS, SNS, PSI, ESRF?

4. Introduction • a NeXus file is a tree structure, like a filesystem. • Datasets are recorded inside typed data groups (NXentry, NXsample, NXdata, NXsource, …) • Experimental data is recorded in NXdata groups • Metadata is recordedIn the other groups • Big datasets may becompressed In this sample, you show the data for 3 experiments

5. Devices servers • DataRecorder • Writes experimental data and associated metadata in NeXus files • AuthServer • Users enter login, password & project code. Then the ds generates a ‘key’ allowing DataRecorder to write data for a particular project in the right place in the storage facility. • NeXusReader • This device reads NeXus files and expose datasets, on demand, as dynamics attributes.

6. Devices servers • TechnicalData • Device that store two lists of ds containing data about the technical environment: • The DataRecorder reads theses lists and records the corresponding data at the begining and at the end of a recording session (i.e for a NXentry group) • SampleData • sample data collector device. The aim of this ds is to provide a unified interface to sample information.

7. DataRecorder : how it works ? • It’s based a on shared library: libDataStorage (written in C++). • This library don’t know Tango • But it knows how to record data in the NeXus format • Experimental data is collected through a plug-ins mechanism: data collectors • 2 collectors at this moment: libTangoCollect.so & libFileCollect.so • A tool can be used to export data from NeXus file into ASCII or binary based format (e.g. EDF) for users convenience

8. Data production diagram • From the DataRecorder point of view Data collectors

9. GUIs • Bolero • Edition of the devices servers recording configuration

10. GUIs • TechnicalData DS lists edition:

11. GUIs • Data Storage Control Center • Manage and monitor storage devices

12. GUIs • Baladi • A graphical tool to explore Nexus files

13. The TWIST project • Web portal for experimental data retreiving • Written in java and use JNLP technology • This not a pure TANGO project but, in order to allow browsing inside NeXus files, client app uses ATK components and exchange data with a NeXusReader device

14. Other software tools • NXextractor: a tool able to export data from a NeXus file into a file in a arbitrary format (ASCII or binary). It uses scripts to describe theses formats. • IGOR extension (XOP) for reading NeXus files • Next steps: adapt post-process software. To be planned.

15. The end ?