WP19 DESY Development Plan

1. WP19 DESY Development Plan Frank Schlünzen Jürgen Starek

2. Objectstores - Motivation • Data catalogues (DC) and digital objects (DO) arelooselycoupled • ACL changes in DC are not propagatedto DO (andviceversa) • Nomessagingbetween DC and DO • Removal, corruption, update of a DO doesn‘tresult in DC notification • Norights-management on filesystemlayer • All filesownedby super-users • Needs totrustapplicationsforrightsmanagement • Synch • Status of DC and DO caneasilyget out ofsync • Metadata in DC and DO caneasilyget out ofsync • ACLs in DC and DO areprobablynever in synch • UIDs insideand outside of DC needsynch

3. Objectstores - Motivation • Objectstoreshavethe potential tosolvea numberofthatissues • Particularlyinterestingsystems (forus) • dCache • isclosetobeinga fullobjectstoreimplementation • Extendableforindexed, efficientlysearchable user-defined meta-dataforanyobject • Full, scalabletape backend • CEPH • veryinterestingsystemwithpotentially high efficiencyforsmallfiles • mightoffergoodperformance in combinationwithenhanced meta-dataandaclhandling • naturallychoiceforclouds (open nebula) • Mature: in linuxkernel

4. Objectstores - Proposal In order to supplement the existing in-house knowledge about the properties of different file systems and storage hardware solutions, we propose to survey existing and planned object store solutions, evaluating possible performance or maintenance advantages over classical POSIX file systems. • Survey Topics • Which Object Store systems are available as commercial products or free software, and which similar systems exist or are planned? • What methods to access data in these object stores exist? • Are there systems that go beyond POSIX I/O and POSIX rights management? How do they present these improved semantics to the user? • What are common designs and best practices in designing and operating object stores? • What is the I/O performance of these systems when storing differently-sized objects? • What is the best performing hardware setup for these systems? • Is there a quality-of-service-mechanism for data streams? Is there an I/O scheduler? • What metadata are supported by the systems?

5. Virtual Analysis Plattform Remote access to compute resources for remote data analysis in a fully interactive and graphical environment is a common requirement at EuroFEL RIs. • Currently, this is realized through rather inconvenient and poorly scalable mechanisms. • Data protection and encapsulation of the environment is an issue. • We aim to evaluate some in-house available virtual/cloud resources and to set-up a prototype system for photon science applications: • provide a virtual host/environment tailored to a specific experiment and application, with a pre-defined software stack and data. • The user should have seamless access to his and only his data. • We will investigate possibilities to provide data access through different path-ways (e.g. ICAT, dCache, object store). • Possibilities to connect IdPs will be investigated as well.

6. (p)HDF5 andFhGFS Visualization of individual slices of images within a HDF-container archived in ICAT is a common user requirement. Virtual appliances are one way to visualize data remotely, alternatively could be realized as web services: • Investigate h5ws, a globus based web service • Investigate paraview which offers webGL, vrml, js representations • if feasible – implement a prototype solution. Many photon-science applications are i/o rather than cpu-limited. pHDF5 is one way to accelerate parallel read/write access to HDF5-container. Currently a number of photon-science labs are supporting development of HDF5-accellerators by HDF.org. • test upcoming (p)HDF5 implementations on available parallel filesystems, in particular FhGFS. • realize a non-persistent FhGFS in distributed memory and test performance and stability of such a system. • Test FhGFS capabilities to create filesystems on demand and entirely in user space