Repositories, Workspaces, Web Services - some ideas -

1. Repositories, Workspaces, Web Services - some ideas - Peter Wittenburg The Language Archive - Max Planck Institute CLARIN Research Infrastructure Nijmegen, The Netherlands

2. scope of workshop • clear focus on technology and architecture issues for preservation and access • many other issues not in focus although relevant • IPR, license issues only partially • quality of data & metadata • certification (RAC, DSA, etc) • AAI • cost aspects • etc. • let's have interactive presentations • should be able to extract essentials

3. Definitions?

4. so simple repository

5. + Metadata metadata registry - apple - 2010 - pear - 2010 - plum - 2010 dangerous since physical paths may change etc ? ? - orange - 2010 repository

6. + replication due to preservation metadata registry - apple - 2010 dangerous since metadata records can be re-used metadata should be stable - pear - 2010 ? - plum - 2010 ? - orange - 2010 repository repository transfer at physical level

7. + replication and PIDs PID registry metadata registry - appel - 2010 - PID4 - 2010 - pear - 2010 - PID3 - 2010 dangerous: another indirection layer - plum - 2010 - PID2 - 2010 - PID1 - URL1 - URL 2 access possible which rights? ? same access rights - orange - 2010 repository repository transfer at physical level

8. what about collections PID registry metadata registry - appel - 2010 - collection - 2010 - PIDx - URL - PID4 - 2010 - pear - 2010 - PID3 - 2010 - plum - 2010 - PID2 - 2010 PS: collections are dynamic - PID1 - URL1 - URL 2 - orange - 2010 repository repository transfer at physical level

9. topic of high relevance • ESFRI Task Force on Repositories (report) • e-IRG/ESFRI Task Force on Data Management (report) • Blue Ribbon Task Force on Sustainable Digital Preservation and Access (report) • EC High Level Expert Group on Scientific Data (report) • ASIS&T Summit Phoenix on Research Data and Access • (slides & summary) • T. Hey et al. The Fourth Paradigm: Data-Intensive Scientific Discovery (book)

10. summarizing the challenges • how to • manage the data Tsunami • maintain data visibility • preserve the data (just seen one solution) • protect the data integrity • ensure that we get the object we wanted • guarantee data authenticity (how to present) • maintain context and provenance information • protect privacy and rights in complex data world • maintain trust in data • federate repositories to (virtually) integrate data • achieve (partial) interoperability • exploit distributed data without copying

11. speaking about metadata harvesting Harvester Metadata Catalog Search Shared Catalog Online Catalog Data Mirror View Information on Data Through Catalog Link to Data at Partner Site Online Analysis Access Data With Extraction and Analysis, Through Catalog Direct to Partner Sites

12. speaking about architectures

13. speaking about federations

14. speaking about federations

15. speaking about federations

16. general configuration repository A - architecture - rights domain - access paths - etc. repository B - architecture - rights domain - access paths - etc. repository C - architecture - rights domain - access paths - etc. adapter(s) adapter(s) adapters adapters adapters adapters can be special does not scale mirror repository X - architecture - rights domain - access paths - etc. mirror repository Y - architecture - rights domain - access paths - etc. mirror repository Z - architecture - rights domain - access paths - etc.

17. general configuration repository A - architecture - rights domain - access paths - etc. repository B - architecture - rights domain - access paths - etc. repository C - architecture - rights domain - access paths - etc. API API API API API API replication layer mirror repository X - architecture - rights domain - access paths - etc. mirror repository Y - architecture - rights domain - access paths - etc. mirror repository Z - architecture - rights domain - access paths - etc.

18. generic HLEG figure Trust Data Curation Data generators Users User functionalities Data capture & transfer Virtual Research Environments Data discovery & navigation Workflow generation Annotation, Interpretability Community Support Services Safe & persistent storage Identifiers, Authenticity, Workflow execution, Mining Common Data Services

19. requirements for intermediate layer • needs to cope with large diversity of solutions and architectures • may only minimally interfere with local repository solutions • (too much has been invested along community traditions) • needs to respect rights domains and preserve access rights • needs to be transparent to proven utilization mechanisms • needs to operate at logical level (canonical collections) • needs to scale with number of (community) data centers • only one way to go: • separate functionality into independent components • (data, metadata, PIDs, etc) • specify proper interfaces (of course)

20. requirements for layer • how to manage procedures/workflows in complex landscape • how to assess quality and correctness of all workflows • how to maintain provenance information • only one way to go • make use of an easy-to-interpret declarative language • establish proper "policy rules on all levels" • map these rules to robust and proven activities • separate declarative language from interpretation engine • iRODS is an attempt in this direction • respect to Reagan Moore and his team • at MPI since some years such a declarative language to manage • access rights for the million objects which need to be treated individually and which are part of collections

21. Reagan's data environments • moving not bytes but collections • need to maintain integrity of collections (incl. relations) • collections are assembled for a certain purpose • collections have properties to ensure their purpose • policies ensure maintenance of properties • procedures implement policies • procedures result in state information • assessment step to validate state • purpose, properties, policies, procedures, state info

22. program - 1st part • Larry Lannom (CNRI): about a digital object architecture • Alex Wade (MS): approach from MS • Malte Dreyer: thoughts about generic API • John Kennedy: heterogeneity of repositories in DEISA • Ken Galluppi: federating several repositories • Willem Elbers: federation tests with iRODS • Jean-Yves Nief: iRODS in professional use • Peter and Johannes: summary + discussion

23. utilization challenge existing utilization software • utilization software may not be affected by replication • utilization software should also make use of copies • any replication solution needs to demonstrate this !!!!

24. work spaces and profiles • users want to • store data • protect data • share data • enrich data • change data • etc. • data is somewhere in • this complex domain • users want transparent • access • how to get this done? profiles attributes quotas etc

25. processing chains - specification data metadata registries tool metadata registries data operation data* operation workflow specification framework this is very discipline specific - various possibilities curation/annotation/enrichment/visualization pipelines, etc

26. processing chains - execution workflow execution framework

27. the challenges • large amounts of data is at mirroring repositories • let's execute operations on the mirroring sites • how to easily deploy operators • how to inform execution environment about invocation way • how to let them act on the user's behalf • etc

28. program - 2nd part • SARA colleagues: workspace in NL • Morris Riedel (FZJ): workspace ideas • Johannes & John (RZG): operational aspects • Thomas & Erhard (U Tübingen): WebLicht example • Mike Papazoglou (U Tilburg): generic SOA aspects • Peter: wrap up and discussion

29. thanks for the attention