230 likes | 325 Vues
EGI Middleware Function. Lud ě k Matyska and Mirco Mazzucato Based on input from MW TF members
E N D
EGI Middleware Function Luděk Matyska and Mirco Mazzucato Based on input from MW TF members Alistair Dunlop (UK), Achim Streit (DE), Farid Ould-Saada (NO), Mirco Mazzucato (IT), Ignacio M. Llorente (ES), Uwe Schwiegelshohn (DE), Ludek Matyska (CZ), Ian Bird (CERN) , Christoph Witzig (CH), Sergio Andreozzi (IT) Ludek.Matyska@cesnet.cz
Disclaimer • These slides have been drafted based on the current version of the EGI Blueprint proposal (v1.2, 25 June 2008). • The purpose of this blueprint is to assess a possible model for the future sustainable grid infrastructure in Europe. • The aim of these slides is to support the discussions in the EGI Geneva Workshop (30 June 2008). • The EGI_DS project will modify this draft blueprint on the basis of feedback provided. www.eu-egi.org
Requirements Context • A vision of large high quality pan-European distributed compute and data infrastructure that • Has no internal barriers • Minimizes operational load • Provides guaranteed services • Is dependable, users can rely on its long term existence • Therefore must run on a high quality stable middleware www.eu-egi.org
Transition Context • EGI Grid must not disrupt the successful basic grids in operation today • Must provide a transition path for everybody interested • Must show its large benefits to help NGIs to convince existing grid users to move to the EGI Grid and also to attract new users • Must preserve the large investments and technical achievements done so far • Therefore a non disruptive approach is needed www.eu-egi.org
Current Situation in Europe • EGEE Grid as the largest World Grid infrastructure • gLite based, strong hierarchical model • Guarantee stable and high quality services • NorduGrid • ARC based, many commonalities with EGEE, more flexible model • DEISA as the supercomputing Grid • UNICORE based, less overlap with ARC/gLite • Smaller Grid islands • Mostly Globus and Condor based • Default choice for general grid R&D publications and prototypes • Some serving specific application communities • Includes also activities of some European countries • Commercial products • Usually single components, basic functions, limited use in academic world www.eu-egi.org
Middleware: Basic Options • A shared large scale distributed infrastructure could be theoretically achieved • Interconnecting “free” individual Grid islands – however no proofs that scalability and quality problem can be solved – can not remove all barriers • Accepting a common middleware certified solution – no barriers, quality and scalability at the price of less flexibility, some danger of middleware o(g)ssification – competition limited by available funds • The proposal values “no barriers” higher since this is what all successful e-Infrastructures have adopted (EGEE/DEISA/Nordugrid/OSG..) – trying to identify risks and providing a process for their minimization www.eu-egi.org
The Middleware Problem • Smooth transition • keep what we have • Barrier less Grid • move to a common certified solution (doesn’t mean a single implementation of each component) • High quality • allow maximal competition compatible with budget • Stability and dependability of services • Require to be conservative, do not change/replace too often the selected deployed components and interfaces – accept innovation rate allowed by budget www.eu-egi.org
Proposes Approach • Define a common middleware solution • The Universal Middleware Toolkit (UMT) • Define processes to provide stable solution • Middleware development function as part of EGI • Common policies, rules, quality and standard compliance criteria for UMT components • Define processes to go for highest quality • Service/Component orientation, interface definition and standardization • Testing and verification process under EGI control • Define processes to satisfy users/NGI requirements • Integrated development-deployment-feedback cycle www.eu-egi.org
Universal Middleware Toolkit • A common general component of the EGI infrastructure • A set of highly integrated and interoperable components and services • But initially not unique implementation – leave time to evaluate-generalize-specialize • Based on standards • Their further definition and evolution an integral part of the UMT evolution • A glue that connects the EGI Grid together guaranteeing to NGIs quality and scalability in sharing their resources www.eu-egi.org
UMT Initialization • UMT cannot be defined anew, from scratch • This would violate the smooth transition requirement • Can only be based on existing proved solutions for large scale Grids • gLite, ARC and UNICORE (in Europe) • The mentioned middleware provides a solid foundation for UMT • Initial component selection (next slide) www.eu-egi.org
Initial UMT Component Selection • Definition of specification/criteria should start almost immediately • After the UMT idea is accepted and endorsed • A role for specific task force • Goal to have a list of basic/core services this year • Establish working relation with DEISA/PRACE • Pragmatically will be based on gLite/ARC • Includes a first set of interface definition (“UMT standards”) involving UNICORE • Must also help to define the vertical interfaces (“hooks”) www.eu-egi.org
UMT Evolution • Further extension and improvement of interface definitions (up to standardization) • EGI core middleware function • Testing and certification • New component submission process open to everybody • Testing primary a responsibility of developers • Accepted components must receive a funding by EGI for their maintenance and eventual further evolution • The process includes components removal (out-phasing), too www.eu-egi.org
UMT Further Properties • A path toward “Grid IP” – a common set of necessary standardized services • Provides “vertical” interfaces for independent development of thematic higher services • Such development in general out of scope of UMT/EGI (but not EC, on the contrary!) • Source of new ideas and approaches to UMT • Novel use (through higher services) provides feedback to UMT and improves component’s quality • UMT as a core part of a Grid software ecosystem www.eu-egi.org
EGI.org Core Middleware Function • EGI.org Development Unit • Not doing actual development, but responsible for the UMT evolution process • A place where UMT evolution will be planned and coordinated – real work outsourced • 8 FTEs • Chaired by EGI Chief Technical Officer (CTO) • Will help to put in a close loop UMT component developers, Operations and Users (Applications) • The only known way to guarantee a successful evolution of a moving target like the Grid middleware www.eu-egi.org
Development Unit Objectives • Common baseline architecture; • Full interoperability of existing services through standardization; • Validation and final testing of the released services; • Increasing complementarities and specializations of the included services (“Grid IP”); • Adoption of application and operation requirements; • Convergence and interoperability through the implementation of standard interfaces with Globus, Condor and other non-EU stacks; • Definition of additional APIs that will allow independent development of higher level services. www.eu-egi.org
Conditions for Component Inclusion • Interoperability • Guaranteeing the barrier less EGI Grid • Completeness • No essential service is missing • Scalability • Guaranteeing the large scale EGI Grid • Simplicity • Easy to deploy and manage • Extensibility • A base for independent development of higher services (or services with limited user base) www.eu-egi.org
Interoperability • Other middleware solutions will continue to exists and be developed • UMT will provide both the vertical (“classical”) and horizontal (“hooks”) interoperability • UMT will actively support independent development of other/higher services • Providing horizontal interoperability through interfaces (“hooks”) for external developers • Both for VO/application based development and for (EU) Computer Scientists to use UMT as the reference base www.eu-egi.org
UMT Development • Not direct part of the EGI.org • However, EGI must guarantee that the development is not stalled • And EGI must also guarantee that it has proper tools to steer the UMT development process • For start, based on the work of the European Middleware Consortia (gLite, ARC, UNICORE) • Include progressively other contributions satisfying quality criteria and scalability if required by NGIs and compatible with the UMT architecture • A gradual move towards a UMT community of development teams responsible for the selected components • UMT should become a world reference www.eu-egi.org
Funding – General Principles • Budget as a tool to steer the UMT development process • Currently, the Middleware Consortia are usually funded on 50% through the developer’s institutions and remaining 50% a combination of national and EC funding • We propose • To ask EC to provide the 50% of the UMT development cost as an expression of interest in support of the necessary level of innovation • To keep the 50% co-funding of developer’s institutions • To have EGI.org controlling of the budget www.eu-egi.org
Funding – More Details • The certification and testing will remain a mandatory service of EGI.org • However, a large fraction of these activities can be outsourced to entities submitting components for testing and certification • EGI.org will provide the final certification • The actual support and maintenance of selected components is a service that can gradually be moved to service charge model as these get more consolidated and used • The UMT services/components will be mandatory for NGIs • Alternative (higher level) components can be accepted, serving only a subset of NGIs – these can be charged directly www.eu-egi.org
Estimated Middleware Development Efforts • Numbers are based on the input provided by current consortia on their total development efforts • Provides a rough estimate of the real development and software hardening (quality) efforts • It is a man power consuming task • Even as currently sometimes hidden in other activities, it must be accounted for • Includes UMT conformance testing • Does not include development of tools for operation automation • gLite: 61 FTEs • All current components • ARC: 29–45 FTEs • The higher number for all components, the smaller for components suggested to become part of UMT • UNICORE: 12 FTEs • All components • EGI.org standardization, certification …: 8 FTEs • Only this effort goes directly with EGI.org www.eu-egi.org
Summary • Middleware considered an essential part of EGI • Middleware existence and further development is of utmost importance for EGI Grid – funds should be assured • UMT as a set of EGI middleware components and services • Service/Interface standardization • Open component inclusion process • Responsibility for support and maintenance of selected components • Development costs shared with developers, expected EC contribution to cover the second half • EGI.org in appropriate control www.eu-egi.org
Some Items for Discussion • The UMT general concept • The UMT initialization/start up • Component selection process • Relationship to independent middleware R&D both in Europe and outside • Industrial involvement • Validation and Certification of new components – necessary tools • Budgetary requirements and budget control www.eu-egi.org