1 / 48

Building and Managing an Open Grid

Building and Managing an Open Grid. Ian Foster Argonne National Laboratory University of Chicago http://www.mcs.anl.gov/~foster. Keynote talk at 8 th IFIP/IEEE Intl Symp. on Integrated Network Management, March 26, 2003. Grids and Integrated Network Management.

awena
Télécharger la présentation

Building and Managing an Open Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Building and Managingan Open Grid Ian Foster Argonne National Laboratory University of Chicago http://www.mcs.anl.gov/~foster Keynote talk at 8th IFIP/IEEE Intl Symp. on Integrated Network Management, March 26, 2003

  2. Grids andIntegrated Network Management • 2003: Production use in science & industry for distributed resource & service integration • Grids as management problem: definitely • Grid technologies as management solution? • 1995: I-WAY • Integration of HPC resources for advanced scientific applications • Management not quite on the agenda

  3. Overview • The Grid: why and what • Building an open Grid • Open standards • Open source • Open infrastructure • Summary

  4. The Grid:On-Demand Access to Electricity Quality, economies of scale Time

  5. By Analogy, A Computing Grid • Decouple production and consumption • Enable on-demand access • Achieve economies of scale • Enhance consumer flexibility • Enable new devices • On a variety of scales • Department • Campus • Enterprise • Internet

  6. Not Exactly a New Idea … • “The time-sharing computer system can unite a group of investigators …. one can conceive of such a facility as an … intellectual public utility.” • Fernando Corbato and Robert Fano , 1966 • “We will perhaps see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country.” • Len Kleinrock, 1967

  7. But, Things are Different Now • Networks are far faster (and cheaper) • Faster than computer backplanes • “Computing” is very different than pre-Net • Our “computers” have already disintegrated • E-commerce increases size of demand peaks • Entirely new applications & social structures • We’ve learned a few things about software

  8. Computing isn’t Really Like Electricity • I import electricity but must export data • “Computing” is not interchangeable but highly heterogeneous: data, sensors, services, … • This complicates things; but also means that the sum can be greater than the parts • Real opportunity: Construct new capabilities dynamically from distributed services • Raises three fundamental questions • Can I really achieve economies of scale? • Can I achieve QoS across distributed services? • Can I identify apps that exploit synergies?

  9. Why the Grid?(1) Revolution in Science • Pre-Internet • Theorize &/or experiment, aloneor in small teams; publish paper • Post-Internet • Construct and mine large databases of observational or simulation data • Develop simulations & analyses • Access specialized devices remotely • Exchange information within distributed multidisciplinary teams

  10. Why the Grid?(2) Revolution in Business • Pre-Internet • Central data processing facility • Post-Internet • Enterprise computing is highly distributed, heterogeneous, inter-enterprise (B2B) • Business processes increasingly computing- & data-rich • Outsourcing becomes feasible => service providers of various sorts

  11. New OpportunitiesDemand New Technology “Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”

  12. Building an Open Grid

  13. Building an Open Grid Open Standards

  14. Building an Open Grid Open Standards Open Source

  15. Building an Open Grid Open Standards Open Source Open Infrastructure

  16. Building an Open Grid Open Standards Open Grid Open Source Open Infrastructure

  17. Building an Open Grid Open Standards Open Grid Open Source Open Infrastructure

  18. Open Grid Services Arch Web services GGF: OGSI, … (+ OASIS, W3C) Multiple implementations, including Globus Toolkit X.509, LDAP, FTP, … Globus Toolkit Defacto standards GGF: GridFTP, GSI Grids and Open Standards App-specific Services Increased functionality, standardization Custom solutions Time

  19. Open Grid Services Architecture • Service-oriented architecture • Key to virtualization, discovery, composition, local-remote transparency • Leverage industry standards • Internet, Web services • Distributed service management • A “component model for Web services” • A framework for the definition of composable, interoperable services “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002

  20. Realizing a Service-Oriented Architecture: How Do I • Create, name, manage, discover services? • Render resources, data, sensors as services? • Negotiate service level agreements? • Express & negotiate policy? • Organize & manage service collections? • Establish identity, negotiate authentication? • Manage VO membership & communication? • Compose services efficiently? • Achieve interoperability?

  21. Web Services • XML-based distributed computing technology • Web service = a server process that exposes typed ports to the network • Described by the Web Services Definition Language, an XML document that contains • Type of message(s) the service understands & types of responses & exceptions it returns • “Methods” bound together as “port types” • Port types bound to protocols as “ports” • A WSDL document completely defines a service and how to access it

  22. Client • Introspection: • What port types? • What policy? • What state? GridService (required) Other standard interfaces: factory, notification, collections Grid Service Handle Service data element Service data element Service data element handle resolution Grid Service Reference Open Grid Services Infrastructure • Lifetime management • Explicit destruction • Soft-state lifetime Data access Implementation Hosting environment/runtime (“C”, J2EE, .NET, …)

  23. Open Grid Services Infrastructure GWD-R (draft-ggf-ogsi- gridservice-23) Editors: Open Grid Services Infrastructure (OGSI) S. Tuecke, ANL http://www.ggf.org/ogsi-wg K. Czajkowski, USC/ISI I. Foster, ANL J. Frey, IBM S. Graham, IBM C. Kesselman, USC/ISI D. Snelling, Fujitsu Labs P. Vanderbilt, NASA February 17, 2003 Open Grid Services Infrastructure (OGSI)

  24. More specialized & domain-specific services Domain- specific profiles Other models OGSA Platform OGSA Platform services: registry, authorization, monitoring, data access, etc., etc. Models for resources & other entities Environment- specific profiles Host. Env. & Protocol Bindings Hosting Environment Transport Hosting Environment Protocol The OGSA Platform OGSI GWD-R (draft-ggf-ogsa-platform-3) Editors: Open Grid Services Architecture Platform I. Foster, Argonne & U.Chicago http://www.ggf.org/ogsa-wg D. Gannon, Indiana U.

  25. Client Client Client Request and manage file transfer operations Notf’n Source Policy Grid Service Fault Monitor Pending interfaces Query &/or subscribe to service data Performance service data elements Policy Perf. Monitor Faults Example:Reliable File Transfer Service File Transfer Internal State Data transfer operations

  26. Open Grid Service Architecture:Next Steps • Technical specifications • Open Grid Services Infrastructure is complete • Security, data access, Java binding, common resource models, etc., etc., in the pipeline • Implementations and compliant products • OGSA-based Globus Toolkit v3, pyGlobus, … • IBM, Avaki, Platform, Sun, NEC, Oracle, … • Rich set of service defns & implementations • Time to start on OGSI-compliant services!

  27. OGSA Manageability • Management is a high-priority OGSA effort • GGF Common Management Model (CMM) WG • Ellen Stokes (IBM) co-chair • Goal: • Define standard schema and interfaces for a manageable resource (modeled as a service) • And also: • Allow existing models to be used & exposed • Leverage CIM schema when applicable • Define how CMM interfaces can integrate with higher-level interfaces (e.g., provisioning)

  28. Common Management Model • A manageable resource is a Grid service, thus • Global resource names: Grid service handles • State data modeling + access: SDEs • Lifetime management • Service Group for grouping resources • Interface definition language: WSDL • Plus additional schema & operations • Standard manageable resource SDE schema • Interfaces for extensible lifecycle and relationship management • BaseManageableResource interface

  29. Base Manageable Port Types OGSA port types GridService HandleResolver ServiceGroup Locate Relationship LifecycleModel CRM port types BaseManageableResource

  30. CMM Schema • WSDL (open content model) & XSD describe resource’s manageable attributes (as SDEs) • Models are CIM-based where applicable • Re-factored for service efficiency; appropriate for higher-level management applications • Additional XML attributes: change control, measuring, lifecycle • Additional XML data types • CIM as basis for schemas: but may modify • E.g. use constructs from XML/XSD where similar ones exist in CIM

  31. “Use CIM Models Where Applicable” • Class is port type, properties of class are port type service data, methods of class are port type operations • Some refactoring of classes -> portTypes • Express in WSDL/GSDL as grid service • Managed resource port type from which other resource port types are derived • Mix in the base GridService port type • Mix in other CMM port types as needed (Identity, relationship, lifecycleState)

  32. CMM Bindings and Operations • A resource service may have more than one binding to access that resource’s manageability information • A resource may support more than one service implementation • e.g., Native, SNMP, CIM, RMC, etc • A resource can still be accessed (outside OGSA) using its existing implementation, resource definition, and access interface, e.g., SNMP, CIM, etc.

  33. Basic Manageability Port Types • Needed to manage the resource • lifecycleModel • relationship • locate • Others defined • From CIM schema, e.g., opererational operations (start/stop/resume/pause) • By OGSO spec, e.g. grid service, notification • By other OGSA components, e.g. policy, logging/metering

  34. lifecycleModel Port Type:A Container for Lifecycle States • There may be multiple models, but only one for a given resource’s port type • Example: Get/set resource’s lifecycle state • down, starting, up, stopping, failed • Each state has additional info, e.g., • up state: idle, busy, degraded Down Starting Failed Stopping Up

  35. Relationships & Dependencies • Relationships describe existence and type of connections between resource instances • Relationships are discovered through the relationship port type and its relatedResource service data element • Relationship port type allows a view of relationships as they are known by the resources at each end of the relationship • Set of predefined relationship types • Hosts, Contains, Federates, Aggregates, Uses, Implements

  36. Locate Port Type • Provided to help in the location of manageable resources of a given type • Contains two service data elements • serviceGroupType, port type that provides the ServiceGroup function for manageable resources of this type • searchProperty, zero or more service data elements that are likely to be used for searching for a manageable resource, and are thus worth caching in the manageable resource registry

  37. Building an Open Grid Open Standards Open Grid Open Source Open Infrastructure

  38. Open Source • Encourage adoption of standards by reducing barriers to entry • Overcome new technology Catch-22 • Enable broad Grid technology ecosystem • Key to international cooperation • Key to science-commerce partnership • Jumpstart Grid industry and allow vendors to focus on value-add • E.g., IBM, Avaki use GT3; Platform Globus • Open source is industry friendly!

  39. Globus Toolkit v3 (GT3)Open Source OGSA Technology • Implement core OGSI interfaces • Support primary GT2 interfaces • High degree of backward compatibility • Multiple platforms & hosting environments • J2EE, Java, C, .NET, Python • New services • SLA negotiation (GRAM-2), registry, replica location, community authorization, data, … • Growing external contributions & adoption

  40. Globus Toolkit Contributors: GT2 • Grid Packaging Technology (GPT) NCSA • Persistent GRAM Jobmanager Condor • GSI/Kerberos interchangeability Sandia • Documentation NASA, NCSA • Ports IBM, HP, Sun, SDSC, … • MDS stress testing EU DataGrid • Support IBM, Platform, UK eScience • Testing and patches Many! • Interoperable tools Many! • $$ DARPA, DOE, NSF, NASA, Microsoft, EU

  41. Globus Toolkit Contributors: GT3 • Replica location service EU DataGrid • Python hosting environment LBNL • Data access & integration UK eScience • Data mediation services SDSC • Tooling, Xindice, JMS IBM • ... • ... • ...

  42. Open Source is More than Software! • Success of Grid concept demands effective community mechanisms for coordinated • R&D for core technologies • Testing, packaging, documentation • Support and training of user communities • All three must become collaborative activities • Based on standards • Centered on a common code base • Supported by appropriate tools • United by appropriate processes & governance

  43. For Example:Distributed Test Facility (Proposed activity for NSF GRIDS Center: grids-center.org)

  44. Building an Open Grid Open Standards Open Grid Open Source Open Infrastructure

  45. The Gridas Enabler of 21st Century Science • Entirely new approaches to enquiry based on • Deep analysis of huge quantities of data • Interdisciplinary collaboration • Large-scale simulation • Smart instrumentation • Enabled by an infrastructure that enables access to, and integration of, resources & services without regard for location

  46. Grid Infrastructure • Broadly deployed services in support of fundamental collaborative activities • Formation & operation of virtual organizations • Authentication, authorization, discovery, … • Services, software, and policies enabling on-demand access to critical resources • Computers, databases, networks, storage, software services,… • Operational support for 24x7 availability • Integration with campus and commercial infrastructures

  47. Edinburgh Glasgow DL Newcastle Belfast Manchester Cambridge Oxford Hinxton RAL Cardiff London Soton Tier0/1 facility Tier2 facility Tier3 facility 10 Gbps link 2.5 Gbps link 622 Mbps link Other link The Foundations are Being Laid

  48. Summary • An open Grid technology has been created • Standards, software, infrastructure • All three must be collaborative activities • We are now scaling this technology base, to • Meet the needs of the science community • Achieve synergy between open & commercial, scientific & industrial contributors • Management is a crucial part of this effort • One might argue, “Grid = management” • We need help from your community

More Related