Introduction to Grid Computing

1. Introduction to Grid Computing

2. Introduction to Grid Computing • The term Grid comes from an analogy to the Electric Grid. • Pervasive access to power. • Similarly, Grid will provide pervasive, consistent, and inexpensive access to advanced computational resources. • Grid computing is all about achieving greater performance and throughput by pooling resources on a local, national, or international level.

3. 2100 2100 2100 2100 2100 2100 2100 2100 2100 Scalable Computing PERFORMANCE + Q o S Administrative Barriers • Individual • Group • Department • Campus • State • National • Globe Inter Planet Grid Personal Device Local Cluster Enterprise Cluster/Grid Global Grid SMPs or SuperComputers

4. GRID Computing • Grids are about large-scale resource sharing. • Spanning administrative boundaries. • Central processors, storage, network bandwidth, databases, applications, sensors and so on • Problem solving in dynamic, multi-institutional environment. • Organizing geographically distributed computing resources • So that they can be flexibly and dynamically allocated and accessed • Providing such capabilities, where Sharing is highly controlled, clear definitions of exactly what is shared, who is allowed to share, and the conditions under which sharing occurs.

5. Elements of Grid Computing • Resource sharing • Computers, data, storage, sensors, networks, … • Sharing always conditional: issues of trust, policy, negotiation, payment, … • Coordinated problem solving • Beyond client-server: distributed data analysis, computation, collaboration, … • Dynamic, multi-institutional virtual organizations • Community overlays on classic org structures • Large or small, static or dynamic

6. Virtual Organizations • A set of individuals and/or institutions defined by a set of sharing rules • The sharing is highly controlled, with resource providers and consumers defining clearly and carefully just what is shared An example: the set of application service providers, storage service providers, cycle providers and consultants engaged by a car manufacturer to plan for a new factory Another example: industrial consortium building a new aircraft

7. More Formal Definition of Grids • A grid is a system that: • Coordinates resource sharing in a de-centralized manner (i.e., different VOs). • Uses standard, open, general purpose protocols and interfaces. • Delivers non-trivial qualities of service. • Guaranteed bandwidth for application. • Guaranteed CPU cycles. • Guaranteed latency.

8. Computational Grid Applications • Biomedical research • Industrial research • Engineering research • Studies in Physics and Chemistry

9. Science Today is a Team Sport!! I. Foster

10. eScience eScience [n]: Large-scale science carried out through distributed collaborations—often leveraging access to large-scale data & computing I. Foster

11. TeraGrid is an Important Project developed by the National Science Foundation (NSF). Slide obtained from B. Wilkinson, http://sol.cs.wcu.edu/~abw/CS493F04/

12. TeraGrid Slide obtained from B. Wilkinson, http://sol.cs.wcu.edu/~abw/CS493F04/

13. UK e-Science Grid Slide obtained from B. Wilkinson, http://sol.cs.wcu.edu/~abw/CS493F04/

14. Applications • NationalVirtual Observatory • Astronomical surveys produce terabytes of data. • Data sets will cover sky in different wave bands (x-rays, optical, infrared, radio). • Challenge is to make this accessible to general research community. • Heterogeneous data producers and consumers. • Resources in this Grid are data sets rather than compute engines.

15. High-Energy Physics • Large-scale collaborations for CERN’s Large Hadron Collider. • Involves 4000 physicists, 150 institutions, in more than 30 countries. • Data sets now at petabyte level. Predicted to generate data at the exabyte level in this decade. • Challenges: • Providing rapid access to subsets of data. • Secure access to distributed computing and data handling resources.

16. Essentially, provide a distributed collaborative infrastructure that will allow physicist from around the globe to effectively analyze results from their home institution.

17. Online Access to Scientific Instruments Advanced Photon Source wide-area dissemination desktop & VR clients with shared controls real-time collection archival storage tomographic reconstruction DOE X-ray grand challenge: ANL, USC/ISI, NIST, U.Chicago

18. NSF Network for Earthquake Engineering Simulation (NEES) Transform our ability to carry out research vital to reducing vulnerability to catastrophic earthquakes I. Foster

19. NEES • network of 15 large-scale, experimental sites • advanced tools such as shake tables, centrifuges that simulate earthquake effects, unique laboratories, a tsunami wave basin and field-testing equipment. • linked to a centralized data pool and earthquake simulation software, bridged together by the high-speed Internet2. • off-site researchers to interact in real time with any of the networked sites.

20. Securely store, organize, and share data within a standardized framework in a central location. • Remotely observe and participate in experiments through the use of synchronized real-time data and video. • Collaborate with colleagues to facilitate the planning, performance, analysis, and publication of research experiments. • Conduct hybrid simulations that combine the results of multiple distributed experiments and link physical experiments with computer simulations.

21. DOE Earth System Grid Goal: address technical obstacles to the sharing & analysis of high-volume data from advanced earth system models www.earthsystemgrid.org I. Foster

22. I. Foster Earth System Grid

23. High-resolution, long-duration simulations performed with advanced DOE climate models produce tens of petabytes of output. • This output made available to global change impacts researchers nationwide, both at national laboratories and at universities, other research laboratories, and other institutions. • a virtual collaborative environment that links distributed centers, users, models, and data. • provides scientists with virtual proximity to the distributed data and resources that they require to perform their research.

24. Lets Play Virtual Organization! • The members of this class represent a VO within the university. • The resources of the VO include: • The laptops, workstations, and printers belonging to the individuals of the VO (that’s you guys1!). • Does this bring up any issues worth concerning yourself about? 1. I do not join virtual organizations

25. Want to tightly control who may use these resources and how they may be used. Thus need security.

26. Security: • Want to tightly control who may use these resources and how they may be used. • How about Larry and Ramm wanting to use your printer at the same time (which happens to be 3:30 AM). Is this a problem?

27. Security: • Want to tightly control who may use these resources and how they may be used. • How about Larry and Sarah wanting to use your printer at the same time (which happens to be 3:30 AM). Is this a problem? • Might want to have a scheduler, which in this case need not be more sophisticated than turning off the printer. • What if David forgot Dan’s IP address and cannot gain access to his laptop? How could this be resolved (assuming you want it resolved)?

28. What if David forgot Dan’s IP address and cannot gain access to his laptop? How could this be addressed (assuming you want it addressed)? • You could provide an information service that could tell David how to find the laptop. • You would also have to deal with allocating multiple resources to a user, e.g., a laptop to write a paper and a printer to print it out. Thus need a resource manager. • Also need a way to monitor your application executing in your VO Grid.

29. Grid Computing Software Infrastructure

30. Open Grid Services Architecture • Developed by the Global Grid Forum to define a common, standard, and open architectures for Grid-based applications. • Provides a standard approach to all services on the Grid. • VO Management Service. • Resource discovery and management service: • Job management service. • Security services. • Data management services. • Built on top of and extends the Web Services architecture, protocols, and interfaces.

31. A stateless Web Service invocation

32. Figure 1.11. A stateful Web Service invocation

33.  Relationship between OGSA, WSRF, and Web Services

34. WSRF • Web Services Resource Framework • a specification developed by OASIS. • WSRF specifies how to make Web Services stateful. • joint effort by the Grid and Web Services communities. • WSRF provides the stateful services that OGSA needs. • OGSA is the architecture, WSRF is the infrastructure on which that architecture is built on.

35. Standards Bodies The primary standards-setting body is1: • Global Grid Forum (GGF) • Started in 1998 • Meets three times a year, GGF1, GGF2, GGF3 … • More than 40 organizations involved and growing … Others: • W3C consortium (Worlds Wide Web Consortium) • Working on standardization of web-related technologies such as XML • See http://www.w3.org • OASIS (Organization for the Advancement of Structured Information Standards) • IETF, DMTF 1 “The Grid Core Technologies” by M. Li and M. Baker, 2005, page 4.

36. Standards in the Web Services World • XML introduced (ratified) in 1998 • SOAP ratified in 2000 • Web services developed • Subsequently, standards have been are continuing to be developed: • WSDL • WS-* where * refers to names of one of many standards

37. Standards in the grid computing world • Open Grid Services Architecture (OGSA) • First announced at GGF4 in Feb 2002 • OGSA does not give details of implementation.

38. Globus Project • Open source software toolkit developed for grid computing. • Roots in I-way experiment. • Work started in 1996. • Four versions developed to present time. • Reference implementations of grid computing standards. • Defacto standard for grid computing.

39. Globus Version 4 • A “toolkit” of services and packages for creating the basic grid computing infrastructure • Higher level tools added to this infrastructure • Version 4 is web-services based • Some non-web services code exists from earlier versions (legacy) or where not appropriate (for efficiency, etc.).

41. Layered diagram of OGSA, GT4, WSRF, and Web Services

42. Each part comprises a set of web services and/or non-web service components. • Some built upon earlier versions of Globus.

43. Globus Open Source Grid Software G T 4 Delegation Service Community Scheduler Framework [contribution] Python WS Core [contribution] C WS Core G T 3 CommunityAuthorization Service OGSA-DAI [Tech Preview] WS Authentication Authorization Reliable File Transfer Java WS Core Grid Resource Allocation Mgmt (WS GRAM) Monitoring & Discovery System (MDS4) G T 2 Pre-WS Authentication Authorization GridFTP Grid Resource Allocation Mgmt (Pre-WS GRAM) Monitoring & Discovery System (MDS2) C Common Libraries G T 3 Replica Location Service XIO G T 4 Credential Management Web ServicesComponents Non-WS Components Security Data Management Execution Management Information Services CommonRuntime I Foster

44. Another view of GT4 Components Your Python Client Your C Client Your Java Client Your Python Client Your Python Client Your C Client Your C Client CLIENT Your Java Client Your Java Client Your Python Client Your C Client Your Java Client Interoperable WS-I-compliant SOAP messaging X.509 credentials = common authentication Trigger Archiver Your C Service GRAM RFT Delegation Index CAS OGSA-DAI GTCP Your Python Service Your Java Service Your Java Service RLS GridFTP SimpleCA MyProxy Pre-WS MDS Pre-WS GRAM C WS Core pyGlobus WS Core Java Services in Apache Axis Plus GT Libraries and Handlers Python hosting, GT Libraries C Services using GT Libraries and Handlers SERVER I Foster

45. GT Core • Provides the ability to create services running inside the GT 4 container.

46. Java WS Core G T 4 Delegation Service Community Scheduler Framework [contribution] Python WS Core [contribution] C WS Core G T 2 Pre-WS Authentication Authorization GridFTP Grid Resource Allocation Mgmt (Pre-WS GRAM) Monitoring & Discovery System (MDS2) C Common Libraries G T 4 Credential Management G T 3 CommunityAuthorization Service OGSA-DAI [Tech Preview] Web ServicesComponents WS Authentication Authorization Reliable File Transfer Java WS Core Grid Resource Allocation Mgmt (WS GRAM) Monitoring & Discovery System (MDS4) Non-WS Components G T 3 Replica Location Service XIO Security Data Management Execution Management Information Services CommonRuntime

47. User Applications Custom WSRF Web Services Custom Web Services GT4WSRF Web Services Registry Administration GT4 Container WS-Addressing, WSRF, WS-Notification WSDL, SOAP, WS-Security GT4 Web Services Core I Foster

48. Execution Management Key component GRAM (Grid Resource Allocation Manager) • For submitting executable jobs • May interface to a local job scheduler

49. GRAM (Grid Resource Allocation Manager) G T 4 Delegation Service Community Scheduler Framework [contribution] Python WS Core [contribution] C WS Core G T 2 Pre-WS Authentication Authorization GridFTP Grid Resource Allocation Mgmt (Pre-WS GRAM) Monitoring & Discovery System (MDS2) C Common Libraries G T 4 Credential Management G T 3 CommunityAuthorization Service OGSA-DAI [Tech Preview] Web ServicesComponents WS Authentication Authorization Reliable File Transfer Java WS Core Grid Resource Allocation Mgmt (WS GRAM) Monitoring & Discovery System (MDS4) Non-WS Components G T 3 Replica Location Service XIO Security Data Management Execution Management Information Services CommonRuntime

50. GT4 GRAM Structure: Sun Grid Engine Service host(s) and compute element(s) GT4 Java Container Compute element Local job control GRAM services GRAM services Local scheduler Job functions sudo GRAM adapter Delegate Transfer request Delegation Client Delegate GridFTP User job RFT File Transfer FTP control FTP data Remote storage element(s) GridFTP Data management components I Foster