The Grid

1. The Grid ”Enter the GRID” af Kristian Mandrup

2. Indeks • Intro • Overview • Architecture • Solutions • Future • Conclusions & discussion

3. What is it ? • The Next-Gen Internet?  • A 21st century time machine?

4. Intro • Future of collaborative problem-solving • Internet's next evolutionary step • The Grid is a new class of infrastructure • Link computers in new ways • Open up storage and transaction power as Web opened up content

5. Intro (2) • Era of distributed, networked computing is just beginning • The WWW a taste, the Grid a vision • Answer to the enterprise computing crisis (ECC)

6. Vision • Applies interconnected model used by power utilities to access services, software and hardware resources as part ofvirtualsupercomp. • Executes jobs on best suited, least loaded systems in a seamless, transparent and secure way • On-demand access to computational power, data bases and services • Manage resource sharing and co-ordinated problem solving across dynamic, multi-institutional virtual organisations both in eScience and eBusiness • Provides scalable, secure, high-performance mechanisms for discovering and negotiating access to remote resources • Geographically distributed groups can work together in new ways

7. Background (evolution) • Breakthrough technologies • Begun in the research environment • Moved to open standards • Applied to business applications • What we are seeing with Grid standards

8. Background (history) • Desire to connect supercomputers into "metacomputers" that could be remotely controlled • Vision of the Grid started in 1960s • Envisioned a computer facility operating "like a power company or water company" • Word "grid" borrowed from the electricity grid • Any compatible device could be plugged in anywhere on the Grid and be guaranteed a certain level of resources, regardless of where those resources might come from

9. Evolution • 1G Grids • Involved local "meta-computers" with basic services such as distributed file systems and site-wide single sign on. • 1G Grids were totally custom made • 2G Grids • Underlying software services and communications protocols Grids offered basic building blocks, but deployment involved significant customization • Interoperability among 2G Grid systems very difficult • 3G Grids • Solves deployment and interoperability issues by providing standard interfaces • Today it feasible to realize the Grid vision • Global Grid Forum (GGF) created in November, 2000

10. Demand • Science & Industry • High-energy physics, needs extra resources to manage and analyze huge amounts of data • Science and industry participants require level of reliability not offered by current peer-to-peer initiatives • Strong need to efficiently manage availability of distributed infrastructures, applications and services • Computational resources are failing to keep up with what scientists demand of them

11. Demand (technical) • Doubling periods (months) • Network bandwidth 9 • Storage capacity 12 • Computing power 18 • Computer power is falling behind storage !

12. Demand (example) • Scientists create high-resolution simulations need petabyte archives • CERN's Large Hadron Collider (LHC) will produce multiple petabytes (1015 byte) of data per year • Scientists demand 10+ Gb/s to work remotely on petabyte data sets • Law of diminishing returns ???

13. Demand (solutions) • If communication is unlimited and free • Not restricted to using local resources to solve problems • Use collective computing power of research collaboration or buy from provider • Look at large datasets using special collaboration and visualization tools • Use remote resources to do things not possible using local resources

14. Benefits • Aggregates compute power and delivers it as a network service • Grid Engine presents users to a seamless, integrated computing capability • Facilitate the deployment of compute farms, the basic building blocks of grid computing • Making large amounts of compute power available for applications and users

15. Benefits ”sales talk” • Raise productivity • Maintain availability • Minimize downtime • Shorter time to market • Reduces costs by better utilisation of resources • Quicker and better results • Increased quality and innovation • Do things not possible before • Increased ROI (Return On Investment)

16. Potential problems • Social and political dimensions (like WWW) • Sharing between strangers where no history of trust

17. Uses • Development of semiconductors • Bioinformatics • Mechanical design • Software development • Oil/gas exploration • Financial analysis • Academic and research pursuits

18. Architecture (Infrastructure) • Open Grid Services Architecture (OGSA) • Integration of Grid and Web services technologies • Open Grid Services Infrastructure (OGSI) • Grid Resource Access and Management (GRAM) protocol and service • Remote resource allocation and process creation • Monitoring • Management services

19. Architecture (OGSA) • Open Grid Services Architecture • Establish standard interfaces and behaviours for distributed system management • Management of service instances (persistent or transient) • Defines fundamental WDSL interfaces: to establish a Grid service in the open source Global Toolkit 3.0 (GT3) • Grid service instance: maintains a set of service data elements by encapsulating XML fragments in standard containers • FindServiceData operation: queries this information and allows notification of service existence and modifications in service • Includes GT3 (Global Toolkit 3) Core and Base Services

20. Architecture • 1) physical devices or resources • 2) Core communication and authentication protocols cryptographically secure mechanisms - verifying identity of users and resources • 3) Protocols, services, and APIs • Implement interactions across collections of resources • Directory and brokering services for resource discovery and allocation • Monitoring and diagnostic services • Data replication services • Membership and policy services • 4) User applications

21. Security • Unlike the Web, the Grid is being designed from the ground up as a secure system • Accept only messages coming from special hosts and reserved ports • Integration with Kerberos5 and DCE exists • Authentication, authorization, and policy • Client and a server need to mutually authenticate each other. • No distinction between client and server. Server one moment, client another moment. • Special requirements for managing transaction

22. Security (method) • Single sign-on: Via creation of a proxy credential • Mapping to local security mechanisms: Grid security infrastructure maps to local solutions at each site • Delegation: Sub-computations created at sites A and B.Both communicate with each other and access files at site C • Community authorization and policy: infeasible for each resource to keep track of community membership and privileges. Group membership identified with cryptographic credential issued by trusted third party

23. Security (how it works) • User calls on computational resources of sites A and B • Communicate with each other , read files located at site C. • Each step requires authorization and authentication • Mediating requests requires the Grid Security Infrastructure (GSI) • Provides: • Single sign-on • Run-anywhere authentication service • Support for delegation of credentials to sub-computations • Local control over authorization • Mapping from global to local user identities

24. Implementation requirements • Implementing architecture requires uniform mechanisms • Creating and managing services on remote computers • Supporting single sign-on to distributed resources • Transferring large datasets at high speed • Forming large distributed virtual communities • Maintaining information about existence, state, and usage policies of community resources

25. Solutions • Sun Microsystems acquired Gridware, a private developer of Distributed Resource Management (DRM) software, in July 2000 • Becomes Grid Engine project • Grid Engine project goals: • New open standards for DRM • Standard API for application integration • Grid Engine Portal(GEP) • Provides a Java based capability for enabling highly secure internet access to applications that run on an existing Grid Engine grid • Loosely coupled to Grid Engine, SunONE Portal Server • Globus Toolkit (1996) • Standards-based protocols for distributed system management for open source implementation

26. Using the Grid • Steps to take • Discover resources exist. • Negotiate access to resources • Configure hardware/software to use resources • Avoid compromising security of self or remote resources

27. How it works • Obtaining: authentication credentials • Querying: Information system and replica catalog to determine availability of computers, storage systems, and networks, and location of required input files (collective services) • Submitting: requests to appropriate computers, storage systems, and networks to initiate computations, move data, and so forth (resource protocols) • Monitoring: the progress of the various computations and data transfers, notifying the user when all are completed, and detecting and responding to failure conditions (resource protocols)

28. Status • Grid Engine software has been ported to many operating systems, including Solaris, Linux • Current v.5.4 can be downloaded

29. The Future • The Grid may give birth to a global file-swapping network or a members-only citadel for moneyed institutions • The future of the Grid is unknown !

30. Conclusions & discussions • Is this the distributed systems utopia ? • Is anything missing ? • What is the next step after ”the Grid” – ”the Matrix” ??? • What is ”the Matrix” ? • CM-systemers rolle ?