1.46k likes | 1.47k Vues
Explore the evolution of computing technologies, challenges, and architecture in Grid Computing at the University of Melbourne. Learn about collaborative projects, technologies, and applications driving tech evolution.
E N D
WW Grid The Gridbus Toolkit for Service-Oriented Grid and Utility Computing Grid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software EngineeringThe University of MelbourneMelbourne, Australiawww.gridbus.org Rajkumar Buyya Collaboration: Nimrod-G Project
Pointers and Acknowledgements Download PPT Slides from: http://www.buyya.com/tut/gridbus.zip
Acknowledgements: Co-authors, Collaborators, and Funding Sources • Gridbus Project Members • Rajkumar Buyya (Gridbus PI) • Srikumar Venugopal (Ph.D. student) • Jia Yu (Ph.D. student) • Elan Kovan (Ph.D. student) • Anthony Sulistio (Ph.D. student) • Chee Shin Yeo (Ph.D. student) • Manjuka Soysa (Ph.D. student) • Shoaib Burq (Research Assistant) • Martin Placek (B.E student) • Rajiv Ranjan (Masters by Research) • Alex Barmouta (from UWA probing GridBank ) • Ding Choon-Hoong • Akshay Luther (Alchemi .NET Grid framework) • Virtual Lab - Docking • Kim Branson, WEHI for Structural Biology • NeuroGrid • Susumu Date, Osaka University • HEPGrid • Lyle Winton, School of Physics (HEP data catalogue and BASF application) • Natural Language Engineering: • Baden Hughes badenh@cs.mu.oz.au • Finance/Portfolio/investment risk analysis • Rafael Moreno-Vozmediano, Complutense University of Madrid • Industry Collaborator • Wolfgang Gentzsch, Sun Microsystems • Benjamin Khoo, IBM Global Services, Singapore • Collaborators • David Abramson & Jon Giddy, Monash – Nimrod-G • Manzur Murshed, Monash (GridSim core 1.0) • Heinz Stokinger, CERN (grid economy models) • Jahanzeb Sherwani, Nosheen Ali, Nausheen Lotia, Zahra Hayat (LUMS) • Hussein Gibbins, X – RA, GRIDS lab, Melborune (Gridscape)
Agenda • Technology Evolution and Introduction to Grid Computing • Grid Challenges, Approaches, and Architecture • Grid Architecture for Computational Economy • Grid Economy driven Technologies [+demo] • Grid Market Directory • Grid Bank • Nimrod-G and Gridbus Data Broker • Visual Parametric Modeler • Grid Simulation (GridSim) Toolkit • Scheduling Applications on Global Grids [+demo] • Summary and Conclusions
* HTC * P2P * PDAs Minicomputers * * PCs * Workstations * Mainframes * Grids COMPUTING * PC Clusters * Crays * MPPs * WS Clusters * XEROX PARC worm * IETF * W3C * TCP/IP Communication * Ethernet * HTML * Mosaic * Web Services * Email * Sputnik * Internet Era * WWW Era * XML * ARPANET 1960 1970 1975 1980 1985 1990 1995 2000 Computing and Communication Technologies Evolution
2100 2100 2100 2100 2100 2100 2100 2100 2100 Scalable Computing PERFORMANCE + Q o S Administrative Barriers • Individual • Group • Department • Campus • State • National • Globe • Inter Planet • Universe Personal Device SMPs or SuperComputers Global Grid Inter Planet Grid Local Cluster Enterprise Cluster/Grid
Application Drivers for Tech. Evolution • Solving grand challenge applications using modeling, simulation and analysis Aerospace Internet & Ecommerce Life Sciences Military Applications CAD/CAM Digital Biology
Internet, Web, and Grid Effect 140 ‘Web Services and Grid Effect’ 120 100 The 'Network Effect’ kicks in, and the web goes critical' Number of hosts (millions) Business $$$ 80 60 40 20 0 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 TCP/IP HTML Mosaic XML 4. with XML PHASE 2. The Internet is Born 3. The World Wide Web 5. The Grid 1. Packet Switching Networks HTML hypertext system created 1969: 4 US Universities linked to form ARPANET TCP/IP becomes core protocol CERN launch World Wide Web 1972: First e-mail program created Domain Name System created IETF created (1986) NCSA launch Mosaic interface 1976: Robert Metcalfe develops Ethernet
Clusters: Emerged as Mainstream, Low Cost and High Performance Computers
Cluster 1 Scheduler Master Daemon LAN/WAN Submit Graphical Control Cluster 3 Execution Daemon Scheduler Clients Master Daemon Cluster 2 Scheduler Submit Graphical Control Execution Daemon Master Daemon Clients Submit Graphical Control Execution Daemon Clients Multi-Clustering: Coupling Multiple Clusters for Solving Lager Problems
http://www.sun.com/hpc/ Grid Emergence: Towards Global Computing Grid enables: • Resource Sharing • Selection • Aggreation - Unification of geographically distributed resources
What is Grid ? • A type of parallel and distributed system that enables the sharing, selection, & aggregationof geographically distributed resources: • Computers– PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc; • Software– e.g., ASPs renting expensive special purpose applications on demand; • Catalogued data and databases– e.g. transparent access to human genome database; • Special devices/instruments – e.g., radio telescope – SETI@Home searching for life in galaxy. • People/collaborators. depending on their availability, capability, cost, and user QoS requirements for solving large-scale problems/applications. Widearea
Formal Definition! • Grid is a type of parallel and distributed system that enables the sharing, selection, and aggregation of geographically distributed "autonomous" resources dynamically at runtime depending on their availability, capability, performance, cost, and users' quality-of-service requirements.
database A Bird Eye View of World-Wide Grid Environment Grid Information Service Grid Resource Broker Application R2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service
What do Grids aim for and how to support them. • Grids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include: • Resource sharing • “On-demand” Virtual Enterprises creation • Aggregation of resources on demand. • For this cooperation to be sustainable, participants needs to have (economic) incentive. • Therefore, “incentive” mechanisms should be considered as one of key design parameters of Grid computing.
Synergies that Result from Cooperation:Classes of Grid Application Drivers • Distributed HPC (Supercomputing): • Computational science. • High-Capacity/Throughput Computing: • Large scale simulation/chip design & parameter studies. • Content Sharing (free or paid) • Sharing digital contents among peers (e.g., Napster) • Remote software access/renting services: • Application service provides (ASPs) & Web services. • Data-intensive computing: • Drug Design, Particle Physics, Stock Prediction... • On-demand, realtime computing: • Medical instrumentation & Mission Critical. • Collaborative Computing: • Collaborative design, Data exploration, education. • Service Oriented Computing (SOC): • Towards economic-based Utility Computing: New paradigm, new applications, new industries, and new business.
Drivers: Next Generation Applications (NGA) • Next generation experiments, simulations, sensors, satellites, even people and businesses are creating a flood of data. They all involve numerous experts/resources from multiple organization in synthesis, modeling, simulation,analysis, and interpretation. ~PBytes/sec High Energy Physics Brain Activity Analysis Newswire & data mining: Natural language engineering Digital Biology Life Sciences Astronomy Quantum Chemistry Finance: Portfolio analysis Internet & Ecommerce
Common Attributes/Needs/Challenges of NGA • They involve Distributed Entities: • Participants/Organizations • Resources • Computers • Instruments • Datasets/Databases • Source (e.g., CDB/PDBs) • Replication (e.g, HEP Data) • Application Components • Heterogeneous in nature • Participants require share analysis results of analysis with other collaborators (e.g., HEP) • Grids offer the most promising solution & enable global collaborations. • The beauty of the grid is that it provides a secure access to a wide range of heterogeneous resources. • But what does it take to integrate and manage applications across all these resources?
Computational Economy Security Data locality Resource Allocation & Scheduling Uniform Access System Management Resource Discovery Application Construction Network Management Grid Challenges
What Grids need to provide • Services that enable the execution of a job on a resource in different admistrative domain. • Security mechanisms that permit resources to be accessed only by authorized users. • App/Data Security (?) – A must for commercial users (protecting from GSPs/other users). • (New) programming tools that make our applications Grid Ready!. • Tools that can translate the requirements of an application/user into the requirements of computers, networks, and storage. • Tools that perform resource discovery, trading, selection/allocation, scheduling and distribution of jobs and collects results.
database Grid Operations Management Challenges Grid Information Service Grid Resource Broker Application R2 2 R3 R4 R5 RN Grid Resource Broker R6 R1 Resource Broker Grid Information Service challenges * Resource Management * Application Construction
Sources of Complexity in Grid Resource Management • Size (large number of nodes, providers, consumers) • Heterogeneity of resources (PCs, Workstatations, clusters, and supercomputers) • Heterogeneity of fabric management systems (single system image OS, queuing systems, etc.) • Heterogeneity of fabric management polices • Heterogeneity of applications (scientific, engineering, and commerce) • Heterogeneity of application requirements (CPU, I/O, memory, and/or network intensive) • Heterogeneity in demand patterns • Geographic distribution and different time zones • Differing goals (producers and consumers have different objectives and strategies) • Unsecure and Unreliable environment
Centralized Vs Distributed Management • They use centralised policy that need • complete state-information and • common fabric management policy or decentralised consensus-based policy. • Due to too many heterogenous parameters in the Grid it is impossible to define: • system-wide performance matrix and • common fabric management policy that is acceptable to all. • Therefore, the Grid resource management policy need to be distributed in natures so that autonomous entities organise themselves and yet benefit from cooperation.
Realizing the Grid Grid Architecture, Approaches, and Efforts
Grid Realization Steps • The integration of individual s/w & h/w components into a combined networked resource (single system image cluster). • Low-level middleware to provide a secure and uniform access to services provided by different resources. • User-level middleware to support application development and aggregation of distributed resources. • The construction of distributed applications.
Layered Grid Architecture APPLICATIONS Applications and Portals … Prob. Solving Env. Collaboration Engineering Web enabled Apps Scientific USER LEVEL MIDDLEWARE Development Environments and Tools Languages/Compilers Libraries Debuggers Monitors … Web tools Resource Management, Selection, and Aggregation (BROKERS) CORE MIDDLEWARE Distributed Resources Coupling Services Process Data Information Accounting … QoS Trading SECURITY LAYER Local Resource Managers FABRIC … Internet Protocols Libraries & App Kernels Queuing Systems Operating Systems Networked Resources across Organizations … Computers Networks Storage Systems Data Sources Scientific Instruments
Australia Nimrod-G Gridbus GridSim Virtual Lab DISCWorld GrangeNet ..new coming up Europe UNICORE Cactus UK eScience EU Data Grid EuroGrid MetaMPI XtremeWeb and many more. India I-Grid Japan Ninf DataFarm Korea... N*Grid USA Globus Legion OGSA Sun Grid Engine AppLeS NASA IPG Condor-G Jxta NetSolve AccessGrid and many more... Cycle Stealing & .com Initiatives Distributed.net SETI@Home, …. Entropia, UD, Parabon,…. Public Forums Global Grid Forum Australian Grid Forum IEEE TFCC CCGrid conference P2P conference Some Grid Projects & Technologies http://www.gridcomputing.com
NetSolve mix-and-match Object-oriented Internet/partial-P2P Grid Computing Approaches Network enabled Solvers Economic-based Service-Oriented Computing Nimrod-G
WW Grid Many Testbeds ? & who pays ?, who regulates demand and supply ? GUSTO (decommissioned) World Wide Grid Legion Testbed NASA IPG
Testbeds so far -- observations • Who contributed resources & why ? • Volunteers: for fun, challenge, fame, charismatic apps, public good like distributed.net & SETI@Home projects. • Collaborators: sharing resources while developing new technologies of common interest – Globus, Legion, Ninf, Ninf,,... Unless you know lab. leaders, it is impossible to get access! • How long ? • Short term: excitement is lost, too much of admin. Overhead (Globus inst+), no incentive, policy change,… • What we need ? Grid Marketplace! • Regulates supply-and-demand, offers incentive for being players, simple, scalable solution, quasi-deterministic – proven model in real-world.
Gridbus: Towards Building Grid Economy and Business Toolkit Offers Incentive and Enable the Creation and Promotion of: Grid Marketplace (competitive) ASP Service Oriented Computing . . . And let users focus on their own work (science, engineering, or commerce)!
The Gridbus Vision: To Enable Service Oriented Grid Computing & Bus iness! WW Grid Gridbus World Wide Grid! + marketplace for Service-Oriented Computing
GRIDS Lab @ the U. of Melbourne, The Gridbus Project: www.gridbus.org • R& D in Distributed Computational Economy and end-to-end infrastructure for Service-Oriented/Utility Computing: : • Architecture, Specification, and “Open Source” Reference Implementation in Collaboration with interested Global Peers. • Platform: Cluster, Grid, P2P for diverse applications. • Visual Tools for Creation of Distributed Applications • Grid Economy & Scheduling (via Nimrod-G Broker) • Grid Modeling and Simulation (GridSim) • Economic Cluster Scheduler (Libra) • Grid Accounting Services Architecture (GridBank) • Grid Service Publication (GMD) • World Wide Grid (WWG) – A Global Grid Testbed • Jxta-based P2P Compute Power Market (CPM) • Application Enabler Projects: • Virtual Laboratory Toolset for Drug Design • High-Energy Physics and the Grid Network (HEPGrid) • Brain Activity Analysis on the Grid (NeuroGrid) • GridEmail for Spam Management
What do users want ? • Grid Consumers • Execute jobs for solving varying problem size and complexity • Benefit by selecting and aggregating resources wisely • Tradeoff timeframe and cost • Strategy: minimise expenses • Grid Providers • Contribute (“idle”) resource for executing consumer jobs • Benefit by maximizing resource utilisation • Tradeoff local requirements & market opportunity • Strategy: maximise return on investment
Grid Characteristics & Operations Management Challenges • Grid has Large Heterogeneity • Resource Types: PC, WS, Clusters • Resource Architecture: CPU Arch, OS • Applications: CPU/IO/message intensive • Access Price: different for different users, resources and time. • Availability: varies with time. • Grid has Distributed • Resources, Ownership, Users • Grid has self-interested entities: • Users/Grid Service Consumers: Minimise expenses • Resource Owners/Grid Service Providers: Maximise profit • The heterogeneity and decentralization that is present in Grid is very similar to one present in “human economies” where market-based mechanisms have been used successfully to manage them. • Therefore, we advocate the use of economic/market-models for managing Grid resources by treating their services as ICT commodities/ utilities.
Grid Economy: Incentive (sustained sharing), Regulation (supply & demand)
Service Pricing and Economic Models • Price-based: Supply, demand, value, and wealth of economic system • Commodity Market Model • Posted Price Model • Bargaining Model • Tendering (Contract Net) Model • Auction Model • English, first-price sealed-bid, second-price sealed-bid (Vickrey), and Dutch (consumer:low,high,rate; producer:high, low, rate) • Proportional Resource Sharing Model • Monopoly (one provider) and Oligopoly (few players) • consumers may not have any influence on prices. • Bartering • Shareholder Model • Partnership Model
Benefits of Computational Economy • It provides an effective paradigm for managing self interested and self-regulating entities (resource owners and consumers) • Helps in regulating supply-and-demand of resources. • Services can be priced in such a way that equilibrium is maintained. • User-centric / Utility driven • Scalable: • No need of central coordinator (during negotiation) • Resources(sellers) and also Users(buyers) can make their own decisions and try to maximize utility and profit. • Adaptable, • It allows to offer different QoS (quality of services) to different applications depending the value users place on them. • It offers incentive for resource owners for being part of the grid! • It offers incentive for resource consumers for being good citizens. • It improves the utilisation of resources.
GRACE: A Reference Grid Economy Services Architecture GRid Architecture for Computational Economy (GRACE)
Grid Economy & Users’ Challenges • Grid Service Providers (GSPs) • How do I decide service pricing models ? • How do I specify them ? • How do I translate them into resource allocations ? • How do I enforce them ? • How do I advertise & attract consumers ? • How do I do accounting and handle payments? • ….. • Grid Service Consumers (GSCs) • How do I decide expenses ? • How do I express QoS requirements ? • How do I trade between timeframe & cost ? • How do I map jobs to resources to meet my QoS needs? • ….. • They need mechanisms and technologies for value expression, value translation, and value enforcement.
Market-based Computing Systems Requirements • To enable users (GSPs and GSCs) to realise economic value, market-based systems need to provide mechanisms for: • Value Expression • a means to express their requirements, valuations, and objectives • Value Translation • scheduling policies to translate them to resource allocations • Value Enforcement • mechanisms to enforce the selection and allocation of differential services, and dynamic adaptation to changes in their availability at runtime • Market mechanisms, accounting and payment, Reservation of resources.
GRACE: A ReferenceService-Oriented Grid Architecture for Computational Economies Data Catalogue Grid Bank Information Service Grid Market Services Sign-on HealthMonitor Info ? Grid Node N … Grid Explorer … Secure ProgrammingEnvironments Job Control Agent Grid Node1 Applications Schedule Advisor QoS Pricing Algorithms Trade Server Trading Trade Manager Accounting Resource Reservation Misc. services … Deployment Agent JobExec Resource Allocation Storage Grid Resource Broker … R1 R2 Rm Grid Middleware Services Grid Consumer Grid Service Providers
Realising Market-based Grid: Minimal New Components • Grid Market Directory Services • Grid Trading Services – • for different economic models • Grid Metering Services • Grid Accounting and Payment Services • Grid Service Broker
Globus Grid Technologies for Realising the Vision of Grid Economy Framework High Energy Physics Brain Activity Analysis Grid Apps. Natural Language Engineering Molecular Docking Portfolio Analysis Grid Email High-level Services and Tools … User-LevelMiddleware (Grid Tools) Gridscape G-Monitor Programming Framework Grid Brokers & Schedulers Nimrod-G Gridbus Data Broker Alchemi: .NET Grid Services +Clustering of desktop PCs Data Management Services GridBank GMD Core Grid Middleware MDS GRAM GASS PKI-basedGrid Security Interface (GSI) .NET JVM Condor PBS SGE Libra Tomcat Grid Fabric Windows Linux AIX IRIX OSF1 HP UX Solaris
Gridbus Technologies • Application Construction Tools • Visual Parametric Modeller (VPM) • Grid Economy Services • Grid Market Directory • A Registry for publication of GSPs and their Services – VO/VE • Grid Bank • A Grid Accounting Services • Grid Trading Services • Data Grid Service Broker • QoS based Scheduling of Distributed Data Oriented Apps on global Grids • Builds on the Nimrod-G work • Gridscape • Interactive Grid Testbed Portal Generator • G-monitor • Grid Application Execution Management Portal • GridSim • A Grid Simulation Toolkit • Libra • Economy based Cluster Scheduling