1 / 21

User-Level Middleware for the Grid

User-Level Middleware for the Grid. Francine Berman University of California, San Diego. A Short History of the Grid. In the beginning, there were applications and resources, and it took ninja programmers and many months to implement the applications on the Grid …. Applications. Resources.

zarita
Télécharger la présentation

User-Level Middleware for the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. User-Level Middleware for the Grid Francine Berman University of California, San Diego

  2. A Short History of the Grid In the beginning, there were applications and resources, and it took ninja programmers and many months to implement the applications on the Grid … Applications Resources

  3. And behold, there were services, and programmers saw that it was good (even though their performance was still often less than desirable) … Applications Applications Grid Middleware Resources Resources

  4. … and it came to pass that user-level middleware was promised to promote the performance of Grid applications, and the users rejoiced … Applications Applications User-Level Middleware Applications Grid Middleware Grid Middleware Resources Resources Resources

  5. Applications User-Level Middleware Grid Middleware Resources The Middleware Promise • Grid Middleware • Provides infrastructure and services to enable usability of the Grid • Provides a single system view of Grid resources to applications • User-level Middleware • Hides the complexity of the Grid for the end-user • Promotes application performance, portability and retargetability

  6. Building User-Level Middleware • Goals • Middleware should make it easier for applications to achieve acceptable performance on the Grid • Middleware should target a reasonable domain of programs • Middleware should mask the complexity of the system for the user • Should be able to accommodate different Grid infrastructure packages • Should be able to adapt to different and variable resource performance behaviors • Middleware should provide “hooks” for better performance Berman, UCSD

  7. AppLeS Template Architecture Application Module Scheduling Module Deployment Module Grid Middleware and Resources The AppLeS Approach to User-Level Middleware • (AppLeS = Application Level Scheduling) • Develop templateswhich • Target structurally similar classes of applications • Promote pretty good application performance • Make it easy and time-efficient to instantiate programs

  8. Common application structure used in various fields of science and engineering (Monte Carlo and other simulations, etc.) Joint work with Henri Casanova First AppLeS Middleware package to be distributed to users Large number of tasks, no task precedences in the abstract What are the scheduling issues? I/O constraints Need for meaningful partial results multiple stages of post-processing APST – AppLeS Parameter Sweep Template • Parameter Sweeps = class of applications which are structured as multiple instances of an “experiment” with distinct parameter sets Berman, UCSD

  9. APST Scheduling Model • Large shared files, if any, must be stored strategically • Post-processing must minimize file transfers • Adaptive scheduling necessary to account for changing environment Berman, UCSD

  10. Computation Computation APST Scheduling Approach • Contingency Scheduling: Allocation developed by dynamically generating a Gantt chart for scheduling unassigned tasks between scheduling events • Basic skeleton • Compute the next scheduling event • Create a Gantt Chart G • For each computation and file transfer currently underway, compute an estimate of its completion time and fill in the corresponding slots in G • Select a subset T of the tasks that have not started execution • Until each host has been assigned enough work, heuristically assign tasks to hosts, filling in slots in G • Implement schedule Network links Hosts(Cluster 1) Hosts(Cluster 2) Resources 1 2 1 2 1 2 Scheduling event Time Scheduling event G

  11. APST Scheduling Heuristics Scheduling Algorithms for APST Applications • Self-scheduling Algorithms • Min-min • Max-min • Sufferage, • XSufferage • Gantt chart heuristics: • MinMin, MaxMin • Sufferage, XSufferage • ... • Self-scheduling Algorithms • workqueue • workqueue w/ work stealing • workqueue w/ work duplication • ...  Easy to implement and quick  No need for performance predictions  Insensitive to data placement  More difficult to implement  Needs performance predictions  Sensitive to data placement • Simulation results (HCW ’00 paper) show that: • Heuristics are worth it • Xsufferage is good heuristic even when predictions are bad • Complex environments require better planning (Gantt chart) Berman, UCSD

  12. scheduler API Workqueue Gantt chart heuristic algorithms Workqueue++ XSufferage MinMin Sufferage MaxMin actuate report actuate retrieve transport API execution API metadata API GASS IBP GRAM NetSolve NWS NFS Condor, Ninf, Legion,.. transfer execute query Legion IBP Globus Ninf Condor NWS NetSolve APST Architecture Command-line client APST Client interacts Controller triggers Scheduler APST Daemon Actuator Metadata Bookkeeper store Grid Resourcesand Middleware Berman, UCSD

  13. APST • APST being used for • INS2D (NASA Fluid Dynamics application) • MCell (Salk, Biological Molecular Modeling application) • Tphot (SDSC, Proton Transport application) • NeuralObjects (NSI, Neural Network simulations) • CS simulation applications for our own research (Model validation) • Actuator’s APIs are interchangeable and mixable • (NetSolve+IBP) + (GRAM+GASS) + (GRAM+NFS) • Scheduler allows for dynamic adaptation, multithreading • No Grid software is required • However lack of it (NWS, GASS, IBP) may lead to poorer performance • More details in SC’00 paper • Will be released in next 2 months to PACI, IPG users Berman, UCSD

  14. University of California, San Diego GRAM + GASS University of Tennessee, Knoxville NetSolve + IBP APST Validation Experiments Tokyo Institute of Technology NetSolve + IBP APST Daemon APST Client NetSolve + NFS Berman, UCSD

  15. MCell= General simulator for cellular microphysiology Uses Monte Carlo diffusion and chemical reaction algorithm in 3D to simulate complex biochemical interactions of molecules Focus of new multi-disciplinary ITR project Will focus on large-scale execution-time computational steering , data analysis and visualization APST Test Application – MCell

  16. workqueue Gantt-chart algs Experimental Results • Experimental Setting: • Mcell simulation with 1,200 tasks: • composed of 6 Monte-Carlo simulations • input files: 1, 1, 20, 20, 100, and 100 MB • 4 scenarios: • Initially • (a) all input files are only in Japan • (b) 100MB files replicated in California • (c) in addition, one 100MB file • replicated in Tennessee • (d) all input files replicated everywhere Berman, UCSD

  17. Grid programs Can reasonably obtain some information about environment (NWS predictions, MDS, HBM, …) Can assume that login, authentication, monitoring, etc. available on target execution machines Can assume that programs run to completion on execution platform Mega-programs Cannot assume any information about target environment Must be structured to treat target device as unfriendly host (cannot assume ambient services) Must be structured for “throwaway” end devices Must be structured to run continuously New Directions: “Mega-programming” Berman, UCSD

  18. Success with Mega-programming • Seti@home • Over 2 million users • Sustains teraflop computing • Can we run non-embarrassingly parallel codes successfully at this scale? • Computational Biology, Genomics … • Genome@home Berman, UCSD

  19. Genome@home • Joint work with Derrick Kondo • Application template for peer-to-peer platforms • First algorithm (Needleman-Wunsch Global Alignment) uses dynamic programming • Plan is to use template with additional genomics applications • Being developed for “web” rather than Grid environment Optimal alignments determined by traceback Berman, UCSD

  20. Algorithm 1 Algorithm 2 Mega-programs • Provide the algorithmic/application counterpart for very large scale platforms • peer-to-peer platforms, Entropia, etc. • Condor flocks • Large “free agent” environments • Globus • New platforms: networks of low-level devices, etc. • Different computing paradigm than MPP, Grid Genome@home DNAAlignment … Condor Entropia free agents Globus Berman, UCSD

  21. Coming soon to a computer near you: Release of APSTv0.1by SC’00 Release of AMWAT (AppLeS Master/ Worker Application Template) v0.1 by Jan ‘01 First prototype of genome@home: 2001 AppLeS software and papers: http://apples.ucsd.edu Thanks! NSF, NPACI, NASA IPG, TITECH, UTK Grid Computing Lab: Fran Berman (berman@cs.ucsd.edu) Henri Casanova Walfredo Cirne Holly Dail Marcio Faerman Jim Hayes Derrick Kondo Graziano Obertelli Gary Shao Otto Sievert Shava Smallen Alan Su Renata Teixeira Nadya Williams Eric Wing Qiao Xin Berman, UCSD

More Related