1 / 23

New Development in the AppLeS Project or User-Level Middleware for the Grid

New Development in the AppLeS Project or User-Level Middleware for the Grid. Francine Berman University of California, San Diego. The Evolving Grid.

yakov
Télécharger la présentation

New Development in the AppLeS Project or User-Level Middleware for the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. New Development in the AppLeS ProjectorUser-Level Middleware for the Grid Francine Berman University of California, San Diego

  2. The Evolving Grid In the beginning, there were applications and resources, and it took ninja programmers andmany months to implement the applications on the Grid … Applications Resources

  3. The Evolving Grid And behold, there were services, and programmers saw that it was good (even though their performancewas still often less than desirable) … Applications Applications Grid Middleware Resources Resources

  4. The Evolving Grid … and it came to pass that user-level middleware was promised to promote the performance of Grid applications, and the users rejoiced … Applications Applications User-Level Middleware Applications Grid Middleware Grid Middleware Resources Resources Resources

  5. Applications User-Level Middleware Grid Middleware Resources The Middleware Promise • Grid Middleware • Provides infrastructure/services to enable usability of the Grid • Promotes portability and retargetability • User-level Middleware • Hides the complexity of the Grid for the end-user • Adaptsto dynamic resource performance variations • Promotes application performance

  6. AppLeS = Application-Level Scheduler Joint project with R. Wolski AppLeS + application = self-scheduling Grid application AppLeS-enabled applications adapt to dynamic performance variations in Grid Resources AppLeS-enabledapplications Grid Middleware Resources How Do Applications Achieve Performance Now?

  7. accessible resources Resource Discovery feasible resource sets Resource Selection SchedulePlanningand PerformanceModeling evaluatedschedules DecisionModel “best” schedule Schedule Deployment AppLeS Architecture AppLeS-enabledapplications Grid Middleware Resources

  8. AppLeS agent integrated within application AppLeS-enabledapplications Applications User-Level Middleware Grid Middleware Grid Middleware Resources Resources From AppLeS-enabled applications to User-Level Middleware

  9. AppLeS User-Level Middleware • Focus is development of templates which • target structurally similar classes of applications • can be instantiated in a user-friendly timeframe • provide good application performance AppLeS Template Architecture Application Module Scheduling Module Deployment Module Grid Middleware and Resources

  10. APST – AppLeS Parameter Sweep Template • Parameter Sweeps = class of applications which are structured as multiple instances of an “experiment” with distinct parameter sets • Joint work with Henri Casanova • First AppLeS Middleware package to be distributed to users • Parameter Sweeps are common application structure used in various fields of science and engineering • Most notably: Simulations (Monte Carlo, etc.) • Large number of tasks, no task precedences in the general case easy scheduling ? • I/O constraints • Need for meaningful partial results • multiple stages of post-processing

  11. APST Scheduling Issues • Large shared files, if any, must be stored strategically • Post-processing must minimize file transfers • Adaptive scheduling necessary to account for changing environment

  12. Computation Computation Scheduling Approach • Contingency Scheduling: Allocation developed by dynamically generating a Gantt chart for scheduling unassigned tasks between scheduling events • Basic skeleton • Compute the next scheduling event • Create a Gantt Chart G • For each computation and file transfer currently underway, compute an estimate of its completion time and fill in the corresponding slots in G • Select a subset T of the tasks that have not started execution • Until each host has been assigned enough work, heuristically assign tasks to hosts, filling in slots in G • Implement schedule Network links Hosts(Cluster 1) Hosts(Cluster 2) Resources 1 2 1 2 1 2 Scheduling event Time Scheduling event G

  13. Scheduling Heuristics Scheduling Algorithms for PS Applications • Gantt chart heuristics: • MinMin, MaxMin • Sufferage, XSufferage • ... • Self-scheduling Algorithms • workqueue • workqueue w/ work stealing • workqueue w/ work duplication • ... Easy to implement and quick No need for performance predictions Insensitive to data placement More difficult to implement Needs performance predictions Sensitive to data placement • Simulation results (HCW ’00 paper) show that: • heuristics are worth it • Xsufferage is good heuristic even when predictions are bad • complex environments require better planning (Gantt chart)

  14. scheduler API Workqueue Gantt chart heuristic algorithms Workqueue++ XSufferage MinMin Sufferage MaxMin actuate report actuate retrieve transport API execution API metadata API GASS IBP GRAM NetSolve NWS NFS Condor, Ninf, Legion,.. transfer execute query Legion IBP Globus Ninf Condor NWS NetSolve APST Architecture Command-line client APST Client interacts Controller triggers Scheduler APST Daemon Actuator Metadata Bookkeeper store Grid Resourcesand Middleware

  15. APST • APST being used for • INS2D (NASA Fluid Dynamics application) • MCell (Salk, Molecular modeling for Biology) • Tphot (SDSC, Proton Transport application) • NeuralObjects (NSI, Neural network simulations) • CS simulation Applications for our own research (Model validation, long-range forecasting validation) • Actuator’s APIs are interchangeable and mixable • (NetSolve+IBP) + (GRAM+GASS) + (GRAM+NFS) • Scheduler API allows for dynamic adaptation • No Grid software is required • However lack of it (NWS, GASS, IBP) may lead to poorer performance • More details in SC’00 paper

  16. University of California, San Diego GRAM + GASS University of Tennessee, Knoxville NetSolve + IBP APST Validation Experiments Tokyo Institute of Technology NetSolve + IBP APST Daemon APST Client NetSolve + NFS

  17. MCell= General simulator for cellular microphysiology Uses Monte Carlo diffusion and chemical reaction algorithm in 3D to simulate complex biochemical interactions of molecules Focus of new multi-disciplinary ITR project Will focus on large-scale execution-time computational steering , data analysis and visualization APST Test Application – MCell

  18. workqueue Gantt-chart algs Experimental Results • Experimental Setting: • Mcell simulation with 1,200 tasks: • composed of 6 Monte-Carlo simulations • input files: 1, 1, 20, 20, 100, and 100 MB • 4 scenarios: • Initially • (a) all input files are only in Japan • (b) 100MB files replicated in California • (c) in addition, one 100MB file • replicated in Tennessee • (d) all input files replicated everywhere

  19. Grid programs Can reasonably obtain some information about environment (NWS predictions, MDS, HBM, …) Can assume that login, authentication, monitoring, etc. available on target execution machines Can assume that programs run to completion on execution platform Mega-programs Cannot assume any information about target environment Must be structured to treat target device as unfriendly host (cannot assume ambient services) Must be structured for “throwaway” end devices Must be structured to run continuously New Directions: “Mega-programming”

  20. Success with Mega-programming • Seti@home • Over 2 million users • Sustains teraflop computing • Can we run non-embarrassingly parallel codes successfully at this scale? • Computational Biology, Genomics … • Genome@home

  21. Genome@home • Joint work with Derrick Kondo • Application template for peer-to-peer platforms • First algorithm (Needleman-Wunsch Global Alignment) uses dynamic programming • Plan is to use template with additional genomics applications • Being developed for “web” rather than Grid environment Optimal alignments determined by traceback

  22. Algorithm2 Algorithm1 Mega-programs • Provide the algorithmic counterpart for very large scale platforms • peer-to-peer platforms, Entropia, etc. • Condor flocks • Large “free agent” environments • Globus • New platforms: networks of low-level devices, etc. • Different computing paradigm than MPP, Grid Genome@home DNAAlignment Legion Entropia Condor … Globus free agents

  23. Coming soon to a computer near you: Release of APSTv0.1by SC’00 Release of AMWAT (AppLeS Master/ Worker Application Template) v0.1 by Jan ‘01 First prototype of genome@home: 2001 AppLeS software and papers: http://apples.ucsd.edu Thanks! NSF, NPACI, NASA Grid Computing Lab: Fran Berman (berman@cs.ucsd.edu) Henri Casanova Walfredo Cirne Holly Dail Marcio Faerman Jim Hayes Derrick Kondo Graziano Obertelli Gary Shao Otto Sievert Shava Smallen Alan Su Renata Teixeira Nadya Williams Eric Wing Qiao Xin

More Related