1 / 14

SP3.1: High-Performance Distributed Computing

SP3.1: High-Performance Distributed Computing. the Ibis Java-centric grid middleware. The KOALA grid scheduler. and. Henri Bal, Thilo Kielmann, Jason Maassen, Rob van Nieuwpoort, et al. Dick Epema Catalin Dumitrescu, Alex Iosup, Hashim Mohamed, Ozan Sonmez. TUDelft: KOALA.

conway
Télécharger la présentation

SP3.1: High-Performance Distributed Computing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SP3.1: High-Performance Distributed Computing the Ibis Java-centric grid middleware The KOALA grid scheduler and Henri Bal, Thilo Kielmann, Jason Maassen, Rob van Nieuwpoort, et al. Dick Epema Catalin Dumitrescu, Alex Iosup, Hashim Mohamed, Ozan Sonmez

  2. TUDelft: KOALA • KOALA is a multicluster/grid scheduler • Main goals of KOALA: • Load sharing of jobs across the sites in a grid: • Automatic resource selection • Co-allocation of jobs across the sites in a grid: • In order to use more resources • As dictated by the structure of applications (e.g., simulation/visualization) • KOALA has been released on the DAS in september 2005

  3. KOALA: Scheduling global queue KOALA local queues with local schedulers load sharing LS LS LS co-allocation clusters global job local jobs

  4. VU: Ibis • Ibis: Java-centric grid middleware for distributed supercomputing • Satin: divide-and-conquer parallelism in grids • GAT: Grid Application Toolkit • Implemented several Java applications from • SP 1.3 (Medical/Vumc) • SP 1.6 (Telescience/AMOLF) • SP 2.1 (iPSE/ UvA) • SP 2.2 (AID/UvA)

  5. Ibis: Grid’5000 experiments • Grid’5000: French computer scienceGrid with 2000 nodes at 9 sites • Used Grid’5000 for • Running Satin applications • Nqueens challenge (2nd Grid Plugtest) Ibis/Satin/GAT application running on 960 nodes at 6 sites, ~85% efficiency • Large-scale peer-to-peer experiments using Zorilla (Gnutella-like latency-based flooding of ads for joining a compution)

  6. KOALA feature 1: the Runners • There are many ugly application types out there • No way they can all be supported by a single scheduler • Solution: runners (=interface modules) • Currently supported: • Any type of single-component job • MPI/DUROC jobs • Ibis jobs • HOC applications runner

  7. KOALA feature 2: the policies • Originally supported co-allocation policies: • Worst-Fit: balance job components across sites • Close-to-Files: take into account the locations of input files to minimize transfer times • Different application types require different ways of component placement • So: • Modular structure with pluggable policies • Take into account internal communication structure of applications

  8. KOALA feature 3: support for HOCs • Higher-Order Components: • Pre-packaged software components with generic patterns of parallel behavior • Patterns: master-worker, pipelines, wavefront • Benefits: • Facilitates parallel programming in grids • Enables user-transparent scheduling in grids • Most important additional middleware: • Translation layer that builds a performance model from the HOC patterns and the user-supplied application parameters • Supported by KOALA (with Univ. of Münster) • Initial results: up to 50% reduction in runtimes

  9. TUDelft: GrenchMark • GrenchMark is a flexible grid workload generator, submitter, and results analyzer • Main goals of GrenchMark: • Generic workload definition for many types of workloads and application characteristics • Grid workload generation • Submitting and replaying workloads in different grid settings • GrenchMark released in november 2005 • GrenchMark used to test KOALA

  10. KOALA future (1) • Support for more applications types, e.g., • Workflows • Parameter sweep applications • Communication-aware and application-aware scheduling policies: • Take into account the communication pattern of applications when co-allocating • Also schedule bandwidth (in DAS3) • Better interface KOALA-local schedulers • KOALA is too nice

  11. KOALA future (2) • Peer-to-peer structure instead of hierarchical grid scheduler • Support heterogeneity • DAS3 • DAS2 + DAS3 • PoC • DAS3 + Grid’5000

  12. CPU’s R CPU’s R CPU’s R NOC CPU’s R CPU’s R KOALA and Ibis future DAS-3

  13. Conclusions • SP3.1 is well on track • SP3.1 has delivered reliable software tools for everybody to use: • KOALA/Grenchmark • Ibis/Satin • SP3.1 has a bright future • Still many research challenges • (Access to) great new heterogeneous testbeds

  14. More information • Web sites: • www.st.ewi.tudelft.nl/koala: • general description • KOALA tutorial • papers • grenchmark.st.ewi.tudelft.nl: • general description • download • papers • www.cs.vu.nl/ibis: • Ibis distribution • documentation • papers

More Related