1 / 36

Overview of CSF4 Meta-Scheduler: Features, Architecture, and Future Developments

The CSF4 Meta-Scheduler, developed by Jilin University, China, is an open-source project that provides a comprehensive solution for resource allocation and management in heterogeneous and dynamic computing environments. It supports various local schedulers like LSF, PBS, SGE, and Condor. Functionality includes submitting jobs without specifying clusters, monitoring, queuing services, and extensible scheduling policies. Users benefit from automatic credential delegation and data staging. The future of CSF4 aims at enhancing its scheduling framework and integrating additional workflow support.

mariam-luna
Télécharger la présentation

Overview of CSF4 Meta-Scheduler: Features, Architecture, and Future Developments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CSF4 Meta-Scheduler Name: Zhaohui Ding, Xiaohui Wei Email : Zhaohui.ding@email.jlu.edu.cn, weixh@jlu.edu.cn College of Computer Science & Technology Jilin University

  2. Agenda • Meta-Scheduler & CSF4 Introduction • CSF4 Architecture • CSF4 Functionalities • Future Work • Demo and Practice

  3. What is Meta-Scheduler • Resource Allocation & Management • Heterogeneous • Distributed • Dynamic • Local Scheduler VS Meta-scheduler

  4. Local Scheduler VS Meta-Scheduler

  5. Meta-Scheduler VS Local Scheduler • Local Scheduler • LSF (Load Sharing Facility) • PBS (Portable Batch System) • SGE (Sun Grid Engine) • Condor • IBM Loadleveler • Meta-Scheduler • CSF • Maui (Silver) • Gridway • Nimrod-G • Condor-G

  6. What is CSF4 • What is CSF Meta-Scheduler • Full Name: Community Scheduler Framework • CSF4 contains a group of grid services host in GT4 • CSF4 is a full WSRF compliant meta-scheduler. • Open Source project and can be accessed at http://sourceforge.net/projects/gcsf • Developed by Jilin University, China • CSF4 has been added to Globus Toolkit 4 as an Execution Component

  7. CSF4 in Globus Toolkit 4

  8. Meta-Scheduler VS Local Scheduler A typical deployment for Meta-Scheduler and Local Schedulers

  9. What CSF4 Can Do? • Basic Functionalities • Submit jobs to Grid without Specifying Cluster • Monitor and Control Jobs • Provide Queuing Service • Schedule jobs and resource by custom-built polices • CSF4 Portlet (A Web browser based User Interface)

  10. What CSF4 Can Do? (cont.) • Advanced Functionalities • Multiple Domains Resource Information Sharing • Automatic user credentials delegation • Automatic data-staging • Extensible scheduling policies • Supporting grid parallel jobs (MPICH-G2)

  11. CSF4 – Architecture

  12. CSF4 – Architecture User view

  13. CSF4 – Architecture (cont.) Resource View NOTE: One RM Gram is for one cluster only Local clusters

  14. CSF4 – Functionalities • Local Scheduler Supported • LSF • PBS • SGE • Condor • Infrastructure Supported • Globus Toolkit 4 • Globus Toolkit 2

  15. CSF4 – Functionalities Scheduling Plug-in Framework Designed For Queuing Service Provide A set of policies Customizable Extensible

  16. Existent Scheduling Policies • FCFS (First Come First Serve) round-robin • Default policy • Throttle • Restrict the number of jobs in a scheduling cycle • Array Job Plug-in • Design for life science applications (such as AutoDock, BLAST) • MPICH-G2 Plug-in • By using VJM, the plug-in guarantee the synchronized resource allocation can be successful • Data intensive applications plug-in (under-developing)

  17. Schedule plug-in & scheduling policies Each policy is implemented inside a scheduling plugin module A queue can load multiple plugin modules

  18. Array Job • AutoDock and Blast-like applications • A large number of sub-jobs. • Execute same binary • Different input/output files

  19. Array Job (cont.) • Advantages • Submit job only once • Save submission time and memory storage

  20. Resource Information Sharing • A MDS information provider for CSF4 • Multiple CSF4 can share the resource information

  21. CSF4 – Functionalities (cont.) Deploy Multiple CSF4 in a Grid Community

  22. User Credential Delegation • Automatic user credential delegation • GT4 Delegation Service • Full/Limited Delegation

  23. Automatic user credential delegation

  24. Data Staging • Manual Data Staging • Which clusters I can use? • Which clusters my jobs will running on? • Where is the output data? • When will the job finish, so that I can stage-out the output data?

  25. Automatic Data Staging • Stage-in and Stage-out without Meta-Scheduler

  26. Automatic Data Staging • With CSF4 Automatic Data Staging

  27. CSF4 – User interface (cont.) • Command line interface - Services available for grid users: • JobService csf-job-create, csf-job-start, csf-job-submit, csf-job-status, csf-job-stop, csf-job-resume, csf-job-list, csf-resource-list, csf-mpi-run • ReservationService csf-rsv-create, csf-rsv-status, csf-rsv-cancel csf-job-submit, … • QueuingService csf-queue-create, csf-queue-conf, csf-queue-data

  28. CSF4 Portlet • Web Browser based Interface • Friendly interface

  29. View Available Clusters

  30. Submit a job

  31. View Jobs history

  32. View Job output

  33. How to get CSF4? • Homepage: • http://sourceforge.net/projects/gcsf • Release Version (Recommended) : • http://sourceforge.net/project/showfiles.php?group_id=103105 • Developing Version (CVS, the newest but not stable) • http://sourceforge.net/cvs/?group_id=103105 • Globus Toolkit 4 Package : • In “contrib” directory

  34. Future Work • Plug-in Framework Improvement • More scheduling policies • Optimize for Parallel Job • Optimize for Data intensive job • Workflow supported

  35. Practices • View Credential Info • Query Available Resource • Submit A Simple Job • Submit An Array Job • Query Jobs History • View Job Output

  36. Thank you

More Related