1 / 26

Job Submission Condor, Globus , Java CoG Kit

Explore the computerized business process automation using Java CoG Kit, Condor job management, and Globus Gram for high throughput computing. Learn job submission techniques, scheduling policies, and fault tolerance strategies.

stu
Télécharger la présentation

Job Submission Condor, Globus , Java CoG Kit

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Job SubmissionCondor, Globus, Java CoG Kit Young Suk Moon

  2. Workflow • The computerized facilitation or automation of a business process, in whole or part. • WiT = ( GrT, GsT, QuT, WmT ) • Wi = Workflow instantiation • Gr = Grid resources • Gs = Grid services • Qu = Quality expectations from the user • Wm = Workflow model. von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing. http://grid.rit.edu

  3. Condor • High throughput computing system • Job management • Scheduling policy • Priority scheme • Resource monitoring / management • Checkpointing • Fault tolerance • Migration http://grid.rit.edu

  4. Condor • Kernel architecture Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu

  5. Condor • Job submission in a condor pool Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu

  6. Condor • Gateway flocking Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu

  7. Condor-G • Interacting with GRAM • Disadvantage – the agent does not know which resource is available in the foreign system Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu

  8. Condor – Gliding In • Step 1 – submit Condor servers as jobs Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu

  9. Condor – Gliding In • Step 2 – submitted servers form an ad-hoc Condor pool with the matchmaker Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu

  10. Condor – Gliding In • Step 3 – use the Condor pool as normal Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu

  11. Condor - Problem Solvers • Master-Worker • Master process assigns jobs to worker processes • Suitable for a large problem with no dependency • DAGMan (The Directed Acyclic Graph Manager) • For multiple jobs with dependancy A Directed Acyclic Graph Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu

  12. Condor Universes • Define execution environments • Standard (default) : checkpointing & remote system calls • Vanila : run programs which cannot be re-linked • PVM : run Parallel Virtual Machine programs • MPI : run MPICH programs • Globus : use a Globus system • Java : run on JVM • Scheduler : executes jobs without waiting to be matched to resources. Used for DAGMan http://grid.rit.edu

  13. Condor • Example submit description file • submit.hello Executable = hello Universe = standard Output = hello.out Log = hello.log Queue • To submit the job • > condor_submitsubmit.hello http://grid.rit.edu

  14. Globus GRAM • Grid Resource Allocation and Management • Program execution requirements • State • User executables • Staging of input and output • Streaming output • Control • Scheduler • Monitoring http://grid.rit.edu

  15. Globus GRAM • State transition diagram for GRAM jobs Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4. http://grid.rit.edu

  16. Globus GRAM • Client APIs • Four portTypes: Managed Job Factory, Managed Job, Delegation Factory, Delegation operations interfaces resources Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu

  17. Globus GRAM • GT4 Container • Hosts Web Services Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu

  18. Globus GRAM • GRAM implementation structure Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu

  19. Globus GRAM • Job submission % globusrun-ws -submit -job-command /bin/program arg Submitting job…Done Job ID: uuid:c51fe35a-4fa3-11d9-9cfc-000874404099 Termination time: 12/17/2004 20:47 GMT Current job state: Active Current job state: CleanUp Current job state: Done Destroying job...Done. Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu

  20. Globus GRAM • Job submission with job description files % globusrun-ws -submit -job-description-file touch.xml -factory-epr-file gram.epr #touch.xml <job> <executable>/bin/program</executable> <argument>arg</argument> </job> #gram.epr <factoryEndpointxmlns:gram="http://www.globus.org/namespaces/2004/10/gram/job" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing"> <wsa:Address> https://viz-login.isi.edu:9000/wsrf/services/ManagedJobFactoryService </wsa:Address> <wsa:ReferenceProperties> <gram:ResourceID>PBS</gram:ResourceID> </wsa:ReferenceProperties> </factoryEndpoint> Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu

  21. Java CoG Kit • Java Commodity Grid Toolkit • Maps Grid functionality into a commodity framework (Java) • Extends the use of Globus • Purpose • To use Grid services more efficiently by using / developing Java classes http://grid.rit.edu

  22. Java CoG Kit • Workflow management layer • It supports multiple Grid middleware Application Nano Materials, Bio-Informatics, etc. Workflow monitoring Workflow Portlet & visualizer/Editor Task & workflow management Karajan (GridAnt, ant) Job submission file transfer & authentication Workflow Abstractions, Queue/Set Abstractions Grid Middleware + CoG Kit providers GT2, SSH, GT4 WS-RF, WebDAV http://grid.rit.edu

  23. Java CoG Kit - Job Submission (GUI) http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf http://grid.rit.edu

  24. Java CoG Kit – Command Line • Job execution • globusrun • File transfer • globus-url-copy • GASS (Global Access to Secondary Storage) • globus-gass-server • globus-gass-server-shutdown http://grid.rit.edu

  25. Java CoG Kit - Shell http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf http://grid.rit.edu

  26. Referances • Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. • http://www.cs.wisc.edu/condor/manual/v6.4/2_4Road_map_Running.html • http://www.cs.wisc.edu/condor/quick-start.html • Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 • von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing. • http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf • von Laszewski, G., Foster, I., Gawor, J., & Lane, P. (2001). A Java Commodity Grid Kit. Concurrency and Computation: Practice and Experience, 13(8-9), 643-662. http://grid.rit.edu

More Related