260 likes | 359 Vues
Explore the computerized business process automation using Java CoG Kit, Condor job management, and Globus Gram for high throughput computing. Learn job submission techniques, scheduling policies, and fault tolerance strategies.
E N D
Job SubmissionCondor, Globus, Java CoG Kit Young Suk Moon
Workflow • The computerized facilitation or automation of a business process, in whole or part. • WiT = ( GrT, GsT, QuT, WmT ) • Wi = Workflow instantiation • Gr = Grid resources • Gs = Grid services • Qu = Quality expectations from the user • Wm = Workflow model. von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing. http://grid.rit.edu
Condor • High throughput computing system • Job management • Scheduling policy • Priority scheme • Resource monitoring / management • Checkpointing • Fault tolerance • Migration http://grid.rit.edu
Condor • Kernel architecture Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu
Condor • Job submission in a condor pool Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu
Condor • Gateway flocking Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu
Condor-G • Interacting with GRAM • Disadvantage – the agent does not know which resource is available in the foreign system Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu
Condor – Gliding In • Step 1 – submit Condor servers as jobs Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu
Condor – Gliding In • Step 2 – submitted servers form an ad-hoc Condor pool with the matchmaker Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu
Condor – Gliding In • Step 3 – use the Condor pool as normal Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu
Condor - Problem Solvers • Master-Worker • Master process assigns jobs to worker processes • Suitable for a large problem with no dependency • DAGMan (The Directed Acyclic Graph Manager) • For multiple jobs with dependancy A Directed Acyclic Graph Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. http://grid.rit.edu
Condor Universes • Define execution environments • Standard (default) : checkpointing & remote system calls • Vanila : run programs which cannot be re-linked • PVM : run Parallel Virtual Machine programs • MPI : run MPICH programs • Globus : use a Globus system • Java : run on JVM • Scheduler : executes jobs without waiting to be matched to resources. Used for DAGMan http://grid.rit.edu
Condor • Example submit description file • submit.hello Executable = hello Universe = standard Output = hello.out Log = hello.log Queue • To submit the job • > condor_submitsubmit.hello http://grid.rit.edu
Globus GRAM • Grid Resource Allocation and Management • Program execution requirements • State • User executables • Staging of input and output • Streaming output • Control • Scheduler • Monitoring http://grid.rit.edu
Globus GRAM • State transition diagram for GRAM jobs Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4. http://grid.rit.edu
Globus GRAM • Client APIs • Four portTypes: Managed Job Factory, Managed Job, Delegation Factory, Delegation operations interfaces resources Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu
Globus GRAM • GT4 Container • Hosts Web Services Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu
Globus GRAM • GRAM implementation structure Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu
Globus GRAM • Job submission % globusrun-ws -submit -job-command /bin/program arg Submitting job…Done Job ID: uuid:c51fe35a-4fa3-11d9-9cfc-000874404099 Termination time: 12/17/2004 20:47 GMT Current job state: Active Current job state: CleanUp Current job state: Done Destroying job...Done. Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu
Globus GRAM • Job submission with job description files % globusrun-ws -submit -job-description-file touch.xml -factory-epr-file gram.epr #touch.xml <job> <executable>/bin/program</executable> <argument>arg</argument> </job> #gram.epr <factoryEndpointxmlns:gram="http://www.globus.org/namespaces/2004/10/gram/job" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing"> <wsa:Address> https://viz-login.isi.edu:9000/wsrf/services/ManagedJobFactoryService </wsa:Address> <wsa:ReferenceProperties> <gram:ResourceID>PBS</gram:ResourceID> </wsa:ReferenceProperties> </factoryEndpoint> Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 http://grid.rit.edu
Java CoG Kit • Java Commodity Grid Toolkit • Maps Grid functionality into a commodity framework (Java) • Extends the use of Globus • Purpose • To use Grid services more efficiently by using / developing Java classes http://grid.rit.edu
Java CoG Kit • Workflow management layer • It supports multiple Grid middleware Application Nano Materials, Bio-Informatics, etc. Workflow monitoring Workflow Portlet & visualizer/Editor Task & workflow management Karajan (GridAnt, ant) Job submission file transfer & authentication Workflow Abstractions, Queue/Set Abstractions Grid Middleware + CoG Kit providers GT2, SSH, GT4 WS-RF, WebDAV http://grid.rit.edu
Java CoG Kit - Job Submission (GUI) http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf http://grid.rit.edu
Java CoG Kit – Command Line • Job execution • globusrun • File transfer • globus-url-copy • GASS (Global Access to Secondary Storage) • globus-gass-server • globus-gass-server-shutdown http://grid.rit.edu
Java CoG Kit - Shell http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf http://grid.rit.edu
Referances • Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356. • http://www.cs.wisc.edu/condor/manual/v6.4/2_4Road_map_Running.html • http://www.cs.wisc.edu/condor/quick-start.html • Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4 • von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing. • http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf • von Laszewski, G., Foster, I., Gawor, J., & Lane, P. (2001). A Java Commodity Grid Kit. Concurrency and Computation: Practice and Experience, 13(8-9), 643-662. http://grid.rit.edu