1 / 76

Condor by Example

Discover the power of Condor in transforming disparate workstations into a unified high-throughput computing facility. Learn the process of submitting jobs, configuring class ads, and utilizing numerous machines effectively. Explore crucial concepts like matchmaking, which ensures job and resource compatibility, while accommodating varying constraints and conditions. Understand the dynamic nature of machine pools, job running procedures, and receive essential feedback for performance tracking. This overview provides a practical guide for maximizing computational throughput and reliability in your projects.

hal
Télécharger la présentation

Condor by Example

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Condor by Example

  2. Outline • Overview • Submitting Jobs, Getting Feedback • Setting Requirements with ClassAds • Using LOTS of Machines • Which Universe? • Conclusion

  3. What is Condor? • Condor converts a collection of unrelated workstations into a high-throughput computing facility. • Condor uses matchmaking to make sure that everyone is happy.

  4. What is High-Throughput Computing? • High-performance: CPU cycles/second under ideal circumstances. • “How fast can I run simulation X on this machine?” • High-throughput: CPU cycles/day (week, month, year?) under non-ideal circumstances. • “How many times can I run simulation X in the next week using all available machines?”

  5. What is High-Throughput Computing? • Condor does whatever it takes to run your jobs, even if some machines… • Crash! • Are disconnected • Run out of disk space • Are removed or added from the pool • Are put to other uses

  6. What is Matchmaking? • Condor uses Matchmaking to make sure that work gets done within the constraints of both users and owners. • Users (jobs) have constraints: • “I need an Alpha with 256 MB RAM” • Owners (machines) have constraints: • “Only run jobs when I am away from my desk and never run jobs owned by Bob.”

  7. “What can Condordo for me?” Condor can… • …increase your throughput. • …do your housekeeping. • …improve reliability. • …give performance feedback.

  8. The INFN Condor Pool

  9. How many machines now? • The map is out of date! • The system is always changing. • First example: What machines (and of what kind) are in the pool now?

  10. First Things First • Set your path: • setenv PATH /library/condor_nfs/XXX/bin • XXX should be your system: • OSF1, LINUX, SOLARIS26, HPUX10 …

  11. How Many Machines? % condor_status Name OpSys Arch State Activity LoadAv Mem lxpc1.na.infn LINUX-GLIBC INTEL Unclaimed Idle 0.000 30 axpd21.pd.inf OSF1 ALPHA Owner Idle 0.266 96 vlsi11.pd.inf SOLARIS26 SUN4u Claimed Busy 0.000 256 . . . Machines Owner Claimed Unclaimed Matched Preempting ALPHA/OSF1 115 67 46 1 0 1 INTEL/LINUX 53 18 0 35 0 0 INTEL/LINUX-GLIBC 16 7 0 9 0 0 SUN4u/SOLARIS251 1 1 0 0 0 0 SUN4u/SOLARIS26 6 2 0 4 0 0 SUN4u/SOLARIS27 1 1 0 0 0 0 SUN4x/SOLARIS26 2 1 0 1 0 0 Total 194 97 46 50 0 1

  12. Machine States • Most machines will be: • Owner: • The machine’s owner is busy at the console, so no Condor jobs may run. • Claimed: • Condor has selected the machine to run jobs for other users.

  13. Machine States • Only a few should be: • Unclaimed: • The owner is gone, but Condor has not yet selected the machine. • Matched: • Between claimed and unclaimed. • Preempting: • Condor is busy removing a job.

  14. More Examples % condor_status -help % condor_status –avail % condor_status –run % condor_status –total % condor_status –pool condor.cs.wisc.edu

  15. Submitting Jobs

  16. Steps to Running a Job • Re-link for Condor. • Submit the job. • Watch the progess. • Receive email when done.

  17. Example Job Compute the nth Fibonnaci number. Fib(40) takes about one minute to compute on an Alpha. % ./fib 40 fib(40) = 102334155

  18. #include <stdio.h> #include <stdlib.h> int fib( int x ) { if( x<=0 ) return 0; if( x==1 ) return 1; return fib(x-1) + fib(x-2); } int main(int argc, char *argv[]) { int n; n = atoi(argv[1]); printf ("fib(%d) = %d\n",n,fib(n)); return 0; }

  19. Re-link for Condor • Normal compile: • gcc –c fib.c –o fib.o • Normal link: • gcc fib.o –o fib • Use the same command, but add condor_compile: • condor_compile gcc fib.o –o fib

  20. Submit the Job • Create a submit file: • vi fib.submit • Submit the job: • condor_submit fib.submit Executable = fib Arguments = 40 Output = fib.out Log = fib.log queue

  21. Watch the Progress % condor_q -- Submitter: axpbo8.bo.infn.it : <131.154.10.29:1038> : ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 5.0 thain 6/21 12:40 0+00:00:15 R 0 2.5 fib 40 Each job gets a unique number. Status: Unexpanded, Running or Idle Size of program image (MB)

  22. Receive E-mail When Done This is an automated email from the Condor system on machine "axpbo8.bo.infn.it". Do not reply. Your condor job /tmp_mnt/usr/users/ccl/thain/test/fib 40 exited with status 0. Submitted at: Wed Jun 21 14:24:42 2000 Completed at: Wed Jun 21 14:36:36 2000 Real Time: 0 00:11:54 Run Time: 0 00:06:52 Committed Time: 0 00:01:37 . . .

  23. Running Many Processes • 100 processes are almost as easy as !. • Each condor_submit makes one cluster of one or more processes. • Add the number of processes to run to the Queue statement. • Use the $(PROCESS) variable to give each process slightly different instructions.

  24. Running Many Processes • Compute Fib(1) through Fib(50) • Output goes in fib.out.1, fib.out.2, and so on… Executable = fib Arguments = $(PROCESS) Output = fib.out.$(PROCESS) Log = fib.log Queue 50

  25. Running Many Processes • Another approach: Each process gets its own directory (dir1, dir2, …) and sends the output to dirX/fib.out. Executable = fib Arguments = $(PROCESS) Initial_Dir = dir$(PROCESS) Output = fib.out Log = fib.log Queue 50

  26. Running Many Processes % condor_q -- Submitter: axpbo8.bo.infn.it : <131.154.10.29:1038> ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 9.3 thain 6/23 10:47 0+00:05:40 R 0 2.5 fib 3 9.6 thain 6/23 10:47 0+00:05:11 R 0 2.5 fib 6 9.7 thain 6/23 10:47 0+00:05:09 R 0 2.5 fib 7 . . . 21 jobs; 2 idle, 19 running, 0 held Cluster number Process number

  27. Where Are They Running? • condor_q –run • Submitter: axpbo8.bo.infn.it : <131.154.10.29:1038> : ID OWNER SUBMITTED RUN_TIME HOST(S) 9.47 thain 6/23 10:47 0+00:07:03 ax4bbt.bo.infn.it 9.48 thain 6/23 10:47 0+00:06:51 pewobo1.bo.infn.it 9.49 thain 6/23 10:47 0+00:06:30 osde01.pd.infn.it Current Location

  28. Help! I’m buried in Email! • By default, Condor sends one email for each completed process. • Add these to your submit file: • notification = error • notification = never • To send it to someone else: • notify_user = mazzanti@bo.infn.it

  29. Removing Processes • Remove one process: • condor_rm 9.47 • Remove a whole cluster: • condor_rm 9 • Remove everything! • condor_rm -a

  30. Getting Feedback

  31. What have I done? • The user log file (fib.log) shows a chronological list of everything important that happened to a job. 001 (007.035.000) 06/21 17:03:44 Job executing on host: <140.105.6.155:2219> 004 (007.035.000) 06/21 17:04:58 Job was evicted. 009 (007.035.000) 06/21 17:05:10 Job was aborted by the user.

  32. What have I done? % condor_history ID OWNER SUBMITTED CPU_USAGE ST COMPLETED CMD 9.3 thain 6/23 10:47 0+00:00:00 C 6/23 10:58 fib 3 9.40 thain 6/23 10:47 0+00:00:24 C 6/23 10:59 fib 40 9.10 thain 6/23 10:47 0+00:00:00 C 6/23 11:01 fib 10 9.47 thain 6/23 10:47 0+00:05:45 C 6/23 11:01 fib 47 9.7 thain 6/23 10:47 0+00:00:00 C 6/23 11:01 fib 7

  33. Brief I/O Summary % condor_q –io -- Schedd: c01.cs.wisc.edu : <128.105.146.101:2016> ID OWNER READ WRITE SEEK XPUT BUFSIZE BLKSIZE 756.15 joe 244.9 KB 379.8 KB 71 1.3 KB/s 512.0 KB 32.0 KB 758.24 joe 198.8 KB 219.5 KB 78 45.0 B /s 512.0 KB 32.0 KB 758.26 joe 44.7 KB 22.1 KB 2727 13.0 B /s 512.0 KB 32.0 KB 3 jobs; 0 idle, 3 running, 0 held

  34. Complete I/O Summaryin Email Your condor job "/usr/joe/records.remote input output" exited with status 0. Total I/O: 104.2 KB/s effective throughput 5 files opened 104 reads totaling 411.0 KB 316 writes totaling 1.2 MB 102 seeks I/O by File: buffered file /usr/joe/input opened 2 times 100 reads totaling 398.6 KB 311 write totaling 1.2 MB 101 seeks (Only since Condor Version 6.1.11)

  35. Complete I/O Summaryin Email • The summary helps identify performance problems. Even advanced users don't know exactly how their programs and libraries operate.

  36. Complete I/O Summary in Email • Example: • CMSSIM - physics analysis program. • “Why is this job so slow?” • Data summary: • read 250 MB from 20 MB file. • Very high SEEK total -> random access. • Solution: Increase buffer to 20 MB.

  37. Who Uses Condor? % condor_q –global -- Schedd: to02xd.to.infn.it : <192.84.137.2:1030> ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 127.0 garzelli 6/21 18:45 1+14:18:16 R 0 17.2 tosti2trisdn -- Schedd: quark.ts.infn.it : <140.105.6.101:3908> ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 600.0 dellaric 4/10 14:57 55+09:20:31 R 0 9.1 john p2.dat 665.0 dellaric 6/2 11:14 20+03:27:30 R 0 9.2 john p1.dat 788.0 pamela 6/20 09:27 3+04:41:43 R 0 15.4 montepamela

  38. Who uses Condor? % condor_status –submitters Name Machine Running IdleJobs MaxJobsRunning rebuzzin@pv.infn.it decux1.pv. 22 34 200 pamela@ts.infn.it quark.ts.i 6 1 200 giunti@to.infn.it to05xd.to. 21 49 200 . . . RunningJobs IdleJobs cattaneo@pv.infn.it 0 1 pamela@ts.infn.it 6 1 rebuzzin@pv.infn.it 22 34 Total 59 86

  39. Who Uses Condor? % condor_userprio Last Priority Update: 6/23 16:27 Effective User Name Priority ------------------------------ --------- meucci@pv.infn.it 0.50 longof@ts.infn.it 0.50 thain@bo.infn.it 0.50 dellaric@ts.infn.it 2.00 clueoff@pd.infn.it 3.00 pamela@ts.infn.it 5.81 rebuzzin@pv.infn.it 18.18 giunti@to.infn.it 19.72 ------------------------------ --------- Number of users shown: 8

  40. Who Uses Condor? • The user priority is computed by Condor to estimate how much of the pool’s CPU resources have been used by each submitter. • Lighter users receive a lower priority: they will be allocated CPUs before heavy users. • Users consuming the same amount of CPU will be allocated an equal amount.

  41. Measuring Goodput • Goodput is the amount of time a workstation spends making forward progress on work assigned by Condor. • This is a big topic all by itself: http://www.cs.wisc.edu/condor/goodput

  42. Measuring Goodput % condor_q –goodput -- Submitter: coral.cs.wisc.edu : <128.105.175.116:45697> : coral.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME GOODPUT CPU_UTIL Mb/s 719.74 thain 6/23 07:35 2+20:47:59 100.0% 87.6% 0.00 719.75 thain 6/23 07:35 2+20:38:45 40.5% 99.8% 0.00 719.76 thain 6/23 07:35 2+20:38:16 96.9% 98.7% 0.00 719.77 thain 6/23 07:35 2+21:10:06 100.0% 99.8% 0.00

  43. Setting Requirementswith ClassAds

  44. Setting Requirements • We believe that Condor must allow both users (jobs) and owners (machines) to set requirements. • This is an absolute necessity in order to convince people to participate in the community.

  45. ClassAds • ClassAds are a simple language for describing both the properties and the requirements of jobs and machines. • Condor stores nearly everything in ClassAds -- use the –l option to condor_q and condor_submit to get the full details.

  46. ClassAd for a Machine • condor_status –l axpbo8 MyType = "Machine" TargetType = "Job" Name = "axpbo8.bo.infn.it" START = TRUE VirtualMemory = 342696 Disk = 28728536 Memory = 160 Cpus = 1 Arch = "ALPHA" OpSys = "OSF1“

  47. ClassAd for a Job • condor_q –l 9.49 MyType = "Job" TargetType = "Machine" Owner = "thain" Cmd = "/tmp_mnt/usr/users/ccl/thain/test/fib" Out = “fib.out.49” Args = “49” ImageSize = 2544 DiskUsage = 2544 Requirements = (Arch == "ALPHA") && (OpSys == "OSF1") && (Disk >= DiskUsage) && (VirtualMemory >= ImageSize)

  48. Default Requirements • By default, Condor assumes the requirements for your job are: “I need a machine with…” • The same operating system and architecture as my workstation. • Enough disk to store the program. • Enough virtual memory to run the program.

  49. Default Requirements • Expressed in ClassAds as: Requirements = (Arch ==“ALPHA”) && (OpSys==“OSF1”) && (Disk >= DiskUsage) && (VirtualMemory >= ImageSize)

  50. ClassAd Requirements • Similar to C/C++/Java expressions: • Symbols: Arch, OpSys, Memory, Mips • Values: 15, 6.5, “LINUX” • Operators: • ==, <, >, <=, >= • &&, || • ( )

More Related