1 / 20

Setting Up Grid Environment for Efficient Job Submission

Learn how to utilize Grid CPUs for job submissions with examples like running TPCana. Detailed steps include setting up the grid environment, submitting jobs, and optimizing file upload rates. Additional suggestions for efficient file server usage and job submission strategies are provided.

rianna
Télécharger la présentation

Setting Up Grid Environment for Efficient Job Submission

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Group MettingSubmit job to Grid 20070704 Grass

  2. outline • Why we need Grid CPUs • example : How to run TPCana • Before run TPCana

  3. Why we need Grid CPUs

  4. IPAS Condor

  5. LHC Taiwan Tier-1

  6. example : How to run TPCana

  7. example for setup grid environment STEP1: copy proxy file [grass@hepfarm2 ~]$ ls -l /usrX/grass/.globus/x509up_grass -rw------- 1 grass spring8 5562 Jul 2 16:26 /usrX/grass/.globus/x509up_grass \> chmod 600 x509up_grass STEP2 setup grid environment ## grid environment \> source /scratch/grid/glite-3.0.26/etc/profile.d/grid_env.csh \> setenv X509_USER_PROXY /usrX/grass/.globus/x509up_grass

  8. the data we need

  9. the data we need is too big

  10. generate drift table online Drift table production time generate drift table 5 times, we get the same drift table the same size no difference

  11. upload data rawdata size size of others data is small

  12. manual of “globus_submit” usage: globus_submit --tarFile=file --output=file --start=num --end=num ... --tarFile=file: path for tar file to be submitted e.g. ./submitme.tgz --outputFile=file: path for output file to be stored e.g. ./cronus.$.tgz --cmd=command: command to be executed, normally a shell script e.g. ./cronus.sh $ --start=num: beginning segment number e.g. 1 --end=num: ending segment number e.g. 100 --segments=num: segment number list e.g. 2-5,7,10 --VO=vo: VO name you belong to e.g. twgrid --verbose: verbose output --cronus: submit to Cronus, it's necessary in ipas003 --test: do not submit, show JDL to standard output --email: email address for summary output e.g. tsan@fnal.gov --globusscheduler: gatekeeper you wan to submit to e.g. lcg00125.grid.sinica.edu.tw:2119/jobmanager-lcgpbs --globusrsl: additional options to gatekeeper special character, $, will be replaced with segment number for --outputFile and --cmd

  13. globus_submit • globus_submit --tarFile=../submitme.tgz --outputFile='tpcana.$.tgz' --cmd='./runTPC.sh /hepdata4/jychen/TPC/data/2004Oct17 $' --segments=30525-30527,30529 --globusscheduler=lcg00125.grid.sinica.edu.tw:2119/jobmanager-lcgpbs --verbose --globusrsl='(queue=twgrid)'

  14. Before run TPCana

  15. grid copy rate upload 3.1G file need 85 second must of file small then 3G

  16. submit 27 run at 23:00 raid: /hepdata4 file server: asscratch2

  17. What we need to do later • Test the best number of upload files in the same time in our file server.(Tsanlung’s suggestion is 16 runs one time ) • Write a perl split to submit job.

  18. backup

  19. grid copy rate upload 1.2G file need 40 second must of file small then 3G

More Related