1 / 33

EDGI E uropean D esktop G rid I nitiative

EDGI E uropean D esktop G rid I nitiative. gLite job submission to EDGI EDGI is supported by the FP7 Capacities Programme under contract nr RI-261556. 2010-04-27. Contents. Single job submission to EDGI through gLite Metajob submission to EDGI through gLite

janna
Télécharger la présentation

EDGI E uropean D esktop G rid I nitiative

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EDGIEuropeanDesktopGridInitiative gLite job submission to EDGI EDGI is supported by the FP7 Capacities Programme under contract nr RI-261556 2010-04-27

  2. Contents • Single job submission to EDGI through gLite • Metajob submission to EDGI through gLite • GBAC submission to EDGI through gLite For each type of submission, there will be a step-by-step explanation in order to ease the understanding.

  3. Single job submission

  4. Step 1: Choose application from the EDGI AR Let us select the dsp app…

  5. Step 2: Get the gridftpurl of its glite binary gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp

  6. Step 3: Get the VO and site list and select cr1.edgi-grid.eu:8443/cream-pbs-edgidemo

  7. Step 4: create jdl • Create your dsp.jdl: Executable = "dsp"; Arguments = "-f 22 -i 22 -p 723 -n pools.txt"; InputSandbox = { "gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" , "pools.txt"}; OutputSandbox = {"cost.txt"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo“; Executable reference in the EDGI AR Input file

  8. Step 5: submit jdl • Proxy creation: > glite-voms-proxy-init -vomsedgiprod.vo.edgi-grid.eu:/edgiprod.vo.edgi-grid.eu/Role=edgidemo • Submit: > glite-wms-job-submit -a -o id dsp.jdl Connecting to the service https://wms.grid.edges-grid.eu:7443/glite_wms_wmproxy_server ================== glite-wms-job-submit Success =================== The job has been successfully submitted to the WMProxy Your job identifier is: https://lb.grid.edges-grid.eu:9000/mhma6si6x__9zyfhG8KCuw The job identifier has been saved in the following file: /home/lpds/smith/TEST-EDGIDEMO/DSP/id ========================================================

  9. Step 6: get status • Status: glite-wms-job-status -i id ================== glite-wms-job-status Success ================= BOOKKEEPING INFORMATION: Status info for the Job : https://lb.grid.edges-grid.eu:9000/YFlhyw24Oh6AhZB6Yyk8Gg Current Status: Running Status Reason: unavailable Destination: cr1.edgi-grid.eu:8443/cream-pbs-edgidemo Submitted: Wed May 16 14:59:48 2012 CEST =====================================================

  10. Step 7: logging • Logging: glite-wms-job-logging-info -v 2 -i id Check UserTags! It returns: 3gbridge_internal_id 3gbridge_dg_url 3gbridge_dg_id The last one is a URL pointingto the WU!

  11. Step 8: track the job in BOINC • Inspect BOINC WU

  12. Step 9: get the result

  13. Metajobsubmission

  14. Step 0: prepare your inputs • Upload your individual input files to web server: • http://somewhere.com/pools1.txt • … • http://somewhere.com/pools10000.txt • Create the description of your metajob: %Required 100% %SuccessAt 100% %Comment pools1.txt Arguments = "-i 0 -n pools.txt -f 22 -p 723“ Input = pools.txt=http://somewhere.com/pools1.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652 Queue [….] %Comment pools10000.txt Input = pools.txt=http://somewhere.com/pools10000.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652 Queue

  15. Step 1: Choose application from the EDGI AR Let us select the dsp app…

  16. Step 2: Get the gridftpurl of its glite binary gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp

  17. Step 3: Get the VO and site list and select cr1.edgi-grid.eu:8443/cream-pbs-edgidemo

  18. Step 4: create and submit jdl • Create your jdl: Executable = "dsp"; Arguments = "-f 22 -i 22 -p 723 -n pools.txt"; InputSandbox = { "gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" , "pools.txt", "_3gb-metajob-dsp-10000" }; OutputSandbox = {"cost.txt"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo"; • Submit: glite-wms-job-submit-oid my-metajob-10000.jdl Reference to the app in the EDGI AR Basic input file Extra input file

  19. Step 5: Query the logging and get the url Location of the status description

  20. Step 6: Query the status Metajob handling rules Actual status of jobs Location of the mapping

  21. Step 7: Download (and extract) the results • Download result as usual: • glite-wms-job-output -i id --dir outputs => • ./outputs/cost.txt • Extract it: • tar zxvf cost.txt => • ./outputs/<subjobid1>/cost.txt • ./outputs/<subjobid2>/cost.txt • ./outputs/<subjobid10000>/cost.txt • See the mapping between your individual job definition and the jobids (which gives the name of directories storing the output files of your app) => one subjob id one job

  22. GBAC job submission

  23. Application and Inputs Overview A User submits her application to EDGIDemo trough gLite. Using GBAC as a virtualization framework requires the real binaries and inputs defined as GBAC inputs. Clients (who have VirtualBox installed) will download the BOINC native GBAC application with the submitted application and its inputs. GBAC starts a Linux Virtual Machine (using VirtualBox). GBAC copies the applications and inputs into the Virtual Machine. The application is executed in the Linux VM. The result is fetched from the VM by GBAC. The VM is shut down and discarded. GBAC finishes and the result is returned to EDGIDemo from the Client. The results are returned to gLite through 3G Bridge from EDGIDemo. User can retrieve the results by using gLite commands EDGIDemo BOINC project … 3G Bridge BOINC Client BOINC Client … GBAC GBAC Application and Inputs Application and Inputs

  24. Step 1: Get the gridftp URL of GBAC • To submit a virtualised app choose the GBAC app from the EDGI AR • Then query the Grid FTP url of the file “gbac.nobin” under the gLite implementation (as you did it during the single submission) • Gridftp url: gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1355/1512/gbac.nobin

  25. Step 2: Get the VO and site list and select cr1.edgi-grid.eu:8443/cream-pbs-edgidemo

  26. Step 3: Create “helloworld” application • Notes: package must have the name X.app.tgz, where X is the name of the executable which must be first launched after unpackaging.

  27. Step 4: create jdl • Create your jdl: Executable = “gbac.nobin"; Arguments = “anything"; InputSandbox = { "gsiftp://edgi-repo.cpc.wmin.ac.uk:2811/srv/edgi/1355/1512/gbac.nobin" , "myhelloworld.sh.app.tgz" }; OutputSandbox = {"output"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo"; Name of GBAC app framework GBAC reference in the EDGI AR Your hello world application

  28. Step 5: Submit jdl • Submitting the job: glite-wms-job-submit –a –o id helloworld.jdl

  29. Step 6a: Checkthe status of thejob under BOINC

  30. Step 6b: Waiting for the job to finish • Get the status of the job: glite-wms-job-status -i id

  31. Step 7: get the result • Get the result: glite-wms-job-output -i id --dir outputs

  32. Controlled hands-on has finished at this point, please play with the tools you learnt or try any of the following tasks: • Try another application in the EDGI AR • Assume dsp is an MC type application, submit it 20 times with the same input (“queue 20”) • Take a real application and submit it using GBAC • For advanced users: combine MetaJob and GBAC: submit the helloworld application 10 times • With different inputs • With the same input

  33. Thank you for your attention! www.edgi-project.eu

More Related