1 / 14

Project Gridification: the UNOSAT experience

Project Gridification: the UNOSAT experience. Patricia M é ndez Lorenzo CERN (IT-PSS/ED) CERN, 1st March 2006 EGEE USER FORUM. Outlook. ◘ UNOSAT Presentation ◘ Collaboration with CERN ◘ UNOSAT and GRID ◘ Production of UNOSAT inside the Grid environment ◘ Future Plans.

stage
Télécharger la présentation

Project Gridification: the UNOSAT experience

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Project Gridification: the UNOSAT experience Patricia Méndez Lorenzo CERN (IT-PSS/ED) CERN, 1st March 2006 EGEE USER FORUM

  2. Outlook ◘UNOSAT Presentation ◘ Collaboration with CERN ◘ UNOSAT and GRID ◘ Production of UNOSAT inside the Grid environment ◘ Future Plans CERN, 1st March 2006 Patricia Méndez Lorenzo

  3. UNOSAT Presentation UNOSAT is a United Nations Initiative ◘ Objectives ➸ Provide the humanitarian community with access to satellite imagery and Geographic Information System services ► Reduce disasters and plan sustainable development ➸Ensure cost-effective and timely products ◘ Core Services ➸Humanitarian Mapping ➸ Image Processing VEGETATION – 1 Km IKONOS – 1m CERN, 1st March 2006 Patricia Méndez Lorenzo

  4. UNOSAT Service Chain a) Detailed satellite maps of the area are required by UNOSAT Satellite responds the requirement with radar and optic techniques b) Satellites send raw data back to Earth c) Advanced data processing performed d) Data relayed by UNOSAT e) Additional processing and download in the web page of UNOSAT i j k Ground station Data suppliers USER WWW UNOSAT Central Unit CERN, 1st March 2006 Patricia Méndez Lorenzo

  5. Relief Projects of UNOSAT ◘ Case Study: Indian Ocean Tsunami Relief and Development ◘ 29th Dec 2004: First Map distributed online to field users ➸ 14th Jan 2005: Imagery Bank online: ► 100 Tsunami-related maps (pre and post) ► 670 raw satellite images ➸ January: 200,000 tsunami maps downloaded in total ◘ After the Tsunami catastrophe ➸Pakistan ➸ Filipinas ◘ In both cases maps are hosted at CERN ◘ Projects founded by European partners as GMES (Global Monitoring Environment and Security) CERN, 1st March 2006 Patricia Méndez Lorenzo

  6. First step: UNOSAT and CERN ◘ UNOSAT is CERN partner since 2002 ◘ CERN supports them with network facilities, with computer infrastructure and with human (support) resources ◘ Asian Tsunami Example: ➸ Central Web Services at CERN under considerable strain ➸ Availability of the UNOSAT data at any time ◘ UNOSAT provides the users with a web interface able to find the files of the images by clicking on the earth images ➸ Attractive method, easy to use ◘ Something similar to do with the GRID ➸One of our goals inside the UNOSAT-GRID Collaboration CERN, 1st March 2006 Patricia Méndez Lorenzo

  7. One step further: GRID ◘ Potential Bottlenecks: ➸ UNOSAT beginning to suffer from limited capacity and processing power ➸ Multiple satellites being launched ➸ Larger and larger storage capacity needed ◘ In summer 2005 we have provided a whole structure at CERN for UNOSAT ➸ UNOSAT Virtual Organization (VO) ➸ 3.5TB in CASTOR ➸ Computing Elements, Resource Brokers ➸ Collaboration with ARDA group ➸ AFS area of 5GB ◘ We have run some UNOSAT tests (images compression) inside the GRID environment (quite successful) ◘ The framework developed for in principle for Geant4 (See Alberto Ribon’s presentation [49]) has been adapted for UNOSAT needs We have provided the whole GRID infrastructure At CERN CERN, 1st March 2006 Patricia Méndez Lorenzo

  8. UNOSAT Production inside GRID ◘ UNOSAT provided us with a set of images for testing ◘ Associated to each image a metadata file was included File name, directory path, geographical coordinates ◘ Steps: STORAGE LEVEL ➸ Copy and registration of the images in Castor@CERN ► Use of the LFC Catalog ➸ Parse the metadata files to extract the different metadata ➸ Use of the AMGA tool to parse metadata to location of the files COMPUTING LEVEL ➸Use of compression tools to compress images inside LCG resources ➸ Use of the general submission tool adapted to UNOSAT needs CERN, 1st March 2006 Patricia Méndez Lorenzo

  9. A GRID Metadata Catalogue ◘ LFC Catalogue ➸Mapping of LFN to PFN ◘ UNOSAT requires ➸User will give as input data certain coordinates ➸ As output, he wants the PFN for downloading ◘ The ARDA Group assists us setting up the AMGA tool for UNOSAT Oracle DB CASTOR ARDA APP Metadata (x,y,z) SRM PFN LFN LFC CERN, 1st March 2006 Patricia Méndez Lorenzo

  10. Tools developed for new Gridifications Generation of a general framework consisting of 2 major tools: 1. Tool to perform the automatic job submission 2. Tool to retrieve and handle the corresponding output 1. Automatic job submission ◘ Given an user’s jdl this tool performs the following actions: ➸It lists all sites able to run the jdl provided by the user ➸ It creates automatically a jdl file based on that provided by the user ➸ It submits the just created jdl containing the user application(s) ➸ Moreover it creates a subdirectory (defined by the user) containing a list of the sites where the jobs have been submitted, the corresponding jdls and the jobs IDs CERN, 1st March 2006 Patricia Méndez Lorenzo

  11. Tools developed for new Gridifications ◘ Additional Features: ➸ The user can define the queues where the jobs are submitted. These queues are checked to see whether it fixes the job requirements. ➸ Requested LFN files can be included. The corresponding TURLs are searched and included in a file passed in the InputSandbox to the WN ◘ Applications ➸ This tool has been used for the 1st and the 2nd phases of the production: software installation and production ◘ Usage: ./submitter_general -vo geant4 -jdl jdlexample -jobfile G4_PROD -data /grid/geant4/production_software CERN, 1st March 2006 Patricia Méndez Lorenzo

  12. Tools developed for new Gridifications 2. Retrieve and handle of the outputs ➸ The 2nd tool checks the status of the jobs from the job IDs included in the directory given by the user ◘ Usage: ./get_output -jobfile G4_PROD -dest G4_PROD/outputs ◘ Output The job run in ramses.dcic.ups.es:2119/jobmanager-torque-dteam is in status: Scheduled The job run in grid01.phy.ncu.edu.tw:2119/jobmanager-torque-dteam is in status: running The job run in scaic10.scai.frauhofer.de:2119/jobmanager-torque-dteam is in status: over CERN, 1st March 2006 Patricia Méndez Lorenzo

  13. Tools developed for new Gridifications ◘ Additional Features: ➸ It is possible to visualize the outputs on the web ➸ A html report is provided showing the files decided by the user CERN, 1st March 2006 Patricia Méndez Lorenzo

  14. Future Plans with UNOSAT ◘ Collaboration between UNOSAT, ARDA and GD ➸1(2) ARDA and 2 UNOSAT Students ➸ Still many discussions needed ➸Support from other sites foreseen (x,y,z) GRID WORLD ◘ User can get the info in his laptop too ◘ Fundamental AMGA Application CERN, 1st March 2006 Patricia Méndez Lorenzo

More Related