1 / 112

网格计算与云计算

网格计算与云计算. “Cloud” Computing is 1+ yr old. Michael Sheehan’s GoGrid Blog, July 25, 2008 http://linux.sys-con.com/node/587717. Confused?. SaaS. P2P. Virtualization. Grid Computing. Cluster Computing. ?. SaaS = Software as a Service. Utility Computing. ?. Cloud Computing.

eamon
Télécharger la présentation

网格计算与云计算

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 网格计算与云计算

  2. “Cloud” Computing is 1+ yr old Michael Sheehan’s GoGrid Blog, July 25, 2008 http://linux.sys-con.com/node/587717

  3. Confused? SaaS P2P Virtualization Grid Computing Cluster Computing ? SaaS = Software as a Service Utility Computing ? Cloud Computing

  4. One can categorize each component Grid Computing Cloud Computing Cluster Computing SaaS P2P Utility Computing Virtualization Infrastructure Usage Model

  5. 网格计算

  6. What is a Grid? Enable “coordinated resource sharing & problem solving in dynamic, multi-institutional virtual organizations.” (Source: “The Anatomy of the Grid”)

  7. Virtual Organizations

  8. TeraGrid

  9. What is the TeraGrid? Technology+Support= Science • NSF已投资2.460亿美元 • 自2004年10月已处于生产运行阶段，目前已用高性能网络集成了每秒750万亿次计算能力、30PB存储空间和100多个学科的数据库资源。

  10. TeraGrid’s 3-pronged strategy to further science DEEP Science: Enabling Terascale Science Make science more productive through an integrated set of very-high capability resources ASTA projects WIDE Impact: Empowering Communities Bring TeraGrid capabilities to the broad science community Science Gateways OPEN Infrastructure, OPEN Partnership Provide a coordinated, general purpose, reliable set of services and resources Grid interoperability working group

  11. TeraGrid Used

  12. TeraGrid PI’s By Institution Blue: 10 or more PI’s Red: 5-9 PI’s Yellow: 2-4 PI’s Green: 1 PI TeraGrid PI’s

  13. TeraGrid Resources 100+ TF 8 distinct architectures 3 PB Online Disk >100 data collections

  14. Science GatewaysA new initiative for the TeraGrid Increasing investment by communities in their own cyberinfrastructure, but heterogeneous: Resources Users – from expert to K-12 Software stacks, policies Science Gateways Provide “TeraGrid Inside” capabilities Leverage community investment Three common forms: Web-based Portals Application programs running on users' machines but accessing services in TeraGrid Coordinated access points enabling users to move seamlessly between TeraGrid and other grids. Workflow Composer

  15. Gateways are growing in numbers 10 initial projects as part of TG proposal >20 Gateway projects today No limit on how many gateways can use TG resources Prepare services and documentation so developers can work independently Open Science Grid (OSG) Special PRiority and Urgent Computing Environment (SPRUCE) National Virtual Observatory (NVO) Linked Environments for Atmospheric Discovery (LEAD) Computational Chemistry Grid (GridChem) Computational Science and Engineering Online (CSE-Online) GEON(GEOsciences Network) Network for Earthquake Engineering Simulation (NEES) SCEC Earthworks Project Network for Computational Nanotechnology and nanoHUB GIScience Gateway (GISolve) Biology and Biomedicine Science Gateway Open Life Sciences Gateway The Telescience Project Grid Analysis Environment (GAE) Neutron Science Instrument Gateway TeraGrid Visualization Gateway, ANL BIRN Gridblast Bioinformatics Gateway Earth Systems Grid Astrophysical Data Repository (Cornell) Many others interested SID Grid HASTAC

  16. OSG(Open Science Grid)

  17. Open Science Grid (OSG) Origins: • National Grid (iVDGL, GriPhyN, PPDG) and LHC Software & Computing Projects Current Compute Resources: • 61 Open Science Grid sites • Connected via Inet2, NLR.... from 10 Gbps – 622 Mbps • Compute & Storage Elemets • All are Linux clusters • Most are shared • Campus grids • Local non-grid users • More than 10,000 CPUs • A lot of opportunistic usage • Total computing capacity difficult to estimate • Same with Storage

  18. OSG Snapshot Snapshot of Jobs on OSGs 96 Resources across production & integration infrastructures Using production & research networks Sustaining through OSG submissions: 3,000-4,000 simultaneous jobs . ~10K jobs/day ~50K CPUhours/day. Peak test jobs of 15K a day. 20 Virtual Organizations +6 operations Includes 25% non-physics. ~20,000 CPUs (from 30 to 4000) ~6 PB Tapes ~4 PB Shared Disk

  19. What is the Open Science Grid? MCGILL HARVARD ALBANY BU BUFFALO UMICH UWM WSU CORNELL BNL MSU WISC PSU FNAL IOWA STATE UIC UNI UCHICAGO LEHIGH ANL UNL PURDUE NERSC UIUC NSF IUPUI UVA IU INDIANA STANFORD KU ORNL VANDERBILT CALTECH UCLA UCR UNM CLEMSON OU UMISS SDSC TTU LTU UTA SMU LSU UFL (+Brazil, Mexico, Tawain, UK)

  20. OSG应用 Genome sequence analysis Sloan digital sky survey STAR: 5 TB transfer (SRM, GridFTP) Earth System Grid: O(100TB) online data

  21. Earth System Grid

  22. EGEE(Enabling Grids for E-sciencE)

  23. European Grid Initiative

  24. Archeology • Astronomy • Astrophysics • Civil Protection • Comp. Chemistry • Earth Sciences • Finance • Fusion • Geophysics • High Energy Physics • Life Sciences • Multimedia • Material Sciences • … >250 sites 48 countries >50,000 CPUs >20 PetaBytes >10,000 users >150 VOs >150,000 jobs/day June 2, 2008

  25. Users and resources distribution June 2, 2008

  26. EGEE workload in 2007 CPU: 114 Million hours Data: 25PB stored 11PB transferred http://gridview.cern.ch/GRIDVIEW/same_index.phphttp://calculator.s3.amazonaws.com/calc5.html? 17/05/08 $58688679.08

  27. LCG(LHC Computing Grid)

  28. LHC - Large Hadronic Collider 4 experiments: ATLASAliceCMSLHCb GRID Tutorial- How to use LCG 27 km long pipe 7+7 TeV Federico Calzolari

  29. LCG - LHC Computing Grid • 目前集成了33个国家的 140个计算中心。 • 2008年将执行1亿个计 算任务。 GRID Tutorial- How to use LCG Federico Calzolari

  30. Proxy certificate • Get your proxy certificate • temporary (usually 24h) certificate • depending on VO: grid-proxy-init voms-proxy-init -voms <VO>:/<VO>/Role=<role> -valid 1000:00 GRID Tutorial- How to use LCG

  31. Certificate • Install your certificate on the User Interface: • Log in into the UserInterface, copy there the file you exported, and create a directory where your certificate + private key will be stored: mkdir ~/.globus • Convert PKCS12 file .p12 into the supported standard .pem This operation will split your mycert.p12 file in two files: the certificate (usercert.pem) and the private key (userkey.pem) openssl pkcs12 -nocerts -in <mycert.p12> -out ~/.globus/userkey.pem openssl pkcs12 -clcerts -nokeys -in <mycert.p12> -out ~/.globus/usercert.pem chmod 0400 ~/.globus/userkey.pem chmod 0600 ~/.globus/usercert.pem • At end you should have something like: [user@userinterface .globus]$ ls -al -rw------- 1 user user 2008 Nov 13 16:50 usercert.pem -r-------- 1 user user 963 Nov 13 16:50 userkey.pem GRID Tutorial- How to use LCG Federico Calzolari

  32. Register to a VO http://grid-it.cnaf.infn.it GRID Tutorial- How to use LCG for generic user

  33. JDL: Job Description Language • JOB overview: • JDL (job encapsulation) • main script • executable program Creation Submission GRID Tutorial- How to use LCG Status Retrieval

  34. JDL • test.jdl Executable = "script.sh"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"script.sh","exe.bin"}; # Input OutputSandbox = {"std.out","std.err","out"}; # Output VirtualOrganisation = "<VO>"; DataAccessProtocol = {"file","gsiftp","rfio","dcap"}; InputData = {"lfn:/grid/<VO>/<FILE>"}; OutputSE = "<SE>"; Requirements=Member("<SITE>", other.GlueHostApplicationSoftwareRunTimeEnvironment && other.GlueCEName=="<QUEUE>"); GRID Tutorial- How to use LCG

  35. Main script • script.sh #!/bin/sh # Environment date >> out2 hostname >> out2 # Get data lcg-cp [-v] --vo <VO> lfn:<file> file:///data.tgz # Unpack input [data.tgz: src.cpp,...] tar -zxvf data.tgz # Compile source g++ src.cpp -o exe.bin chmod u+x exe.bin # Exec program ./exe.bin > out # Pack output tar -zcvf out.tgz out out2 GRID Tutorial- How to use LCG

  36. Submit a Job • Submit a JOB edg-job-submit -o ID <JDL> # save JOBid on file ID Selected Virtual Organisation name (from JDL): cms Connecting to host rb119.cern.ch, port 7772 # Resource Broker Logging to host rb119.cern.ch, port 9002 ********************************************************************************************* JOB SUBMIT OUTCOME The job has been successfully submitted to the Network Server. Use edg-job-status command to check job current status. Your job identifier (edg_jobId) is: - https://rb119.cern.ch:9000/tG3Xp2jT_58IUeXoY1GoZQ # JOBid ********************************************************************************************* • Control JOB status edg-job-status <JOBid> [https://rb119.cern.ch:9000/tG3Xp2jT_58IUeXoY1GoZQ] ************************************************************* BOOKKEEPING INFORMATION: Status info for the Job : https://rb119.cern.ch:9000/tG3Xp2jT_58IUeXoY1GoZQ Current Status: Waiting / Scheduled / Running / Done (Success/Abort) Status Reason: Job successfully submitted to Globus Destination: ce0001.m45.ihep.su:2119/jobmanager-lcgpbs-cms reached on: Sat Nov 17 22:38:34 2007 ************************************************************* GRID Tutorial- How to use LCG

  37. Get the output • JOB output retrieve edg-job-get-output <JOBid> [https://rb119.cern.ch:9000/tG3Xp2jT_58IUeXoY1GoZQ] Retrieving files from host: rb119.cern.ch( for https://rb119.cern.ch:9000/tG3Xp2jT_58IUeXoY1GoZQ) ********************************************************************************* JOB GET OUTPUT OUTCOME Output sandbox files for the job: - https://rb119.cern.ch:9000/tG3Xp2jT_58IUeXoY1GoZQ have been successfully retrieved and stored in the directory: /tmp/jobOutput/<USER>_ tG3Xp2jT_58IUeXoY1GoZQ ********************************************************************************* ls -al /tmp/jobOutput/calzolar_ tG3Xp2jT_58IUeXoY1GoZQ -rw-r--r-- 1 calzolar cms 11 Nov 17 23:59 out -rw-r--r-- 1 calzolar cms 133 Nov 17 23:59 std.err -rw-r--r-- 1 calzolar cms 8 Nov 17 23:59 std.out GRID Tutorial- How to use LCG

  38. Job Requirements • JDL Requirements • everywhere NO Requirements • at Pisa Requirements=Member("INFN-PISA",other.GlueHostApplicationSoftwareRunTimeEnvironment); • on a queue 1 day at least long Requirements=(other.GlueCEPolicyMaxCPUTime>60*24); • on a site with at least 20 free CPU Requirements=(other.GlueCEStateFreeCPUs>20); • on a site with at least 1 TB (unit:kb) local disk available Requirements=anyMatch(other.storage.CloseSEs,target.GlueSAStateAvailableSpace > 1000000000); • on a site with a given software locally installed Requirements=Member(”VO-<VO>-TAG",other.GlueHostApplicationSoftwareRunTimeEnvironment); GRID Tutorial- How to use LCG

  39. Requirements TAGs • from SINICA http://goc.grid.sinica.edu.tw/gstat/<SITE>/ GlueHostOperatingSystemName: Scientific Linux CERN GlueHostOperatingSystemRelease: 4.5 GlueHostOperatingSystemVersion: Beryllium GlueSubClusterPhysicalCPUs: 0 GlueSubClusterLogicalCPUs: 0 GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2 LCG-2_1_0 LCG-2_1_1 LCG-2_2_0 LCG-2_3_0 LCG-2_3_1 LCG-2_4_0 LCG-2_5_0 LCG-2_6_0 LCG-2_7_0 GLITE-3_0_0 R-GMA INFN-PISA SI00MeanPerCPU_1800 SF00MeanPerCPU_2000 MPICH MPI_HOME_NOTSHARED AFS VO-atlas-cloud-IT VO-atlas-production-12.0.5 VO-atlas-production-12.0.6 VO-atlas-production-12.0.7 […] GRID Tutorial- How to use LCG

  40. Resources search • Query CPU / Storage available per VO lcg-infosites --vo <VO> ce #CPU Free Total Jobs Running Waiting ComputingElement ---------------------------------------------------------- 165 1 1 0 1 ce.phy.bg.ac.yu:2119/jobmanager-pbs-cms 120 11 0 0 0 fangorn.man.poznan.pl:2119/jobmanager-pbs-cms 192 110 0 0 0 gridce.atlantis.ugent.be:2119/jobmanager-pbs-cms 212 0 529 146 383 gridce.iihe.ac.be:2119/jobmanager-pbs-cms 227 5 312 222 90 ingrid.cism.ucl.ac.be:2119/jobmanager-lcgcondor-cms 15 15 0 0 0 ce002.ipp.acad.bg:2119/jobmanager-lcgpbs-cms 80 43 0 0 0 ce02.grid.acad.bg:2119/jobmanager-pbs-cms 24 13 0 0 0 ce001.grid.uni-sofia.bg:2119/jobmanager-lcgpbs-cms lcg-infosites --vo <VO> se Avail Space(Kb) Used Space(Kb) Type SEs ---------------------------------------------------------- 97470000 n.a n.a dpm.phy.bg.ac.yu 395467659 779205896 n.a cmsse01.ihep.ac.cn 27664924 59878772 n.a se001.grid.uni-sofia.bg 149180000 n.a n.a se.hpc.iit.bme.hu 1 1 n.a dcsrm.usatlas.bnl.gov 190040000 208 n.a lxdpm101.cern.ch 1000000000000 500000000000 n.a castorgrid.cern.ch 1000000000000 500000000000 n.a srm.cern.ch GRID Tutorial- How to use LCG

  41. Resources search • Query available sites for my Job edg-job-list-match <JDL> Selected Virtual Organisation name (from JDL): cms Connecting to host rb119.cern.ch, port 7772 *************************************************************************** COMPUTING ELEMENT IDs LIST The following CE(s) matching your job requirements have been found: *CEId* a01-004-128.gridka.de:2119/jobmanager-pbspro-cmsS a01-004-128.gridka.de:2119/jobmanager-pbspro-cmsXS ares02.cyf-kr.edu.pl:2119/jobmanager-pbs-cms beagle14.ba.itb.cnr.it:2119/jobmanager-lcgpbs-cms bogrid5.bo.infn.it:2119/jobmanager-lcgpbs-cms ce-fzk.gridka.de:2119/jobmanager-pbspro-cmsL ce-fzk.gridka.de:2119/jobmanager-pbspro-cmsS ce-fzk.gridka.de:2119/jobmanager-pbspro-cmsXS ce.bg.ktu.lt:2119/jobmanager-lcgpbs-cms ce.cc.ncu.edu.tw:2119/jobmanager-lcgpbs-cms […] gridce.ilc.cnr.it:2119/jobmanager-lcgpbs-cms gridce2.pi.infn.it:2119/jobmanager-lcglsf-cms4 gridce.sns.it:2119/jobmanager-lcgpbs-cms GRID Tutorial- How to use LCG

  42. Grid Monitoring GridICE INFN GRID Tutorial- How to use LCG GOC Sinica

  43. Grid Monitoring GRID Tutorial- How to use LCG AOB

  44. 云计算

  45. Cloud Computing

  46. Cloud Computing • Definition • Cloud computing is a concept of using the internet to allow people to access technology-enabled services. It allows users to consume services without knowledge of control over the technology infrastructure that supports them. - Wikipedia

  47. Enterprise IT spending challenge Global Annual IT Spending Estimated US$B 1996-2010 300 Power and Cooling Costs Server Mgt and Admin Costs 250 New Server Spending 200 150 100 50 $0B 1997 1999 2001 2004 1996 1998 2000 2002 2003 2005 2006 2007 2008 2009 2010 Source: IBM Corporate Strategy analysis of IDC data, Sept. 2007

  48. Dream or Nightmare?

  49. Seasonal Spikes

More Related