Challenge in Building National Scale Grid Infrastructure

1. Challenge in Building National Scale Grid Infrastructure Somsak Sriprayoonsakul Thai National Grid Center Software Industry Promotion Agency (SIPA) Ministry of Information and Communication

2. Thai National Grid Project • A 4 years project under Ministry of Information and Communication Technology • 5.5 Million US$ program • Start from 13 universities and 1 government organizations • Extending rapidly

3. Partner Institutes Chaingmai Chaingmai University (CMU) Khonkhan Khonkhan University (KKU) Nakhon Ratchasima Bangkok Suranaree University of Technology (SUT) Chulalongkorn University (CU) Kasetsart University (KU) King Mongkut’s Institute of Technology North Bangkok (KMITNB) King Mongkut’s University of Technology Thonburi (KMUTT) King Mongkut’s Institute of Technology Ladkrabang (KMITL) Mahidol Unversity (MU) Silpakorn University (SU) Meteorological Department of Thailand (TMD) Nakhonsrithammarat Walailak University (WU) Songkla Prince of Songkla University (PSU)

4. Potential Application Domain • Life-science Grid • Cheminformatics Grid • Geoinformatics Grid • Digital Archive and E-learning • Computational Fluid Dynamics Grid • Simulation and Modeling Grid • Digital Media and Animation

5. Capability • Production Grid and Cluster infrastructure • Rich set of services • Computational Grid • Enterprise analysis Grid • Data and knowledge Grid • Collaborative Grid • Broad commercial adoption • Stable Grid and Cluster infrastructure • Interoperable Middleware and basic services • Basic computational Grid • Basic Data Grid • Grid Portal • Grid applications • R&E • Non production enterprise app • Broad academic adoption • Limited commercial adoption • Pilot Grid and Cluster infrastructure • Interoperable Middleware • Basic computational Grid • Basic Data Grid • POC applications Time 2007 2008 2009 2006 Enterprise Grid Pilot Grid R&E Grid

6. เชียงราย แม่ฮ่องสอน พะเยา น่าน เชียงใหม่ ลำปาง ลำพูน หนองคาย แพร่ อุตรดิตถ์ นครพนม สกลนคร เลย อุดรธานี หนองบัว สุโขทัย ลำภู ตาก มุกดาหาร กาฬสินธุ์ กำแพงเพชร พิจิตร เพชรบูรณ์ มหาสารคาม ชัยภูมิ อำนาจ ยโสธร ร้อยเอ็ด เจริญ นครสวรรค์ อุทัยธานี อุบลราชธานี ลพบุรี นครราชสีมา สิงห์ ศรีสะเกษ บุรี บุรีรัมย์ สุรินทร์ สุพรรณ สระบุรี กาญจนบุรี บุรี พระนครศรี นครนายก ปทุมธานี ปราจีนบุรี นนทบุรี สระแก้ว ฉะเชิงเทรา ราชบุรี สมุทรปราการ สมุทรสาคร สมุทรสงคราม จันทบุรี ระยอง เพชรบุรี ประจวบ ตราด คีรีขันธ์ ชุมพร ระนอง สุราษฏร์ธานี พังงา นครศรีธรรมราช กระบี่ ภูเก็ต ตรัง พัทลุง สตูล สงขลา ปัตตานี ยะลา นราธิวาส Inter-University Network (UniNet) Commission on Higher Education (Center) 24 Public Universities 29 IT Campus 39 Rajabhat Universities 8 Private Universities 13 Mahachulalongkorn Rajawitayalai 9 Other agencies 9 Rajamangala Institue of Technology (44 Campus) Ministry of Education (for Under Higher Education) 2.5 Gbps- STM16 Internet - 622 Mbps (to Internet 2 155 Mbps) SUT 1 Gbps CMU 155 Mbps- STM1 34 Mbps - E3 8 Mbps - 4*E1 2 Mbps - E1 64 k – 2Mbps Domestic 1Gbps SU_NMJ MUA BU THAISARN 1Gbps MOEnet (K-12) 1Gbps (coming soon) PSU Update on 24 Jan 2006

7. Grid Infrastructure Teraflop Cluster at TNGC NECTEC TMD 2.5Gb CMU ThaiRen (Uninet/ThaiSarn) CU KMITL MU KMUTT KKU SU AIT KU KMITNB SUT WU PSU

8. Implementation

9. Users see grid as a large single cluster GH Grid Scheduler Super Cluster Grid Model Network H H H C C C C H C C C C C C C C C C C C

10. Applications Core : Gamess , Autodock Option : Blast Portal : GridSphere Option : GridPort Management Scheduler Programming Core : SCMS Core : SQMS , SGE Ninf - G Option : MOGAS Option : PBS , Condor , CSF MPI - G 2 Grid Middleware Globus Toolkit 4 . 0 Fabric Core : Linux / Rocks on x 86 _ 64 / 32 Option : Windows Thai National Grid Standard Software Stack

11. Building the Grid Digital Media HEP Medical Bioinformatics Grid Middleware Network

12. Technical Contact • Each ThaiGrid member institute need to assign at least one person as technical contact point for all ThaiGrid activity • Technical contact tasks • Software setup (both cluster and Grid) • Contact point for people inside institute • Our communication medium are mailing-list, webboard, wiki, skype, and telephone • http://rswg.thaigrid.or.th are the central point for all technical contacts

13. ThaiGrid deployment step • Site firewall compliance test • Site deployment • Install NPACI Rocks 4.1 • Install ThaiGrid Roll • Site setup • Grid CA setup • Setup of Grid services • Register site with Grid Control Center node • Site Certify

14. Site firewall compliance test • ThaiGrid using Globus 4.0 software that comes with NAPCI Rocks • Focus on using Pre-WS Grid component • 2119 – Globus Gatekeeper • 2811 – Grid FTP • 2135 – Globus MDS • 80 – For web access • 22 – For SSH access • 10001 – 20000 – reserved for applications • Firewall is a nightmare for Grid deployment • A Firewall testing tools is needed to identify firewall problem

15. Firewall testing tool • A firewall tester program “TGCheckport” is developed. [bank@tgcc TGcheckport-1.3]$ ./TGcheckport.sh maeka.ku.ac.thStart testing maeka.ku.ac.thTue May 1616:48:31 ICT 2006determining globus-gatekeeper service [2119] ... Passdetermining gsiftp service [2811] ... Passdetermining globus-mds service [2135] ... Passdetermining scmsweb service [80] ... Passdetermining globus-mpi-required-spare-port service [10001-20000] ... PassCongratulation, you pass all the test. Your system is ThaiGrid FirewallCompliance.Finish testingTue May 1616:48:32 ICT 2006[bank@tgcc TGcheckport-1.3]$

16. Site Deployment • Sitedeployment process • OS installation using • Grid core middleware (Globus) installation • Grid additional middleware installation • MPICH-G2, ninf-G, GSI-ssh

17. Challenge • All these must be done on every cluster nodes to ensure compatibility for all components • Manual installation is time consuming, costly, and error prone • Rocks simplify all of these process due to the roll concept • OS and Middleware installation are highly automatic

18. ThaiGrid Roll Grid HPC SCE Java SGE Ninf-G Challenge • Many Grid middleware components are needed • SGE/NINF-G/GLOBUS/MPICH/SCE • Each has already been packed into Rocks Roll • Nevertheless, to completely install all software require many rolls • Solution: every software are bundled to a new ThaiGrid Roll • A Jumbo Roll bundle all standard Grid software together

19. Site Setup for ThaiGrid • Setup host certificates for Grid • Metadata Service Setup • Enable some services • Jobmanager-sge • Grid-FTP • GSI-SSH • Joining with Grid Control Center • Currently using SCMSWeb for central web-base monitoring • Technical contact of each site will send an e-mail with URL of cluster monitoring nodes • Grid meta-scheduler on GCC will distribute jobs to each institutes cluster • Setup Grid user

20. Site Setup • Traditional Grid setup are based on manual setup • Complicate • Error prone • ROCKS 4.1 provide mechanism for automatic grid installation and configuration • Rocks has already configured almost every services for basic Grid system • Anyway, some additional manual setup is needed • We currently use a well-documented admin guide to provide a step-by-step to join ThaiGrid

21. Thai Grid Control Center • Grid Control Center Software • Monitoring • Automatic testing • Service Monitoring • Accounting • System monitoring • Netwwork Bandwidth • Grid-level Job scheduling

22. Thai Grid Control Center (con’t) • Automatic Grid services status monitoring • Command line for immediate status checking

24. Site Certify • Site certification program • Level 1: Network firewall compliance • Level 2: Middleware compliance (MPI/G2, Ninf-g) • Level 3: Job scheduling compliance • Level 4: Data Grid compliance • Standard • Define clear software stack

25. ThaiGrid Firewall Status Overall : 15 Pass : 12 Partial : 1 Fail : 2

26. ThaiGrid Middleware Status Overall : 15 Pass : 4 Partial : 1 Fail : 10

27. Grid Applications • ThaiGrid Drug Discovery Infrastructure • Grid based data assimilation using RS data • Implicit Narrowband Level Set Method for Fire Spread Model • The Parallel Simulation Model for Thin Film Deposition Using the DSMC Method • Medical Grid • Access Grid

28. ThaiGrid Drug Discovery Infrastructure • Partners: • Dr. Jak Saengma, KU • Somsak Sriprayoonsakul, TNGC • IBM Thailand • Challenges • From over 10000 active compounds available from Thai medicinal plants database, find a smallest set of compounds that has a potential to be used as a drug • Very compute intensive. Several month of computing time • Solution: Use grid to increase computing power to 10-100 times • Distributed screening process to hundred of hosts on the Grid • Benefit • Speeding up time to results • Allow scientists to spend more time analyzing results

29. Grid based data assimilation using RS data • Dr. Kiyoshi Honda, AIT • Dr. Putchong Uthayopas, THNC • Challenges • Using GA based data assimilation for agricultural monitoring • Very long computation time from months to years • Large amount of RS data needed to be moved around and process • Solution: • Using Grid to harvest more computing power • Hide data assimilation process behind the service using MPI, GridRPC (ninf-g) • Benefit • Speeding up the calculation

30. Implicit Narrowband Level Set Method for Fire Spread Model Dr. Pallop Huabsomboon Mahidol University • Challenges • Simulation of large scale firespred for forest fire • Very time consuming process • Solution • Distribute the computation on the Grid • Benefit • Expect 10x faster simulation speed • Ability to model much larger problem within a reasonable time • Part of TNGP/ Hewlett Packard Join Research Program

31. The Parallel Simulation Model for Thin Film Deposition Using the DSMC Method Dr. Rardchawadee Silapunt Dr. Tiranee Achalakul KMUTT, Thailand • Thin film deposition process is used for many manufacturing industries including very large scale integrated circuits, optics, magnetic, hard and corrosion resistant coatings, micro-mechanics. • Grid system will be used to accelerate the computation and simulation process • Part of TNGP/ Hewlett Packard Join Research Program

32. GRID Services XML IMAGE Data Agent Service Registry Meta Data (HL7) Medical Grid Suwachai Sieanoon, Somkiat Chatchuenyot, Dr. Panjai Tantasnawong Computer Science Department, Silpakorn University • To build and develop a distributed information management system using DataGrid technology. • Apply the distributed information management system to test on exchanging and managing the Patient Records System, which is record complexity and, different format and large size of data.

33. Access Grid • First APEC Virtual Symposium on Pandemic Preparedness using Access Grid, January 20, 2006 • Presentation from Medical Organization in many economies • China, Taiwan, USA, Vietnam, Thailand, Singapore, Philippine • Access Grid are used as a key communication infrastructure

34. Future Work • Grid Infrastructure • Include “Grid ZeroConf” roll to auto configure the cluster to join ThaiGrid • Install additional CA certificates • Generate host certificate request • Modify yum repository • Grid Application • Drug Design application will be deployed fully on ThaiGrid clusters before July • GIS-base Grid-RPC application due on August • More application coming

35. End

36. Additional Slides

37. Technical Contact • Each ThaiGrid member institute need to assign at least one person as technical contact point for all ThaiGrid activity • Technical contact tasks • Software setup (both cluster and Grid) • Contact point for people inside institute • Our communication medium are mailing-list, webboard, wiki, skype, and telephone • http://rswg.thaigrid.or.th are the central point for all technical contacts

38. Grid: What and Why • Grid is a technology that incorporates distributed computing systems into a massive computing and storage pool with efficient resource sharing. • Grid can strengthen national competitiveness in ICT industries as well as promote sciences and engineering research. • Grid benefits several application domains: • E-commerce, E-government, E-learning • Digital Media and Animation • Geoinformatics, Bioinformatics • Financial Analysis • Etc.

39. Solving network problem • General problem • Firewall Compliance • Dealing with smart intrusion detection system • Running Grid on Shared REN/Service network • Network underutilized, unbalance bandwidth • Develop a network measurement infrastructure • Collaboration PSU/TNGC/KU/Uninet • Collaboration with network provider (Uninet)

40. ThaiGrid Central Monitoring • ThaiGrid currently use OpenSCE to monitor the whole ThaiGrid • OpenSCE is installed during Rocks installation using SCE Roll • Each cluster provides monitored information through HTTP protocol to central monitoring site