Grid Task Team Status

1. Grid Task Team Status Yonsook Enloe Sept 15, 2003 Yonsook@harp.gsfc.nasa.gov

2. Grid Talks This Week • At this Grid Session : • Overview, lessons learned in Phase 1, Grid benefits, accomplishments, future work– Yonsook Enloe • Updates on CA, Firewalls, MapCenter Monitoring tool – Allan Doyle • CEOS Grid Toolkit – Allan Doyle • GMU status and demo – Liping Di • ESA status – Ivan Petiteville • Overview of China Spatial Information Grid – Dr.Li Guoqing • USGS and WTF Cal/Val support – John Faundeen • NOMADS update – Dave Clark • CNES Grid experience - Paul Knopp • At the ICS (CINTEX) meeting • Update on Grid Catalog MCS component – Yonsook Enloe • At WGISS Plenary • Discussion about future Grid support for inter-agency project – Yonsook Enloe

3. Who Is On the Grid Task Team? • NASA : Yonsook Enloe, Allan Doyle, Jeff Smith, Dick DesJardins Ananth Rao, Dave Hartzell, R. Suresh, Gene Major, Dave Kendig,… • NOAA NOMADS : Glenn Rutledge (NGDC), Danny Brinegar, Ted Smith… • ESA Data Integration: Pedro Goncalves, Luigi Fusco, Ivan Petiteville • George Mason University ECS Data Pools : Liping Di, Aijun Chen • GSFC Advanced Data Grid : Jeff Lubelczyk, Sam Gasster, Bob Harberts…. • Univ of Alabama Data Mining : Sara Graves, Helen Conover, Sandi Redman, Mike McEniry,… • USGS Data Delivery : Stu Doescher, Mike Neiers, Tim Smith,… • GSFC Mayurtech : R. Suresh, Raghu • IPG (Grid Experts) : Judith Utley, Tom Hinke, Jana Nguyen,… • Observers and future joinees…. : Wyn Cudlip,…

4. How Does the Grid Task Team Work? • Task Team – overall coordination, identify issues and key technical areas of interest, initiate and staff tiger teams, coordinate implementation schedules, make general team agreements, work with reps from science programs that could need future Grid support • Network Team – supports network issues; e.g. bandwidth testing, study firewall issues • Tech Team – get technical expertise to provide tech support, identify technical areas of interest, provide support for baseline Globus install issues, and implement grid capabilities, build a toolbox of Grid & application components, identify cross CEOS Grid capabilities needed • Small focussed tiger teams to explore specific topics and issues

5. Issues • CEOS Grid issues: • Six application projects with widely differingapplication areas. • Many issues are common to all six projects.  • Project team is working together to gain insight into these common problems. • Issue 1: Lack of Grid software how-to install and use documents: • Team is producing how-to documentation: Grid Cookbook pages. • 1st cookbook page: How to install and configure Globus 2.2. • 2nd cookbook page: How to install and configure GridFTP with multiple hosts and multiple clients. • 3rd cookbook page (in progress): How to put simple applications on the Grid – e.g. Web Map Server Application on the GRID • Issue 2: Lack of Grid expertise by participants: • Grid Experts (IPG, . . .) are acting as consultants on various specialty topics. • Formed Tech Team to help each other and help later participants.

6. Main Issues, Cont’d. • Issue 3: Most agencies have firewalls. How to deal with these and how to configure to allow access? • Network Team is gathering requirements for firewalls and drafted a "CEOS Grid Firewall Best Practices" document. Jeff Smith gave a featured talk in May. Allan will provide update today. • Technical POCs interested in this issue will review document with their firewall administrators and will iterate on the document. • Issue 4: Grid Monitoring: • Network bandwidth performance testing and checkout of network routing has been performed between testbed nodes. Jeff gave talk this morning • Several Grid Monitoring tools (Map Center, Ganglia, NWS) have been studied and tried out. Map Center tool selected and integrated with Globus Interoperability Test Suite (GITS) • Map Center monitoring tool can monitor host machine ports and perform process level monitoring. • CEOS Grid application sites are linking to this tool to try it out

7. Main Issues, Cont’d. • Issue 5: Certificate Authority (CA): How should host and user certificates be implemented in an international multi-agency consortium? • Small tiger team formed to study issue with Grid expert. • Procedure for CA has been drafted. Allan gave a talk about this in May and will provide update today. • IPG Certificate software has been tested and installed at 3 sites • Planning to use certificates from multiple sources (work in progress). • Issue 6: Catalog Issues: Because EO data have huge volumes from many sources, need a product catalog that is searchable and scalable. What kinds of catalog components are available on the Grid and do these components have the necessary capabilities for CEOS catalogs? • Grid experts on SDSC SRB/MCAT and Globus MCS invited to give presentations to entire team. • Catalog Tiger team formed (small team to study and analyze catalog issues and report back to the main team - work in progress). Liping gave a featured talk in May at the ICS meeting. Yonsook will provide update on Tuesday. • Issue 7: Putting EO applications on the Grid: • Small tiger team formed to study this issue and prototype at least one approach. • Prototyping OGC Web Map Server (WMS) and Web Coverage Server (WCS) on the Grid. • Will generate Cookbook pages on putting WMS and WCS on the Grid.

8. Early Lessons Learned • Grid software components difficult to install and configure: analogy is using IBM operating systems JCL in the 1970s. Not enough documentation by Grid community on how to install and configure software. The CEOS Grid generated cookbook pages very helpful • Because Grid software constantly undergoing change, it’s difficult to stabilize on one version • Grid software collection of varied components working together – but each piece being developed by different group on different schedules. If one component advances to new version, not sure which existing components will work with new upgraded component. Configuration control difficult in this environment. Must talk to a lot of people to understand things • Sometimes there is a drastic change from one version of Grid software to another version (e.g. Globus 2.x to Globus 3.0) that creates compatibility problems

9. Early Lessons Learned (cont) • Not enough “component” level tools to allow groups to build and configure a Grid for their own VO • There is a lack of clear guidelines on how to test components as they are being installed and configured • Grid is very complicated. If you want to build extensions, you have to learn a lot and do a lot of tinkering • Dealing with heterogeneous set of systems is hard (e.g. different flavors of operating systems). You need to know what the remote sites are running in order to do remote executions • Public Key Infrastructure (PKI) systems universally hard to administer and requires a steep learning curve • Learning curve very steep – each site needs at least one person who has systems administration type knowledge – cannot switch people out – lose the knowledge • Pooling of experiences with Grid TT very helpful • Schedules are hard to keep • Grid experts on the team have been invaluable

10. 2003 Wish List • Turnkey system of standard components • Easier installation of components • Better documentation of Grid software : installation and configuration, usage with other Grid components, and future development plans and schedules • Good job manager – workflow scheduling/job management software that can handle load balancing and run to completion management

11. Which Applications Best for Grid? • Back room capabilities – security, Grid FTP, things behind the scenes • Enhanced location service transparency – user will not see the complex backend that mobilizes multiple computer and data resources • Provide collaborative, virtual community environment; tight virtual enterprise or intranet. Grid probably not good for loosely coupled (in terms of common purpose), distributed sites. • Distributed processing for on demand virtual products • Large volume (or distributed sites) Data delivery • Sharing of computing, data, and scientific algorithms. • Compute intensive and large data volumes (scalability)

12. What have we done?

13. Grid Task Team Accomplishments • Strong team culture of cooperation established and working • Initial startup difficulties at all agency application projects overcome. All groups have new staff and new machines. • Gaining Grid technical expertise • Small focussed teams (tech team, network team, tiger teams,..) working well • Started the CEOS Grid Cookbook – how to implement/install various Grid Capabilities and applications – 1st page – how to install Globus 2.x, 2nd page: how to install GridFTP with multiple hosts and multiple clients • Initial network bandwidth testing completed and continuous monitoring being performed

14. Grid Task Team Accomplishments (cont) • CEOS Grid Monitoring Tool web prototype working that tests Grid access to CEOS host machines • Draft CA Procedure with IPG certificates completed. Completed test of software implementation with IPG staff. Installing on multiple CEOS Grid hosts. • Analysis and study of Grid Catalog issues initiated. Discussion with MCS component developers on how and when they will implement the high priority CEOS Grid Catalog requirements • Grid components baseline established – a listing of “stable” components that CEOS Grid will use – don’t want bleeding edge components • Analysis and study of how to install EO applications as Grid Web services started.

15. What are we going to do?

16. Future Work : Application Projects • Get Application working on the CEOS Grid • Define collaborative application project expansions • Potential collaborative project expansions : • NOMADS & UAH • NOMADS & ADG • USGS, GMU, & ESA in support of parts of WTF Cal/Val

17. Future Work : Task Team • Define high level application components for CEOS Grid toolbox from the different application projects. Define which ones are reusable and which ones offer a service at the service provider site. Identify any “missing” components • Prototype a resource catalog (MDS) for a directory of compute, disk, data, software resources • Study and or prototype work flow scheduling/job management capability • Work with at least one global science program (e.g. CEOP, Cal/Val) to identify how the CEOS Grid can support a subset of capabilities needed • Continue to update Cookbook and Firewall document • Continue to upgrade Mapcenter monitoring prototype • Implement the use of the IPG CA Software at all CEOS Grid sites • Interoperability testing between projects running with certificates signed by different Certificate Authorities • Continue study and analysis of Grid Web Services applications for use with EO services/applications

18. Old Plan

19. Revised Plan • Oct 2002-September 2003: Phase 1 Establish CEOS Grid Technology Core Testbed • Objectives: • Establish an immediate Grid capability base within participating CEOS agencies: • Grid software • Access to existing Grids • Pilot applications • Knowledgeable people • October 2003-Sept 2004: Phase 2 Demonstrate CEOS Grid-enabled Applications • Objectives: • Demonstrate Grid-enabled applications, each involving at least two CEOS agency sites. • Show proof of concept. • Evaluate benefits. • Obtain lessons learned from infusion of Grid technologies from the Technology Core into real CEOS agency information systems and applications.

20. Revised Plan (cont) • Oct 2004-Sept 2005: Phase 3 Create persistent CEOS Grid within WTFs • Decide if to continue based on results • Infuse applicable Grid technologies into selected CEOS agency information systems and WTFs, to create a persistent CEOS Grid that would be available to support future CEOS agency initiatives.

21. Website urls • Public Task Team website at http://harp.gsfc.nasa.gov/grid • Password protected working website at http://grid-tech.ceos.org/gridwiki