TeraGrid Resources Enabling Scientific Discovery Through Cyberinfrastructure (CI)

1. TeraGrid ResourcesEnabling Scientific Discovery Through Cyberinfrastructure (CI) Diane Baxter, Ph.D. San Diego Supercomputer Center University of California, San Diego

2. The National TeraGrid Grid Infrastructure Group (UChicago) UW PSC UC/ANL NCAR PU NCSA UNC/RENCI IU Caltech ORNL U Tenn. USC/ISI SDSC LSU TACC Resource Provider (RP) Software Integration Partner

3. http://www.teragrid.org/ A complex collaboration of over a dozen organizations working together to provide cyberinfrastructure that goes beyond what can be provided by individual institutions, to improve research productivity and enable breakthroughs not otherwise possible.

4. TeraGrid . . . . • Deep - provides leadership class resources at 11 partner sites • Wide - is an integrated, persistent computational resource for broad user communities • Open - is an open scientific discovery infrastructure • Is the world's largest, most comprehensive distributed cyberinfrastructure for open scientific research.

5. To be more specific, TeraGrid . . . • Uses high-performance network connections (10-30 Tb/sec) • Integrates high-performance computers; resources for data analysis, visualization, and storage; data collection tools, high-end experimental facilities; and supporting expertise around the country; • Provides more than a petaflop of computing capability; • Offers more than 30 petabytes of online and archival data storage, as well as systems to manage data acquisition and access; and • Provides researchers access to over 100 discipline-specific databases.

6. What’s in it (TeraGrid) for me? • Instruments that delivers high-end IT resources - computation, storage, visualization, and data/service • A computational facility – over a PetaFLOP in parallel computing capability • A data storage and management facility - over 30 PetaBytes of storage (disk and tape), over 100 scientific data collections • A high-bandwidth national data network • Services: help desk and consulting, Advanced Support for TeraGrid Applications (ASTA), education and training events and resources • Access - without financial cost • Research accounts allocated via peer review • Startup and Education accounts automatic

7. TeraGrid Compute Power PSC UC/ANL PU NCSA IU NCAR 2009 (~1PF) ORNL Tennessee 2007 (504TF) LONI/LSU SDSC TACC Slide Courtesy Tommy Minyard, TACC Computational Resources (size approximate - not to scale)

8. TeraGrid Data Storage and Management • Persistent storage on disk and tape • Allocatable tape-based, geographically distributed storage systems for backups of critical data : • IU (Indiana University) • NCAR (National Center for Atmospheric Research) • NCSA (National Center for Supercomputing Applications) • SDSC (San Diego Supercomputer Center) • Command line usage with GridFTP, using the File Manager tool in the TeraGrid User Portal • GPFS-WAN (General Parallel File System Wide Area Network). ~ 1 petabyte • IU Data Capacitor (1 Pb spinning disk for short-term data storage) • Long term disk storage allocations

9. TeraGrid Architecture POPS User Portal Science Gateways Command Line Viz Service Data Service RP 1 RP 2 TeraGrid Infrastructure (Network, Authorization, Accounting, …) Network, Accounting, … RP 3 Compute Service

11. (Are your eyes glazing over?) Translation please!

12. Enter: Science Gateways • A Science Gateway • Enables scientific communities of users with a common scientific goal and vocabulary • Has a common interface • Leverages community investment • Three common forms: • Web-based Portals • Application programs running on users' machines but accessing services in TeraGrid • Coordinated access points enabling users to move seamlessly between TeraGrid and other grids.

13. Today, there are approximately 29 gateways using the TeraGrid

14. How do Gateways help? • Make science more productive • Researchers use same tools • Complex workflows • Common data formats • Data sharing • Bring TeraGrid capabilities to the broad science community • Lots of disk space • Lots of compute resources • Powerful analysis capabilities • A community-friendly interface to information and research tools

15. But it’s not just ease of use. What can scientists do that they couldn’t do previously? • LEAD - access to radar data • NVO – access to sky surveys • OOI – access to sensor data • PolarGrid – access to polar ice sheet data • SIDGrid – analysis tools for social scientists • GridChem – developing multiscale coupling How would this have been done before gateways?

16. Gateways can enhance and support investments in other projects • Increase access • To instruments • Increase capabilities • To data analysis tools • Improve workforce development • For underserved populations, through broad access to learning resources • Increase outreach • Increase public awareness • Public sees value in investments in large facilities • Slice bread

17. Gateways Greatly Expand Access • Almost anyone can investigate scientific questions using high end resources • Not restricted to those in research groups with allocations • Gateways allow anyone with a web browser to explore • Fosters new ideas, cross-disciplinary approaches • Encourages students to experiment • But Gateways are used in production too • Significant number of papers resulting from gateways including GridChem, nanoHUB • Scientists can focus on challenging science problems rather than challenging infrastructure problems

18. How do we develop a new gateway? Advanced support for Gateway Development • Same peer review process used to request resources • 30,000 CPUs • + 6 months of help from a TG Gateway Team member • Reviews based on appropriate use of resources, science is not reviewed if already funded • Petascale • Multisite workflows • Gateways • Domain expertise

19. Support is Very Targeted • Start with well-defined objectives • Focus on efficient or novel use of national CI resources • Minimum .25 FTE for months to a year • Enough investment to really understand and help solve complex problems • Must have commitment from PIs • Want to make sure work is incorporated into production codes and gateways • Good candidates for targeted support include: • Large, high impact projects • Ability to influence new communities • Suggestions from NSF directorates on important projects • Lessons learned move into training and documentation

20. When is a gateway be most appropriate? • Researchers using defined sets of tools in different ways • Same executables, different input • GridChem, CHARMM • Creating multi-scale or complex workflows • Shared datasets • Common data formats • National Virtual Observatory • Earth System Grid • Some groups have invested significant efforts already, e.g.: • caBIG, extensive discussions to develop common terminology and formats • BIRN, extensive data sharing agreements • Difficult to access data/advanced workflows • Sensor/radar input • LEAD, GEON

21. Things you can do with the TeraGrid:Simulate cell membrane processes Work by Emad Tajkhorshid and James Gumbart, of University of Illinois Urbana-Champaign. • Mechanics of Force Propagation in TonB-Dependent Outer Membrane Transport. Biophysical Journal 93:496-504 (2007). • Results of the simulation may be seen at www.life.uiuc.edu/emad/TonB-BtuB/btub-2.5Ans.mpg • Modeled mechanisms for transport of molecules through cell membrane. • Used 400,000 CPU hours [45 processor-years] on systems at National Center for Supercomputing Applications, IU, Pittsburgh Supercomputing Center Image courtesy of Emad Tajkhorshid, UIUC

22. Predict storms • Hurricanes and tornadoes cause massive loss of life and damage to property • TeraGrid supported spring 2007 NOAA and University of Oklahoma Hazardous Weather Testbed • Major Goal: assess how well ensemble forecasting predicts thunderstorms, including supercells  tornadoes. • Delivers “better than real time” prediction • Used 675,000 CPU hours for the season • Used 312 TB on HPSS storage at PSC Slide courtesy of Dennis Gannon, IU, and LEAD Collaboration

23. Watch Polar Ice Caps Melt (PolarGrid) • Cyberinfrastructure Center for Polar Science (CICPS) • Experts in polar science, remote sensing and cyberinfrastructure • Indiana, ECSU, CReSIS • Satellite observations show disintegration of ice shelves in West Antarctica and speed-up of several glaciers in southern Greenland • Most existing ice sheet models, including those used by IPCC cannot explain the rapid changes http://www.polargrid.org/polargrid/images/4/42/C0050-polargrid-big.m4v Source: Geoffrey Fox

24. CY2007 Usage by Discipline Advanced Scientific All 19 Others Atmospheric Computing 4% Sciences 2% 3% Earth Sciences 3% Molecular Chemical, Thermal Biosciences Systems 31% 5% Materials Research 6% Astronomical Sciences 12% Chemistry 17% Physics 17% 3.95B SUs delivered in CY2007

25. Do you want to see more Gateway examples? • Yes • No

26. Recent Gateways using TeraGrid Significantly • SCEC • SIDGrid • CIG

27. SCEC using gateway to produce hazard map • PSHA hazard map for California using newly released Earthquake Rupture Forecast (UCERF2.0) calculated using SCEC Science Gateway • Warm colors indicate regions with a high probability of experiencing strong ground motion in the next 50 years. • High resolution map, significant CPU use

28. LEAD (portal.leadproject.org/) • Simple enough an undergraduate can use it! http://wxchallenge.com/ • National Center for Supercomputing Applications (NCSA) and IU teamed up to support WxChallenge weather forecast competition. 64 teams, 1000 students, ~16,000 CPU hours on Big Red • XBaya is available from http://www.collab-ogce.org/

29. NanoHub Harnesses TeraGrid for Education Nanotechnology education • Used in dozens of courses at many universities • Teaching materials • Collaboration space • Research seminars • Modeling tools • Access to cutting edge research software

30. Social Informatics Data Grid • Heavy use of “multimodal” data. • Subject might be viewing a video, while a researcher collects heart rate and eye movement data. • Events must be synchronized for analysis, large datasets result • Extensive analysis capabilities are not something that each researcher should have to create for themselves. http://www.ci.uchicago.edu/research/files/sidgrid.mov

31. Social scientists have traditionally worked in isolated labs without the capability to share data or insights with others. • SIDGrid enables a number of capabilities. • Data that is expensive to collect can now be shared with others, increasing the potential for scientific impact. • Geographically distant researchers can collaborate on the analysis of the same data set. • Complex analysis tools and workflows are now available for all to use, rather than having each lab duplicate efforts. • All researchers now have access to the highest quality computational resources • SIDGrid uses TeraGrid resources for computationally-intensive tasks such as media transcoding algorithms for pitch analysis of audio tracks and fMRI image analysis • SIDGrid is unique among social science data archive projects • Focused on streaming data which change over time • Provides the ability to investigate multiple datasets, collected at different time scales, simultaneously • Active users of the SIDGrid system include a human neuroscience group and linguistic research groups from the University of Chicago and the University of Nottingham, UK

32. SIDGrid sidgrid.ci.uchicago.edu

33. TeraGrid Pathways Activities • 2 Gateway components • Adapt gateways for educational use by underrepresented communities • GEON – SDSC, Navajo Tech • Teach participants from underrepresented communities how to build gateways • PolarGrid – IU, ECSU

34. Navajo Technical College and gateways • Incorporating the use of gateways in their curricula • GEON, GISolve areas of initial interest

35. Menu TG Resources and Services • Computing – over a petaflop of computing power and growing • Data • Data storage facilities & management tools • Scientific data collections • Over 30 Science Gateways • Remote visualization servers and software • Technical Support • Central point of contact for support of all systems • Advanced Support for TeraGrid Applications (ASTA) • Education and training events and resources • K-12 Education • Pathways • Campus Champions 35

36. Human Connection: Your Campus Champion • The Campus Champions program supports campus representatives as the local source of knowledge about high-performance computing opportunities and resources. Knowledge plus assistance will empower campus researchers, educators, and students to advance scientific discovery. • Your campus will benefit by having direct access to the TeraGrid and input to its staff, resource allocations awarded for their use, and assistance in using those resources. • TeraGrid will support the Campus Champion. See • http://www.teragrid.org/eot/campuschamps.html • To join the Campus Champions program, contact the TeraGrid Campus Champions Program Coordinator, at tgcc-help@teragrid.org.

37. Online Resources • Online resources at www.teragrid.org • TeraGrid User Portal for managing allocations and job flow • Documentation • Knowledge Base for quick answers to FAQ’s • HPC University to increase general HPC knowledge • Calendar of events including upcoming workshops and training • Annual conference - TG09 • Arlington, VA • June 22-26, 2009

38. TeraGrid: greater than the sum of its parts • Leadership in cyberinfrastructure development, deployment and support • Expertise in building national computing and data resources • Leveraging extensive resources, expertise, R&D, and EOT • Leveraging activities at other participant sites • Learning from each other improves expertise of all TG staff • Shared training, education, and outreach resources benefit all • Simplified access to high end resources • Single unified allocations process • Single point of contact for problem reporting • Coordinated software environments • Uniform access to heterogeneous resources to solve a single scientific problem

39. Would you like to learn more about getting a TeraGrid allocation ? Yes Not today

40. How does the Allocations process work? • Startup allocations: for code development, experimentation with TeraGrid platforms, and application testing. Startup requests may total up to 200,000 service units (SUs) of computation, up to 5TB on disk and 25TB on tape of storage. • Education allocations: for use in classroom instruction or training activities, with the same SU and storage limits as Startup allocations. • Research allocations: requires a detailed justification of resource usage. Requests are reviewed four times a year by the Resource Allocations Committee. • National peer-review process • allocates computational and data resources • makes recommendations on allocation of advanced direct support services • Currently awarding >1B Normalized Units of resources • Principal investigator (PI) must be a researcher, educator, or postdoctoral researcher at a US academic or non-profit research institution.

41. Go to the POPS page - https://pops-submit.teragrid.org

42. Create a POPS Login

43. Indicate that you are “New” to the Teragrid

44. Indicate this is a “Start-up” Request

45. Select Startup or Educational

46. Fill out PI information

47. Skip Co-PIs info

48. Fill out info on your project

49. Fill out info on your funding

50. Estimate your computing need (reasonably)