Science Gateways and the XSEDE Advanced User Support Services (AUSS) program

1. Science Gateways and the XSEDE Advanced User Support Services (AUSS) program Nancy Wilkins-Diehr

2. Science GatewaysWhat are they? • A novel concept for accessing supercomputers • Web-based portal or client software front ends, supercomputers on the back end • Individuals have an account on the portal, but not the supercomputer • “Community accounts” or service accounts used to do all the processing for the portal • Portal developer keeps track of per user CPU use

3. How did the Gateway program develop?A natural result of the impact of the internet on worldwide communication and information retrieval • Implications on the conduct of science are still evolving • 1980’s, Early gateways, National Center for Biotechnology Information BLAST server, search results sent by email, still a working portal today • 1989 World Wide Web developed at CERN • 1992 Mosaic web browser developed • 1995 “International Protein Data Bank Enhanced by Computer Browser” • 2004 TeraGrid project director Rick Stevens recognized growth in scientific portal development and proposed the Science Gateway Program • Today, Web 3.0 and programmatic exchange of data between web pages • Simultaneous explosion of digital information • Growing analysis needs in many, many scientific areas • Sensors, telescopes, satellites, digital images and video, • #1 machine on Top500 today is 10000x more powerful than all combined entries on the first list in 1993 Less than 20 years since the release of Mosaic!

4. vt100 in the 1980s and alogin window on Ranger today

5. Why are gateways worth the effort? ======= # Full path to executable executable=/users/wilkinsn/tutorial/bin/mcell # Working directory, where Condor-G will write # its output and error files on the local machine. initialdir=/users/wilkinsn/tutorial/exercise_3 # To set the working directory of the remote job, we # specify it in this globus RSL, which will be appended # to the RSL that Condor-G generates globusrsl=(directory='/users/wilkinsn/tutorial/exercise_3') # Arguments to pass to executable. arguments=nmj_recon.main.mdl # Condor-G can stage the executable transfer_executable=false # Specify the globus resource to execute the job globusscheduler=tg-login1.sdsc.teragrid.org/jobmanager-pbs # Condor has multiple universes, but Condor-G always uses globus universe=globus # Files to receive sdout and stderr. output=condor.out error=condor.err # Specify the number of copies of the job to submit to the condor queue. queue 1 #! /bin/sh #PBS -q dque #PBS -l nodes=1:ppn=2 #PBS -l walltime=00:02:00 #PBS -o pbs.out #PBS -e pbs.err #PBS -V cd /users/wilkinsn/tutorial/exercise_3 ../bin/mcell nmj_recon.main.mdl • Increasing range of expertise needed to tackle the most challenging scientific problems • How many details do you want each individual scientist to need to know? • PBS, RSL, Condor • Coupling multi-scale codes • Assembling data from multiple sources • Collaboration frameworks +( &(resourceManagerContact="tg-login1.sdsc.teragrid.org/jobmanager-pbs") (executable="/users/birnbaum/tutorial/bin/mcell") (arguments=nmj_recon.main.mdl) (count=128) (hostCount=10) (maxtime=2) (directory="/users/birnbaum/tutorial/exercise_3") (stdout="/users/birnbaum/tutorial/exercise_3/globus.out") (stderr="/users/birnbaum/tutorial/exercise_3/globus.err") )

6. Gateways democratize access to high end resources • Almost anyone can investigate scientific questions using high end resources • Not just those in the research groups of those who request allocations • Gateways allow anyone with a web browser to explore • Opportunities can be uncovered via google • My then 11-year-old son discovered nanoHUB.org himself while his class was studying Bucky Balls • Foster new ideas, cross-disciplinary approaches • Encourage students to experiment • But used in production too • Significant number of papers resulting from gateways including GridChem, nanoHUB • Scientists can focus on challenging science problems rather than challenging infrastructure problems

7. 3 steps to connect a gateway to XSEDE • Request an allocation • Only a 1 paragraph abstract required for up to 200k CPU hours • Register your gateway • Visibility on xsede.org • Request a community account • Run jobs for others via your portal • Staff support is available! • www.xsede.org/gateways

8. Today, there are approximately 35 gateways using the XSEDE

9. Gateways in the marketplaceKids control telescopes and share images • “In seconds my computer screen was transformed into a live telescopic view” • “Slooh's users include newbies and professional astronomers in 70 countries” • Observatories in the Canary Islands and Chile, Australia coming soon • 5000 images/month since 2003 • Increases public support for investment in these facilities

10. CyberGISSoftware Integration for Sustained Geospatial Innovation • Application of high-end cyberinfrastructure to GIS • Influence on multiple domains • Improved decision support • Spatial joins, layers of multiple datasets at different resolutions • Goal is core set of composable, interoperable, manageable, and reusable software elements • Collaborative geospatial problem solving environment

11. Linked Environments for Atmospheric DiscoveryLEAD • Providing tools that are needed to make accurate • predictions of tornados and hurricanes • Data exploration and Grid workflow

12. Social Informatics Data GridCollaborative access to large, complex datasets • SIDGrid is unique among social science data archive projects • Streaming data which change over time • Voice, video, images (e.g. fMRI), text, numerical (e.g. heart rate, eye movement) • Investigate multiple datasets, collected at different time scales, simultaneously • Large data requirements • Sophisticated analysis tools • Especially valuable for multi-disciplinary course • Computer science, linguistics, psychology http://www.ci.uchicago.edu/research/files/sidgrid.mov

13. Viewing multimodal data like a symphony conductor • “Music-score” display and synchronized playback of video and audio files • Pitch tracks • Text • Head nods, pause, gesture references • Central archive of multi-modal data, annotations, and analyses • Distributed annotation efforts by multiple researchers working on a common data set • History of updates • Computational tools • Distributed acoustic analysis using Praat • Statistical analysis using R • Matrix computations using Matlab and Octave Source: Studying Discourse and Dialog with SIDGrid, Levow, 2008

14. Now back to Advanced User Support Services • Really EUSS, not AUSS – extended support • But that sounds like something a first grader would say • Mission is to provide extended support to users in many areas of cyberinfrastructure • Not just for advanced users! • Support can last from 1-12 months • Requested through allocations process • Significant area of focus in XSEDE • Over one third of total XSEDE funding • Extensive and varied expertise • 70 staffers (35 FTEs) at 8 sites

15. Backdrop against which AUSS existsScience requires diverse digital capabilities Source: John Towns • XSEDE will be a comprehensive, expertly managed set of advanced heterogeneous high-end digital services, integrated into a general-purpose infrastructure. • XSEDE is about increased user productivity • increased productivity leads to more science. • increased productivity is sometimes the difference between a feasible project and an impractical one.

16. XSEDE will support a breadth of research From direct contact with user community as part of requirements collections • Earthquake Science and Civil Engineering • Molecular Dynamics • Nanotechnology • Plant Science • Storm modeling • Epidemiology • Particle Physics • Economic analysis of phone network patterns • Brain science • Analysis of large cosmological simulations • DNA sequencing • Computational Molecular Sciences • Neutron Science  • International Collaboration in Cosmology and Plasma Physics Sampling of much larger set. Many examples are new to TeraGrid/HPC. Range from petascale to disjoint HTC to large shared memory, many are data driven. XSEDE will support thousands of projects. Source: John Towns

17. AUSS Focus Areas Ralph Roskies Nancy Wilkins- Diehr • Advanced Support for Research Teams • Novel and Innovative Projects • Advanced Support for Community Codes • Science Gateways • Advanced Support for Training, Education and Outreach

18. Support available through XSEDE allocations process • Peer-reviewed • Request CPU time, data storage and dedicated supportfrom XSEDE staff • https://www.xsede.org/allocations • AUSS team has expertise in many areas of cyberinfrastructure, many domains • Support can last up to a year

19. Required questions as part of a request • What do you want to accomplish with the help of advanced support? Have you already done any work on this aspect of your software? • How would the success of this advanced support benefit your project? • Which member(s) of your team would collaborate with XSEDE advanced support staff? • Have you had significant interaction on previous projects related to your current proposal or discussed your advanced support needs with any XSEDE staff? If so, please indicate with whom. • Have you received TeraGrid/XSEDE advanced support in the past? If so, please indicate the time period, and how the support you received then relates to the support you request now.

20. Advanced Support for Research TeamsMark Fahey, NICS • Examples: • optimization and scaling of application codes to >100,000 way parallelism; • aggregating petabyte databases from distributed heterogeneous sources and mining them interactively; • developing efficient work and dataflow solution for simulations generating ~100 TB of persistent data per 24-hour run • Deliverable: ≥20 projects completed per year

21. Cross-cutting; Novel and Innovative ProjectsSergiu Sanielevici, PSC • Proactive technical effort to broaden participation in XD • novel science areas e.g. biodiversity, linguistics, and social networks. • demographic diversity e.g. researchers based at MSIs and EPSCoR institutions, SBIR recipients • innovative technologies e.g. supporting mobile computing clients, seamless integration of distributed, heterogeneous databases; development and operation of science gateways, data repositories, and campus bridging • Ability to quickly hire consultants if new expertise is needed

22. Advanced Support of Community CodesJohn Cazes, TACC • Deploying, hardening, and optimizing useful software systems • collaborate with the developers of widely used community codes and tools,including PetaApps, SDCI, STCI, SI2, MREFC and industrial partners • Deliverable: ≥10 projects completed per year

23. Advanced Support of Science GatewaysSuresh Marru, IU • Support communities that wish to use XSEDE resources via science gateways and/or data repositories • Gateway Software Incubator: Evaluates new software and makes recommendation to gateway community. Assist with integration of emerging software like Information Services and gateway-user-attributes. • Gateway community building: Actively engage gateway developers in sharing experiences and learning from each other. • Gateway Helpdesk: Specialized personnel assist in gateway tickets. Many a times, involves in breaking broad gateway problems into focused tickets for relevant support teams. • Interface between gateway developers and XSEDE software and operations groups. Provide usage patterns and help define requirements. • Deliverable: ≥10 projects completed per year

24. Advanced Support for Training, Education and OutreachGalen Arnold, NCSA • Examples: • development and delivery (in person and/or online) of training modules on petascale programming techniques, both general and specific to each XSEDE system; • modules on workflow building, using co-scheduled data transfers, data reduction concurrent with simulation, and algorithms for petascale data mining; • participation in scientific conferences to present the results of relevant advanced user support projects. • With demonstrated interest, researchers can request training events at their institutions via help@xsede.org • Deliverables: ≥50 events per year

25. New Campus Champion Fellows programLaura McGinnis, PSC • Campus Champions • http://www.xsede.org/campus-champions • Campus reps who provide local source of knowledge about XSEDE • Campus Champions paired with AUSS staff on real-world projects • Professional development • Expertise with new technologies, new domains on XD resources • Bring expertise back to campuses • ~40 AUSS-trained Campus Champions in the community at the end of 5 years • Alumni program, regional directory of expertise envisioned • Annual symposium at XSEDE conference to feature Fellows

26. We’re here to help with your Gordon work Questions? wilkinsn@sdsc.edu