Hosting Cloud, HPC and Grid Educational Activities on FutureGrid

1. Hosting Cloud, HPC and Grid Educational Activities on FutureGrid Renato Figueiredo – U. of Florida Geoffrey Fox, Barbara Ann O’Leary – Indiana University

2. Introduction • Some of the pressing needs faced by educators when teaching Cloud, Grid, HPC: • Availability of suitable resources • A flexible environment • Fast turn-around times • Reuse and sharing of educational materials • FutureGrid sees education as one of its main use cases and seeks to address these needs • Goals of this BOF: • Highlight key aspects of FG and its use to date • Benefit from your input and discussions to make it a primary resource used by you and the community for education activities

3. Needs addressed by FutureGrid • Resources: • FutureGrid is an XSEDE resource available to the community at large which provides unique opportunities for education • Thousands of cores, multiple sites • Heterogeneous: supporting HPC, cloud, Grid • Flexibility: • Cloud, virtualization, dynamic provisioning • Environment can adapt to the user, rather than expect user to adapt to the environment

4. Needs addressed by FutureGrid • Fast turn-around times • Reduce barriers to entry and engage new users • Web 2.0 portal to create class project and manage users; fast turn-around • Reuse and sharing of educational materials • Use of encapsulated “appliances” as a primary delivery mechanism of education/training modules • Promoting reuse, replication, and sharing

5. Guiding principles • Fidelity: activities should use full-fledged, executable software: education/training modules • Learn using the proper tools • Reproducibility: Creators of content should be able to install, configure, and test their modules once, and be assured of the same functional behavior regardless of where the module is deployed • Incentive to invest effort in developing, testing and documenting new modules

6. Guiding principles • Deployability: Students and users should be able to deploy modules in a simple manner, and in a variety of resources • Reduce barriers to entry; avoid dependences upon a particular infrastructure • Community-oriented: Modules should be simple to share, discover, reuse, and expand • Create conditions for “viral” growth

7. FutureGrid Web Portal • How classes are setup and managed • Create a Web account • Submit a project proposal: define a class, workshop, short course, tutorial • Education is one of the key use cases of FG; eager to have more projects!!! • Needs approval, but not tied to a fixed schedule • Typically, turn-around of days • Add users to project • Users create own accounts using the portal • Project leaders authorize them to gain access • Students can then interactively use FG resources (e.g. to start VMs) • Share and reuse – appliances, documents

8. Educational appliances • A flexible, extensible platform for hands-on, lab-oriented education on FutureGrid • Support clustering of resources • Virtual private clusters: Virtual machines + virtual networking to create isolated, repeatable environments • Virtual appliances: self-contained, pre-packaged VMs • Virtual networks: simple management of virtual clusters by students and educators

9. Virtual appliance example • Virtual appliance + virtual network Virtual network Hadoop + Virtual Network Another Hadoop worker A Hadoop worker instantiate Virtual machine copy Repeat…

10. Use of FutureGrid in classes • Cloud/distributed computing classes • U.of Florida, U. Central Florida, U. of Puerto Rico, Univ. of Piemonte Orientale (Italy), Univ. of Mostar (Croatia) • Distributed scientific computing • Louisiana State University • Tutorials, workshops: • Big Data for Science summer school • A cloudy view on computing • SC’11 tutorial – Clouds for science • Science Cloud Summer School (July/Aug’12)

11. Cloud computing classes • Massimo Canonico, U. Piemonte Orientale • Difficulties to overcome: • Hardware issues: find enough free physical machines able to host virtual machines • Software issues: time to install/configure as many as possible different cloud platforms • University was not able to provide me the necessary hardware and software support • Students started to play with FutureGrid • After attending few lessons, they were able to start/stop virtual instance with several Cloud Computing platforms

12. Cloud computing classes • Students used Eucalyptus, OpenStack and Nimbus • Half were not computer scientists. • As FutureGrid freely shares their physical machines and their cloud platforms, decided to freely share all materials of my class. • Hands-out, configuration files and link to useful documentation are available • https://portal.futuregrid.org/contrib/cloud-computing-class

13. Cloud computing classes • Graduate-level “Cloud computing for Data-Intensive Sciences” (Judy Qiu, Fall 2010) • Virtualization technologies and tools • Infrastructure as a service • Parallel programming (MPI, Hadoop) • FutureGrid provided a set of software options that made it possible for students to work on different projects along the system stack.

14. Term Projects Dryad/DryadLINQ #1 Matrix Multiplication (Swapnil,Amit,Pradnay) #2PhyloD (Ratul,Adrija,Chengming) Higher Level Languages Iterative MapReduce #3 LDA (Changsi, Yang) #4MemCache (Saliya, Yiming ,Jerome) #5 Avro (Yuduo, Yuan, patanachai) #6PageRank (Shuo-Huan,Parag) Cloud Platform Cloud Infrastructure #7 Nimbus, Eucalyptus (Stephen, Sonali, Shakeela) Cloud Infrastructure Cloud Storage #8 Cloud Storage Survey (Xiaoming, Nixiaogang) Hypervisor/Virtualization Virtualization #9 Hypervisor Performance Analysis Project (James , Andrew) (Slide courtesy of Judy Qiu)

15. Big Data for Science Johns Hopkins Iowa State Notre Dame Penn State University of Florida Michigan State San Diego Supercomputer Center Univ.Illinois at Chicago Washington University University of Minnesota University of Texas at El Paso University of California at Los Angeles IBM Almaden Research Center 300+ Students (200 on sites from 10 institutes; 100 online) IU MapReduce and UF Virtual Appliance technologies are supported by FutureGrid. July 26-30, 2010 NCSA Summer School Workshop http://salsahpc.indiana.edu/tutorial Indiana University University of Arkansas (Slide courtesy of Judy Qiu)

16. Uploading and sharing images • APIs available to upload images, customize, save, and share images • Community education pages are available • FutureGrid Web portal allows users to publish their own content • Tutorials, presentations on Web portal; VMs on image repositories

17. Discussion starters • At a high-level, does FutureGrid provide capabilities that are aligned with your expectations for an CI useful to hosting educational activities? • What would you need to evaluate whether using FutureGrid is a good idea?

18. Cyberinfrastructure • What cyber-infrastructure features does FutureGrid have that may help you with educational needs? • What cyber-infrastructure features may be missing which would be important?

19. Educational materials • What documentation do educators, students need to use this infrastructure? • What formats are effective? Tutorials, manuals, videos?

20. Topics • IaaS cloud computing? • Map/reduce? • HPC? • Grid? • Applications?

21. Outreach, Community-building • What information may be missing, unclear, hindering ability to reach out? • Ways to attract/foster a community that will share and benefit from sharing educational materials? • Within and beyond XSEDE

22. Questions? • More information: • http://www.futuregrid.org • http://grid-appliance.org • This document was developed with support from the National Science Foundation (NSF) under Grant No. 0910812 to Indiana University for "FutureGrid: An Experimental, High-Performance Grid Test-bed." Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF