Internet2 Networks in Support of e-Science

1. Internet2 Networksin Support of e-Science Rick Summerhill Director Network Research, Architecture, and Technologies Russ Hobby Program Manager for Science and Engineering Internet2 NSF Internet2 Day 19 October 2006

2. Agenda • The Existing Internet2 Network (Abilene) • History • The Network • Peering with International Networks • Science Applications • The New Challenge • The new Internet2 Network • A Broader Design • Topology • New Capabilities • Connections and Peerings

3. History • With the end of NSFnet and beginning of privatization, the ability to support large science applications was limited on the commodity network • The vBNS days brought much better connectivity, especially to super computer centers • NSF connections program was of great benefit to the community • Initial meetings of what was to become Internet2 in 1996 • There was a clear need to support the university research community • The Internet, even for “small” applications was not as robust as it is today • The university networking needs for science are not met today by the commodity Internet and likely will not be met in the near future.

4. The Internet2 Network Abilene • First Internet2 network began in mid 1998 with first backbone (Abilene) at 2.5 Gbps (OC-48 SONET backbone) • Partnership with Qwest, Cisco, and Nortel • NSF connections program played a crucial role in this period • Backbone upgrade to 10 Gbps started in 2001 and completed in early 2003 • Juniper became a partner at that time. • Supported advanced services like IPv6 and Multicast • Focus on performance and reliability primarily for e-Science applications

5. Current Internet2 Network (Abilene) Rick Summerhill

6. National Architecture (Current) Rick Summerhill

7. Internet2 Connectors and Peers • Connectors • 35 direct connections (OC-3c  10 Gbps) • 246 Primary Participants – research universities and labs • Cost recovery model • Regional to National • Campus to National and Regional • Campuses and Regionals typically also connect to the commodity network • Connections are Transparent - Many users don’t know they use Internet2 • Peerings • Connections through exchange points • 48 International agreements and 80+ International networks reachable through Abilene • Abilene supports transit for International peers • Peerings with other Federal Research and education networks in the US See: http://abilene.internet2.edu/

9. Science and Engineering Applications Russ Hobby Program Manager for Science and Engineering

10. More Than High-End Computing and Connectivity • Focused on sharing and making greater capabilities available across the science and engineering research communities • Allows applications to interoperate across institutions and disciplines • Ensures that data and software acquired at great expense are preserved and easily available to all • Empowers enhanced collaboration over distance and across disciplines. Report of the National Science Foundation Blue-Ribbon Advisory Panel on Cyberinfrastructure

11. Instrumentation Control Middleware Security and Access Data Generation Security Access Control Authentication Authorization Security Analysis Computation Simulation Program CI Functions and Interactions Collab Tools Security Researcher Control Program Education And Training Viewing 3D Imaging Security Display Tools Security Input Retrieval Display and Visualization Data Sets Storage . Search Data Input Archive Schema Metadata Data Directories Ontologies

12. Instrumentation Control Middleware Security and Access Data Generation Security Access Control Authentication Authorization Security Analysis Computation Simulation Program CI Functions and Interactions Collab Tools Security Researcher Control Program Education And Training Viewing TheNetwork 3D Imaging Security Display Tools Security Input Retrieval Display and Visualization Data Sets Storage . Search Data Input Archive Schema Metadata Data Directories Ontologies

13. The Applications Area is part of the Community side of Internet2 • Program managers are one channel through which research disciplines can interact with Internet2 and the broader Internet2 community • Disciplines that do not normally interact can share experiences and solutions to cyberinfrastructure problems • Joint projects can be created to accomplish more than one discipline can do by itself

14. Application Communities • One way to organize is by their stage of CI development: • Mature-Helping Develop CI • Using CI • Implementing • Planning • Thinking about it • Those with more experience teach those with less

15. High Energy and Nuclear Physics (HENP) • Physics has traditionally been one of the “power users” of all networks • Physicists are generating Terabytes of data (1,000,000,000,000 or 1x1012) per experiment from the CERN lab in Switzerland • Types of network usage: • Bulk data transfers that are extremely resistant to data loss. • VRVS expects multicast and low-latency/jitter networks for effective video conferencing As a mature community, we learn as much from the HENP community as they do from us (it could be argued that we are the students).

16. Network for Earthquake Engineering Simulation • NEES has multiple labs at various geographical locations • NEES has it’s own IT organization, NEESit, that provides support for the community • Internet2 works with NEESit and the NEES headquarters

17. Electronic Very-Long-Baseline Interferometry (eVLBI) • Astronomers collect data about a star from many different earth based antennae and send the data to a specialized computer for analysis on a 24x7 basis. • eVLBI is not as concerned with data loss as they are with long term stability. • The end goal is to send data at 1Gb/s from over 20 antennae that are located around the globe. Internet2 works closely with the eVLBI researchers and assists where we can. We are the teachers, but probably not for long.

18. NEON and Earthscope • Both NSF sponsored programs are in the implementation stages • Their research goals and science plans are fairly well understood. • Using CI and advanced networks to connect researchers, data and sensors is assumed. • The specific ways in which advanced networking will be integrated into their project still needs to be investigated As newer groups to the Internet2 community, Program Managers help identifying areas in which advanced networking experience of others can be used to ease their efforts

19. Sensor Networks • NEON and EarthScope are just two of many sensor network projects. Many of these projects are in the early planning stages and can benefit from sharing experiences. • They will need all the CI resources AND • Many will want to connect instrumentation at remote sites that lack any network connectivity • There is a need for national wireless coverage for sensor networks (connecting the last square kilometer)

20. Oceanography The Ocean Research Interactive Observatory Networks (ORION) is a program that focuses the science, technology, education and outreach of an emerging network of science driven ocean observing systems As a planning community, we help them learn from the experiences of others. We also observe their planning process so others may benefit.

21. Security • Spiderman’s Uncle said, “with great power comes great responsibility.” Security is a inter-community, cross-application concern. • Cutting edge applications are often indistinguishable from security breaches. • Implementing systems to support advanced applications while addressing security is a requirement for all future applications.

22. The Dynamic Virtual Global Collaboratory • Collaborating Virtual Organizations will become increasingly important • Consider the emerging e-science paradigm… • Global science • For example, astrophysics, astronomy, earth sciences, climate modeling, etc. • Global shared resources • Large Hadron Collider, radio telescopes, polar research stations, computational resources, etc. • Global collaborating science teams • E-VLBI, HEP, Genomic Research, etc • These “affinity groups” combine resources and people into a globally distributed virtual collaborating organizations to pursue a common discipline or objective.

23. The Coming Challenge • “Grid applications will incorporate in excess of 100,000 processors within 5 years.” • Dr. Larry Smarr, “On Vector” Workshop, UCSD Feb 2006 • “The Global Information Grid will need to store and access exabytes of data on a real-time basis by 2010” • Dr. Henry Dardy, Optical Fiber Conference, Los Angeles, CA USA, Mar 2006 • “Each LHC experiment foresees a recorded raw data rate of 1 to several PetaBytes/year” • Dr. Harvey Neuman (Cal Tech) • “US Bancorp backs up 100 TB financial data every night – now.” • David Grabski (VP Information Tech. US Bancorp), Qwest High Performance Networking Summit, Denver, CO. USA, June 2006. • “The VLA facility is now able to generate 700 Gbps of astronomical data and will reach 3.2 Terabits per second by 2009.” • Dr. Steven Durand, National Radio Astronomy Observatory, E-VLBI Workshop, MIT Haystack Observatory., Sep 2006.

24. The Networking Challenge • 10,000 processor compute cluster • 4 Gbyte/processor memory, 1 GigE NIC • Burst capability = 10 Tbps • Parallel and distributed clusters are incorporating nodes faster than Moore’s Law is reducing their size.. • Power requirements for single clusters will be too large to support in a single location - moved to geographically distributed clusters • How will they communicate? • These reflect some fundamental design decisions/assumptions of the existing (and original) internet architecture that may not be applicable today, especially for e-Science applications • The GENI initiative (NSF) hopes to construct a Global Environment for Network Innovation • Need to examine new architectures for e-Science applications now

25. The New Internet2 Network • Agreement with Qwest for the current Abilene Network ends in October of 2007 • The new Intenet2 network must be capable of supporting e-Science applications • Strategic Objectives • Ensure community control of the underlying (optical) network infrastructure • Leverage the capabilities of a global telecommunications leader • Providing carrier class reliability and expanded breadth of services, along with a broad set of partnership options • Capitalize on the latest technological advancements in networking • Create an asset that benefits the entire community - researchers, universities, regional optical networks, industry, government, K-12, and the international community

26. Architecture Goals • Develop an innovative optical system on a national footprint to serve the broad research and education community - a hybrid network capable of providing point-to-point services (circuits) together with an IP network • Community should have complete control of the lower layer optical system including provisioning and switching of wavelengths • The system should be capable of supporting network research in wide variety ways • Platform support for highly experimental projects to production services • Provide capabilities to support e-Science using new paradigms in networking

27. New Network Paradigms to Support e-Science • Example: eVLBI • The ability to dynamically create the necessary network to support particular experiments

28. The New Internet2 Network Rick Summerhill

29. Internet2 ESnet Partnership • Internet2 and ESnet have formed a partnership to build their respective networks on this DWDM footprint • ESnet hybrid network • An IP network connecting the labs • An lower layer network for deterministic services - SDN • Internet2 hybrid network • An IP network similar to the existing Abilene network • A layer 1 dynamically provisioned network providing deterministic services • Static and Dynamic services will be available to other partners

30. The New Internet2 Network - What’s Different? • Hybrid IP and Optical System utilizing Level3 fiber platform; equipment and fiber dedicated to Internet2, sparing and equipment maintenance by Level3, including SLA for wave system • Initially provisioned with ten times the capacity scalable to much larger • Dynamic provisioning of circuits and waves across the network within seconds • Connections and Peerings through IP and circuits (lightpaths)! • The ability to create circuits between between researchers and facilities such as international radio telescopes and particle accelerators • Geographically distributed Instruments appear as on a LAN rather than use the public network

31. National Architecture (New) Rick Summerhill

32. More Info Rick Summerhill rrsum@internet2.edu 734-352-4952 Russ Hobbyrdhobby@internet2.edu530-863-0513 www.internet2.edu science.internet2.edu