About the EnLightened Computing Project Prepared for Grid Computing Class at UNCW and UNCC

1. About the EnLightened Computing ProjectPrepared for Grid Computing Classat UNCW and UNCC Carla S. Hunt Senior Solutions Architect MCNC April 12, 2007

2. Your Presenter • Education: • B.S. Computer Science, University of Wilmington • M.S. Information Sciences, North Carolina Central University • Industry Experience • IBM, Network and Systems Engineer • Robert Taylor Associates, Chief Internet Services Engineer • Sentinel Services, Information Technology Consultant • MCNC, Senior Solutions Architect

3. MCNC & NCREN • Who we are: • MCNC (Microelectronics Center of NC) • We own and operate NCREN (NC Research and Education Network) • One of the most advanced statewide research and education networks in the country • Provide Internet, video and related services to University of North Carolina’s 16 campus system, Duke, Wake Forest, etc. • Non-profit organization, about 50 employees • History: • Commissioned by the General Assembly in 1980 to further economic development in the state of North Carolina • What we do: • Provide non-commodity Internet and related services to more than half a million users daily! • Network has an optical ring based architecture with speeds up to 2.5 Gbps

4. EnLIGHTened Computing Project • International recognition and collaboration • NSF seed funded • 10 Gbps national research testbed • Grid Middleware developed to co-schedule and co-allocate network and compute resources • Major Accomplishments: • First ever demonstration of trans-pacific, inter-domain co-scheduling of network and compute resources across two national grid computing research testbeds • First ever demonstration of External Network to Network Interface (E-NNI) capability to establish on-demand end-to-end light paths via Calient optical switches • First ever high definition video broadcast over the Internet of classroom instruction—LSU high performance computing class

5. EnLightened and G-Lambda GLIF Demonstration • Who: EnLIGHTened (US) and G-Lambda (Japan) • Where: 11th floor of AIST building in Tokyo, Japan • When: September 11th-13th, 2006 • What: Researchers from US and Japan demonstrated trans-pacific, inter-domain advance reservation of coordinated network and compute resources across two national research testbeds during GLIF’s 6th Annual Global Lambdagrid Workshop

6. Organization of Research Teams • Testbed • Network and Compute Resources • Application • Middleware • Performance and monitoring

7. EnLIGHTened Testbed

8. Testbed Components • Compute Resources: • Clusters • Storage • Client machines/displays • Network Resources: • Core Optical Switches • Other Network Devices

9. US Domain: Compute Resources RA1 VC1 CH1 BT1 X1U BT2 LA1

10. Caltech—LA1 • 4 nodes (LA1-LA4) • 2 nodes---4x2.2GHz Sun Opteron • 2 nodes---4x2.4GHz Tyan S2891 Opteron • RHELv4, 64 bit • Torque and Maui • Globus Toolkit • 10 Gig Nic cards • Shared resource sometimes

11. CCT at LSU: BT1 • Pelican.cct.lsu.edu • 14 nodes • 8 x 1.9 GHz IBM POWER5 processors • 16 GB RAM per node • AIX 5.2 w/ Loadleveler • Shared resource

12. CCT at LSU: BT2 • Santaka.cct.lsu.edu • SGI shared memory • 32 – 1.5 GHz Intel Itanium2 processors • 128 GB RAM per system • SUSE Enterprise 9/ PBSPro • 10 Gig Nic card (I think) • Shared resource

13. MCNC: RA1 • Kite1.enlightenedcomputing.org • Approximately 40 nodes available • 2x2.8GHz Intel Xeon • 4 GB RAM • RHELv4 • Torque/Maui • Globus Toolkit v 4.0.2 • Shared resource

14. VCL at NCSU: VC1 • vclb5-1.hpc.ncsu.edu • IBM Blade Center • RHELv4 • Torque/Maui • Globus Toolkit v 4.0.2 • Dedicated Resource

15. Coordinating Compute Resources • Policy for shared & borrowed resources • User accounts • Security • Installation and configuration of • Operating systems • Cluster management software (scheduler, resource mgr) • Grid middleware (globus toolkit and grid security) • IP address management • Other • Application requirements (software libraries or pkgs) • Performance tuning (tcp parameters for network card)

16. Request Network bandwidth and Computers Request Network bandwidth and Computers CRM CRM CRM CRM CRM CRM Reservation From xx:xx to yy:yy Reservation From xx:xx to yy:yy EL NRM KDDI NRM NTT NRM CRM CRM CRM CRM CRM CRM CRM CRM Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster Application (Visualization) Application (MPI) US JAPAN

17. G-lambda/Enlightened middleware coordination diagram Japan Application GL Grid Resource Scheduler EL Grid Resource Coordinator/ App. Launcher US Application GNS-WSI EL→GL wrapper GL: G-lambda EL: Enlightened GL request EL request EL→GL CRM wrapper EL→GL GNS-WSI wrapper HARC Acceptor US JAPAN EL NRM KDDI NRM NTT NRM CRM CRM CRM CRM CRM CRM CRM CRM Cluster Cluster Cluster Cluster Cluster Cluster Cluster Cluster

18. Resource map CH1 RA1 VC1 KMF FUK BT1 TKB BT2 (UR1) 0.11a.6 (UO2) 0.11a.7 (UO3) (UO1) 0.11a.2 4G (UR2) (UO4) X1 X1N 5G X1U 5G US 10.16a.2 Japan North (UR3) KAN X2N BT3 X1S 2G X2 X2S AKB LA1 OSA Japan South NR3 KHN

19. Performance and Monitoring • Tools • Mona Lisa • Monitor connections between Calient Optical Switches • Iperf • Test throughput end to end • Smartbits • Test throughput on pipes only

20. Demonstration Display

21. Conclusions • Recently forged new partnership with Phosphorus, EU researchers---”Collaboration Across Three Continents” • Questions about the project or these slides? • Remember any researcher is welcome to test his/her application on the EnLIGHTened Testbed!!