Caging the Wild Resnet

1. Caging the Wild Resnet Dennis Rice, Director, Infrastructure Support Group Dave Edick, Systems Administrator Computer And Technology Services, Saint Mary’s College of California

2. Overview • Our Residential Network suffered frequent outages locally and globally due to the actions of users, and infected and defective computers • User equipment and practices difficult to control. • Many solutions were looked at; Most required client installation etc. – Intrusive, user technical skills low, and required resources we didn’t have • Novel solution proposed by one of our senior engineers: Use capabilities of Layer 3 switching to eliminate the problems of shared networks.

3. Saint Mary’s College of California • Located in Moraga, California (East Bay). • Founded in San Francisco in 1863 by Archbishop Joseph Alemany. • Operated by the De La Salle Christian Brothers since 1868. • Current Campus in Moraga built and occupied in 1927 and 1928. • Current student enrollment approximately 4400 full and part time undergraduate and graduate students • Full and Part-time Staff and faculty number around 980

4. SMC Residence Halls • 1660 Residential Beds • 22 Residence Halls • Most have two students per room. • Freshmen guaranteed housing, Sophomores can renew; Juniors and Seniors in a Lottery for remaining beds • Students buy and maintain (or more regularly fail to maintain) their own computers – no in-house service for non-SMC owned computers

5. Residential Network (ResNet) • Up to 2000 Network access in the resident halls was via dial-up • Summer of 2000: • First fiber optic infrastructure and internal Ethernet wiring installed to two newly built residence halls and two older dorms – only dorms with internal Ethernet wiring. • The rest: TUT

6. Residential Network (ResNet) • Required modem • 1350 beds outfitted with TUT modems that summer • Better than dial-up, but failure prone • Very time-consuming to service and keep running • We used TUT High-speed DSL modems for uplinks to network core • Tut Systems Long Run Technology - Permitted using existing copper phone line infrastructure to carry data to each telephone jack along with voice. 1MB bandwidth • Gift to College

7. Residential Network (ResNet) • After 2000, gradually started replacing TUT with Ethernet and fiber optic infrastructure in residence halls. • Started wiring residential halls one by one for Fast Ethernet. • Extended fiber cable runs area by area as the work fit into other improvement projects funded by Bond.

8. Problems • User generated problems started showing up in this mixed architecture. Entire residence Halls or larger areas of ResNet would experience outages for a variety of reasons. This kept us hopping. • User connects a wireless router to ResNet backwards, paralyzing the hall network because two DHCP servers are giving out addresses. Difficult to find in a 100 bed dorm • User loops a Layer 2 switch by plugging the same cable into both data jacks in their room, causing traffic overloads in network backbone. • Illegal routing broadcasts caused switches to intermittently stop passing traffic.

9. Problems • Effects of Malware Infections • Broadcast and ARP storms from one or two infected computers could fill up the bandwidth of a TUT DSL uplink, blocking all other traffic. • Easy worm propagation over shared network • students ignorant of measures to protect their computers • Authentication Appliance (Bluesocket) also vulnerable to actions of malware exploits, affecting the whole Residential Network

10. Welshia • August 2003 “Blaster” worm and variant “Welshia” released on an unsuspecting world. • August - September 2003: SMC Residential students arrive with computers already infected with Welshia, bypassing firewall etc. • Worm rapidly spreads over entire Residential network and to academic-staff network • Infected computers generate ever larger packet storms. • ResNet grinds to a halt.

11. Welshia • Severe restrictions (and patches from Bluesocket) enabled us to return some limited network functioning after a one day complete outage, but insufficient for academic mission. • All other IT projects and most services halted to devote time of entire staff to cleanup of residential students’ computers. • Cleanup lasted into November. • Damaged the confidence of faculty in web and network based learning technologies as students struggled with a poorly functioning computers and network during half a semester.

12. What to do? • Welshia outbreak and difficult recovery focused our attention on the vulnerability of ResNet to interruption by unprotected and unmanaged student computers, the majority of which are owned and maintained by users with minimal knowledge or interest in properly securing them.

13. What to do? • Three paths clearly identified for action • Policy • Publicity • Technology • Several vendor and open source solutions explored • Client-based Network Access Control systems • Example: Cisco Secure Access • IPS/IDS systems for perimeter security • Example: Tipping Point

14. What to do? • Compliance solutions presented many difficulties • Disruption at beginning of Fall term • all computers would have to install client • Large additional support load on Help Desk at busiest time of year. • Computers that did not comply would have to be fixed – no staff to do this task. • Potential for frustration and more bad publicity high.

15. Inspiration • One of our Senior Systems Administrators, Dave Edick, thought about these problems and came up with a unique solution, one that promised: • the elimination of the means by which our network could be made to fail or be compromised. • did not rely on any compliance by the users or cause any disruption to their use of the Residential Network.

16. The Idea • Use the capabilities of Layer 3 switching to subnet each jack in every Resnet room • Broadcast traffic limited to jack network and local switch; packet storms eliminated • DHCP provided in local switch • ACL’s block dangerous ports • Any misconnected personal equipment (like a wireless router) can only affect the local jack, rather than a whole building.

17. Other Benefits of Layer 3 switching • L3 switching makes sniffing useless. Sniffer will only see their own traffic. Not even broadcasts from other machines in the ResNet can be seen • L3 switching prevents one user from impersonating another user. • L3 blocks non IP traffic (ipx, netbios, etc) which would be considered "unauthorized". • L3 prevents users accessing open file shares on other user's machines. • Hackers can’t capture packets by overloading switch tables (common trick to compromise L2 switches)

18. Network Design • Criteria for switch selection • Layer 3 capable • Able to provide local DHCP and VLANS • Robust and reliable • 24 10/100 ports, two gigabit ports (must have LX, SX and gigabit twisted pair GBIC’s available) in a 1RU package as the basic building block • Stackable at gigabit speeds • Able to store and run a large amount of code (current configuration is over 400 lines). • Must fit in a “Hubbell can.”

19. Switch selected: Cisco 3550

20. Hubbell ReBox Wall Enclosure with Mounted Cisco Switch

21. Network Design • Additional elements • Leverage installed and planned installation of fiber optic cable to every residence hall to provide gigabit speed up link connections to core. • Eliminate choke point of Bluesocket authentication appliance - Authentication no longer needed. • Ports and room jacks to be renumbered to be identical with the assigned phone extension • IP address ranges assigned to each jack can be cross checked to identify user for DMCA etc.

22. Switch Configuration • Five sections of code: • IP address range configuration • Per port DHCP configuration • Each port has five usable addresses • Port IP configuration • Incoming packet access control list • Outgoing packet access control list • Configuration is scripted

23. Configuration Scripting (in English) • get switch name, subnet number, and list of phone extensions • write out basic IP config substituting subnet (for management ip 4th octet) and switch name • write out both acls • loop for each port (1-24) • gateway=(port-1)*8+1 • get phone extension from list    • write port config substituting port, subnet, gateway, and phone number    • write dhcp config substituting port, subnet and gateway • repeat

24. Configuration examples: • IP block configuration ip subnet-zeroip routingip domain-name stmarys-ca.eduip name-server 10.1.1.101ip name-server 10.1.1.102!interface Vlan1 ip address 10.1.127.[subnet] 255.255.255.0!ip classlessip route 0.0.0.0 0.0.0.0 10.1.127.254

25. Configuration examples: • DHCP configuration ip dhcp excluded-address 10.1.[subnet].1!ip dhcp pool port-[port].dhcp   network 10.1.[subnet].[gateway] 255.255.255.248   default-router 10.1.[subnet].[gateway]   domain-name stmarys-ca.edu   dns-server 10.1.1.101 10.1.1.102

26. Configuration examples: • Port IP Configuration interface FastEthernet0/[port] description [phone] no switchport ip address 10.1.[subnet].[gateway] 255.255.255.248 ip access-group 101 in ip access-group 102 out

27. Configuration examples: • Incoming packet access control list access-list 102 remark access list for incoming packetsaccess-list 102 permit ip host [insert file server address] anyaccess-list 102 deny udp any any eq netbios-nsaccess-list 102 deny udp any any eq netbios-dgmaccess-list 102 deny udp any any eq netbios-ssaccess-list 102 deny udp any any eq 445access-list 102 deny tcp any any eq 137access-list 102 deny tcp any any eq 138access-list 102 deny tcp any any eq 139access-list 102 deny tcp any any eq 445access-list 102 permit ip any any

28. Configuration examples: • Outgoing packet access control list access-list 101 remark outgoing packet access listaccess-list 101 permit tcp any any establishedaccess-list 101 permit tcp any host [insert mail server address] eq smtpaccess-list 101 permit ip any host [insert file server address]access-list 101 deny tcp any any eq smtpaccess-list 101 deny tcp any any eq 137access-list 101 deny tcp any any eq 138access-list 101 deny tcp any any eq 445access-list 101 deny udp any any eq netbios-nsaccess-list 101 deny udp any any eq netbios-dgmaccess-list 101 deny udp any any eq netbios-ssaccess-list 101 permit ip any any

29. Monitoring and Backup • Scripts running on Network Services server • Status Page with a table of ports for each switch. • Port number and corresponding 4 digit data and phone number assigned to jacks in room. • Port link status (active connection or not) • Port IP info - Gateway and usable addresses • Configuration backup script – TFTP server • Telnet’s into each switch and TFTP’s the configuration back to the server.

30. Monitoring and Backup • Port status page

31. Monitoring and Backup • Port and Switch on-line reset utility

32. Finances • The project required over 100 switches, plus GBIC’s and other equipment, cables etc. • With the help of our Advancement office and led by our CTO, Ed Biglin, we wrote a grant request to the Fletcher-Jones Foundation in June of 2004. Fortunately, the project was funded by them at $250,000. • This amount purchased the needed equipment, but there was no funding for implementation labor.

33. Installation • Because of the small amount of internal resources available to this project, installation was one residence hall at a time • Installation of switches was originally scheduled to be launched and completed over the Summer break of 2005 • Problems and scheduling conflicts were encountered • Twelve residence halls completed by beginning of Fall term 2005 • Fiber optic infrastructure build out completed and the last TUT equipped dorms rewired for Fast Ethernet by the beginning of Fall term 2005 as well. • Switch installation in all residence halls was finally completed during Spring term 2006

34. Results • The stability and reliability of the Residential Network has improved dramatically, as has customer satisfaction • Time spent on ResNet network problems by network engineers and techs has been reduced tremendously. • One tech said, “ResNet used to run me ragged. This year, most of the calls have gotten me to help individual students having trouble connecting, but none of them involved tracking down dozens of infected computers or bad router connections – the sort of problem that used to take days or a week to fix.”

35. Results • Comparison of Trouble tickets from Fall of 2004 and Fall 2006. • In Fall 2004, we had 36 tickets for building or larger area outages – many of these represented multiple customer calls. • In Fall 2006, we had only two similar tickets, for a floor rather than a building outage, both caused by the failure of a switch that was quickly replaced. • Comparison of student surveys also indicated a large increase in satisfaction with network performance.

36. Questions?