1 / 22

Network Troubleshooting Methods, Tools and Techniques

Sharing the knowledge behind the network. Network Troubleshooting Methods, Tools and Techniques. Presented by Scott Hogg shogg@lucent.com Rick Blum rickblum@lucent.com. Lucent Enterprise Professional Services. Network consulting services for enterprises

orsin
Télécharger la présentation

Network Troubleshooting Methods, Tools and Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Sharing the knowledge behind the network Network Troubleshooting Methods, Tools and Techniques Presented by Scott Hogg shogg@lucent.com Rick Blum rickblum@lucent.com

  2. Lucent Enterprise Professional Services • Network consulting services for enterprises • Premier provider of multivendor, data networking services • Software solutions to manage optimize network performance • 15,000+ engagements over 10+ years • Spin off in early 2002 • Seminar objectives • Explore the strengths of various troubleshooting methods and tools • Provide advice for troubleshooting Ethernet and routers

  3. Scientific Method for Troubleshooting

  4. data data AH data PH data SH data TH data NH data Data Unit (I field) FC F A C F Troubleshoot with OSI Model in Mind Outgoing Frame Construction Incoming Frame Reduction app "X" app “X" Application 7 Application 6 Presentation Presentation 5 Session Session Transport 4 Transport 3 Network Network Data link 2 Data link Physical Physical 1 Transmitted Frame

  5. Troubleshooting Tools • Network Management Systems (NMS) • Element Management Systems (EMS) • Vendor-specific • Network application aware monitoring/ trending - proactive • FCAPS model • Logs, event correlation, time of day • Time synchronized logs (NTP) • Server stats, CPU, memory, disk, I/O

  6. Troubleshooting Tools • Upper OSI Layers • Protocol analyzers - Type of protocols • Middle OSI Layers • SNMP, RMON probes, remote probes • Protocol analyzers • Lower OSI Layers • Wire scopes, cable testers, optical power level meters, OTDRs, BERT • Voice convergence • Voice quality tester • Tools for testing echo, delay, jitter, etc.

  7. Troubleshooting Impact • In-Band techniques • Ping – ICMP echo request/reply • Traceroute – TTL and High UDP Port • Telnet – TCP port 23 connection • Mock/synthetic transactions • Loopback interfaces - BERT • Protocol analysis • Out-of-Band techniques • Isolates management network from production network • Can’t troubleshoot a network with problems and congestion with the network itself • Access to manage equipment is critical

  8. Ethernet Cabling Problems • Troubleshooting fiber cables • Multi-mode vs. single-mode • SC connectors are popular • Dirt and Grime - polishing • Re-terminate fiber if necessary • Make sure GBIC matches • 62.5-um vs. 50-um vs. 10-um (SM) fiber cables • Cat 5 certification and testing • Pairs, impedance, etc. • Cable length • Bends and kinks • 5-4-3 maximum topology rule of repeaters

  9. Media Layer Ethernet Problems • Collisions (remote, local, late) • Runts, giants, jabbers, long • FCS errors • Alignment errors • Frame misalignment errors • Capturing these bad frames • Software-based protocol analyzers • Hardware-based protocol analyzers

  10. Troubleshooting Ethernet • No link light!? • Auto-Negotiation (Speed/Duplex) • Packet capture effect – wire hogging • Faster system collides, then chooses lower back-off time • Increased secondary collisions • Phantom MAC addresses • 55555555 or AAAAAAAA from jam bits • 01010101 = 55 or 10101010 = AA (10Mbps) • 10110000 = D0 or 01010011 = 53 (100Mbps)

  11. Troubleshooting Ethernet • Hubs (multiport repeater) • Device is collision domain • Switches (multiport bridge) • Each port is own collision domain • Ethernet switches don’t forward bad frames • Integrated Layer 2-3 and Layer 4-7 devices • View Ethernet switch’s MAC address table • View spanning tree (IEEE 802.1d) parameters • Check ISL or 802.1q trunking • Look at switch port stats • Traffic load, error rates, # of broadcasts and multicasts, # of discards

  12. “Sniffing” Ethernet • Capturing Ethernet frames • Port mirroring, port aliasing, SPAN, tap, pass-through analyzer • Capture traffic flowing in both directions • Put analyzer “in-line” • Use a hub – temporarily • Port-level security can make this difficult • VLAN decode is helpful (ISL or 802.1q) • Network-based intrusion detection competition

  13. Troubleshooting Routers – Checking the Basics • Check Layers 1 & 2 • View interface statistics/status • Clear counters, then watch them increment • Check Layer 2 to Layer 3 mappings – ARP caches • Ping/traceroute in both directions • Check last router to reply to traceroute • Try extended ping options • Make sure hosts configured properly • Check addresses, numbers, and name resolution • Operational techniques • Router logs and SNMP traps • Use router debugging • Change/Configuration management

  14. Routing Table Problems • Routing protocol table and forwarding table • “Show ip route” vs. “show ip ospf route” • Inactive or flapping routes • Check routing table and routing metrics for specific routes – check in both directions • Clear out specific route or entire routing table and let it build again – last resort • Check route summarization and redistribution • Administrative distance (believability/favorability) • Equal-cost load balancing • Asymmetrical routing • Open jaw routes • Black hole routes

  15. Distance-Vector Protocols • Hop-by-hop updates get lost • Check RIPv1 and RIPv2 compatibility • Problems caused by summarization • Automatic redistribution into other routing protocols • Discontiguous subnet mask problems • Is split-horizon enabled? • Convergence times may be longer than ever thought possible (~10-15 minutes)

  16. Link-State Protocols • Link-state metrics • Neighbor adjacencies • Understand state table for protocol • Timers should be same on both neighbors • Router authentication • Check redistribution with distance-vector protocols • Summarization and discontiguous networks • View topology/route database • What routes made it into forwarding table? • Think about storing copies of the routing tables in a file for future reference – anomaly checking • Save the router’s state – on any “normal day” when the network is fully converged

  17. Path-Vector Protocols (BGP-4) • Peering takes place on TCP port 179 • Neighbors should be in “established” state • Reset peer – soft reconfiguration • BGP – exchanges just hello’s after initial peering • Synchronization with IGP • Transit AS - use synchronization • Check BGP table & decision algorithm • View routes in BGP table to see which ones make it into forwarding table • Know BGP’s attributes – well-known, mandatory, optional, transitive, non-transitive • Route flap dampening

  18. The Bottom Line • Use good methodology • Document actions and results • Leverage all tools to gather information • Use protocol analyzer to help troubleshoot problems • Understand protocols you are troubleshooting

  19. Be a Super Sleuth • Sherlock Holmes, Quincy, Colombo, Matlock, Hercule Poirot, Miss Marple • Start at the scene of the crime • Find evidence, pay attention to details • Follow leads even if they’re weak • Dig deeper, use forensic techniques • Never give up the Quest!

  20. Lucent Professional Consulting Services for Enterprises • IP Data Networking • Network Management • Performance Engineering • Network Security • Microsoft Technologies • Business Consulting

  21. Glossary • OSI – Open System Interconnection • OSPF – Open Shortest Path First • OTDR - Optical Time Domain Reflectometer • RIP – Routing Information Protocol • RMON - Remote MONitoring extensions to SNMP • SC connector – Fiber optic cable connector • SM - Single mode • SNMP – Simple Network Management Protocol • SPAN – Switch Port Analyzer • TCP – Transmission Control Protocol • TTL – Time To Live • UDP – User Datagram Protocol • VLAN – Virtual Local Area Network • ARP – Address Resolution Protocol • AS - Autonomous System • BERT – Bit Error Rate Test • BGP – Border Gateway Protocol • Cat – Category • FCAPS – Fault, Configuration, Availability, Performance, Security • FCS – Frame Check Sequence • GBIC – GigaBit Interface Converter • ICMP – Internet Control Message Protocol • IGP – Interior Gateway Protocol • ISL – Inter-Switch Link • IP – Internet Protocol • MAC – Media Access Control • NTP – Network Time Protocol

  22. Resources • Knowledge Web Seminar • Network Fault Management Tools and Techniques http://www.lucent.com/knowledge/documentdetail/0,1494,inContentId+12680-inLocaleId+1,00.html • Other • Telecommunications Management Network Roadmap http://www.itu.int/TMN/ • Ethernet Web sites http://www.faqs.org/faqs/LANs/ethernet-faq/ http://wwwhost.ots.utexas.edu/ethernet/ http://www.alumni.caltech.edu/~dank/fe/ http://www.cavebear.com/CaveBear/Ethernet/ http://grouper.ieee.org/groups/802/3/index.html http://www.10gea.org • General Network Troubleshooting Web site http://www.networktroubleshooting.com/

More Related