1 / 29

Debug basic operation

Troubleshoot packet loss issues in the packet forwarding process, including hardware and software causes. Learn debugging commands and best practices.

mhead
Télécharger la présentation

Debug basic operation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Debug basic operation Lei Shi 2017/0718

  2. You can add or delete by your requirements, and adjust the size You can add or delete by your requirements, and adjust the size You can add or delete by your requirements, and adjust the size Packet forwarding process Debugging basic operation You can add or delete by your requirements, and adjust the size # # # # # #

  3. Packet forwarding process • Packet forwarding process • Switch • CPU hardware • software handle

  4. Packet forwarding process

  5. Troubleshooting for packet loss • External port issue Check duplex, speed negotiation, LAN cable quality, utilization of interface bandwidth show controller slot 0 port x statistic to check the packets statistics in interface , is the bad packets existing? ( please change the cable , and change the port negotiation type to auto or force 1000M)

  6. Packets loss by hardware • show controller packet-drop

  7. CPU • show cpu/detail ； check CPU utilization

  8. Troubleshooting for packet loss Packets loss by software, the reason is complex in this case . Non-bug for packets loss ： configuration causes loss , for example( AD, policy, QOS, session- limit, etc.) exceed the device capacity Bug for packets loss ： Software BUG We mainly talk about packets loss by configuration

  9. Preface before perform debugging operation • There will be some risk if you perform the debug operation. Especially when there are huge traffic in the device, • It will cause the high utilization of CPU resources. • Please use the debug function with caution, the debugging operation is prohibited when CPU utilization exceeds 50% • It is better to have the support from Hillstone engineer for the debug process. • End-user is not recommended to do.

  10. Debugging command • Debug dp basic # diagnose the basic traffic flow • Debug self # need to enable debug self when the source IP or destination IP is Firewall self IP. • Debug dp snoop # You can view TCP information, it will rise the CPU utilization to 20%-30%, so enable snoop with caution. • Debug arp #diagnose ARP packets • Debug vpn #diagnose the process about VPN negotiation • Debug ha #diagnose process about HA negotiation • Etc.

  11. DEBUG operation cases TOP:

  12. DEBUG operation cases  • Configure debug dp filter，for example, if I only care about the packet to destination port tcp80: SG-6000(config)# debug dp filter src-ip 192.168.1.254 dst-ip 1.1.1.2 dst-port 80 assigned id: 1 SG-6000(config)# debug dp filter src-ip 1.1.1.2 src-port 80 assigned id: 2 2. Configure output of debug log SG-6000(config)# logging debug on #enable the debug log SG-6000(config)#logging debug to buffer #Send logto buffer SG-6000(config)# no logging debug to console #No logsent to console

  13. DEBUG operation cases   3. Enable the debug function SG-6000(config)# debug dp basic SG-6000(config)# debug dp drop 4. Check debuginformation SG-6000[DBG](config)# show logging debug 5. Clear debughistory SG-6000[DBG](config)# clear logging debug 6. Finally, please disable the debugfunction SG-6000[DBG](config)# undebugall Or double click the “ESC”key in keyboard to disable all debug; if you only want to disable the debug for specific function, please use the command “undebug”, for example：“undebugdp basic”

  14. Case 1 SG-6000[DBG](config)# show logging debug 2015-10-27 11:20:37, DEBUG@FLOW: core 1 (sys up 0x3d42ce1c ms): rx_handle_prepare: 0024.8c01.4056->001c.544d.f60a, size 66, type 0x800, vid 0, port ethernet0/0 dp_prepare_if_for_pak Switchid is 9(interface ethernet0/0) port ethernet0/0 Switchidis 9(interface ethernet0/0) port ethernet0/0 ,pakiif=ethernet0/0 rx_handle_preparei_if is ethernet0/0 rx_handle_prepare calling dp_sanity ethernet0/0 Start l3 forward Packet: 192.168.1.254 -> 1.1.1.2, id: 2305, ip size 52, prot: 6(TCP): 51809 -> 80 ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 6 dp_prepare_pak_lookupsrcip: 192.168.1.254, dstip: 1.1.1.2, src-port:51809, dst-port:80, prot 6 No session found, try to create session dp_first_crt_sess_init_flow0_from_pak_iif: set cpuid 0 -----------------First path creating new session----------------- dp_sess_sm_transtion: Do session state machine transtion, state: 0, event: 0! allocate pending session and install flow0 begin lookup predefine prot:6 port:80 Identified as app HTTP (prot=6). timeout 1800. --------VR:trust-vr start-------- 192.168.1.254:51809->1.1.1.2:80 Failed to get route to 1.1.1.2 Dropped: Can't find forwarding route. Abort!! dp_sess_sm_transtion: Do session state machine transtion, state: 1, event: 4! deny session:flow0 src 192.168.1.254 --> dst 1.1.1.2 Deny session installed successfully --------VR:trust-vr end-------- -----------------------First path over (session not created) Droppped: failed to create session, drop the packet (action=0) Reason : missing destination route

  15. Case 2 SG-6000[DBG](config)# shlogg deb 2015-10-27 11:32:04, DEBUG@FLOW: core 1 (sys up 0x3d4d48c2 ms): rx_handle_prepare: 0024.8c01.4056->001c.544d.f60a, size 62, type 0x8 00, vid 0, port ethernet0/0 dp_prepare_if_for_pak Switchid is 9(interface ethernet0/0) port ethernet0/0 Switchid is 9(interface ethernet0/0) port ethernet0/0 ,pakiif=ethernet0/0 rx_handle_preparei_if is ethernet0/0 rx_handle_prepare calling dp_sanity ethernet0/0 Start l3 forward Packet: 192.168.1.254 -> 1.1.1.2, id: 4431, ip size 48, prot: 6(TCP): 52015 -> 80 ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 6 dp_prepare_pak_lookupsrcip: 192.168.1.254, dstip: 1.1.1.2, src-port:52015, dst-port:80, prot 6 No session found, try to create session dp_first_crt_sess_init_flow0_from_pak_iif: set cpuid 0 -----------------First path creating new session----------------- dp_sess_sm_transtion: Do session state machine transtion, state: 0, event: 0! allocate pending session and install flow0 begin lookup predefine prot:6 port:80 Identified as app HTTP (prot=6). timeout 1800. --------VR:trust-vr start-------- 192.168.1.254:52015->1.1.1.2:80 Get nexthopif_id: 16, flags: 0, nexthop: 2.2.2.2 Found the reverse route for force or prefer revs-route setting --------VR:trust-vr end-------- Start policy lookup. Pak src zone trust, dst zone trust, prot 6, dst-port 80. No policy set, default ===DENY=== Dropped: Can't find policy/policy denied. Abort!! dp_sess_sm_transtion: Do session state machine transtion, state: 1, event: 4! deny session:flow0 src 192.168.1.254 --> dst1.1.1.2 Deny session installed successfully -----------------------First path over (session not created) Droppped: failed to create session, drop the packet (action=0) Reason: no relevant policy to permit traffic

  16. Case 3 SG-6000[DBG](config)# shologg deb 2015-10-27 11:35:12, DEBUG@FLOW: core 1 (sys up 0x3d5029a3 ms): rx_handle_prepare: 0024.8c01.4056->001c.544d.f60a, size 66, type 0x8 00, vid 0, port ethernet0/0 dp_prepare_if_for_pak Switchid is 9(interface ethernet0/0) port ethernet0/0 Switchid is 9(interface ethernet0/0) port ethernet0/0 ,pakiif=ethernet0/0 rx_handle_preparei_if is ethernet0/0 rx_handle_prepare calling dp_sanity ethernet0/0 Start l3 forward Packet: 192.168.1.254 -> 1.1.1.2, id: 4929, ip size 52, prot: 6(TCP): 52065 -> 80 ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 6 dp_prepare_pak_lookupsrcip: 192.168.1.254, dstip: 1.1.1.2, src-port:52065, dst-port:80, prot 6 No session found, try to create session dp_first_crt_sess_init_flow0_from_pak_iif: set cpuid 0 -----------------First path creating new session----------------- dp_sess_sm_transtion: Do session state machine transtion, state: 0, event: 0! allocate pending session and install flow0 begin lookup predefine prot:6 port:80 Identified as app HTTP (prot=6). timeout 1800. --------VR:trust-vr start-------- 192.168.1.254:52065->1.1.1.2:80 Get nexthopif_id: 16, flags: 0, nexthop: 2.2.2.2 Found the reverse route for force or prefer revs-route setting --------VR:trust-vr end-------- Start policy lookup. Pak src zone trust, dst zone trust, prot 6, dst-port 80. Policy 3 matches, ===PERMIT=== crt_sess->flow0_io_cpuid 0

  17. flow0 src 192.168.1.254 --> dst 1.1.1.2 with nexthop 2.2.2.2 ifindex 16 flow1 src 1.1.1.2 --> dst 192.168.1.254 nexthop not lookup or invalid flow0's next hop: 192.168.1.254 flow1's next hop: 2.2.2.2 crt_sess->revs_rres.gw: 192.168.1.254, crt_sess->forw_rres.gw 2.2.2.2 Calculate flow1 hash, srcip: 1.1.1.2, dstip: 192.168.1.254, lports: 50cb61, prot: 6, token: 1 in flow_firstprofile_merge ------sess:42,app :10 init in first proc Initpak_cache_list in dp_trigger_app_init() Application 10 initfailed.ret=[-1] crt_sesspolicy_flag is 0000, session flag1 is 100000 HTTP: create session: atomic bit 0 dp_sess_sm_transtion: Do session state machine transtion, state: 1, event: 3! The following session is installed session: id 42, prot 6, flag0 0,flag1 100000, created 1028663, life 1800 flow0(if id: 9 flow id: 84 flag: 40000810):192.168.1.254:52065 ->1.1.1.2:80 flow1(if id: 16 flow id: 85 flag: 800): 1.1.1.2:80 ->192.168.1.254:52065 Session installed successfully -----------------------First path over--------------------- Found the session 42 session: id 42, prot 6, flag0 0,flag1 100000, created 1028663, life 1800 flow0(if id: 9 flow id: 84 flag: 40000810):192.168.1.254:52065 ->1.1.1.2:80 flow1(if id: 16 flowid: 85 flag: 200810): 1.1.1.2:80 ->192.168.1.254:52065 TCP seqence handling flow mac_copy L3 forward, out if is ethernet0/7, mtu 1500 Reason :Missing SNAT, you need configure SNAT at egress for traffic that goes to Internet. If in LAN, you need follow the requirement. In this case,the bidirectional addresses of sessionare private IP, so the SNAT rule is missed.

  18. Case 3 SG-6000[DBG](config)# shlogg deb 2015-10-27 11:59:32, DEBUG@FLOW: core 1 (sys up 0x3d667039 ms): rx_handle_prepare: 0024.8c01.4056->001c.544d.f60a, size 62, type 0x8 00, vid 0, port ethernet0/0 dp_prepare_if_for_pak Switchid is 9(interface ethernet0/0) port ethernet0/0 Switchid is 9(interface ethernet0/0) port ethernet0/0 ,pakiif=ethernet0/0 rx_handle_preparei_if is ethernet0/0 rx_handle_prepare calling dp_sanity ethernet0/0 Start l3 forward Packet: 192.168.1.254 -> 1.1.1.2, id: 9874, ip size 48, prot: 6(TCP): 52449 -> 80 ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 6 dp_prepare_pak_lookup srcip:192.168.1.254, dstip: 1.1.1.2, src-port:52449, dst-port:80, prot6 No session found, try to create session dp_first_crt_sess_init_flow0_from_pak_iif: set cpuid 0 -----------------First path creating new session----------------- dp_sess_sm_transtion: Do session state machine transtion, state: 0, event: 0! allocate pending session and install flow0 begin lookup predefine prot:6 port:80 Identified as app HTTP (prot=6). timeout 1800. --------VR:trust-vr start-------- 192.168.1.254:52449->1.1.1.2:80 Get nexthopif_id: 16, flags: 0, nexthop: 2.2.2.2 Found the reverse route for force or prefer revs-route setting Matched source NAT: snat rule id:1 Matched source NAT: source port52449->port52449

  19. --------VR:trust-vr end-------- Start policy lookup. Pak src zone trust, dst zone trust, prot 6, dst-port 80. Policy 3 matches, ===PERMIT=== crt_sess->flow0_io_cpuid 0 flow0 src 192.168.1.254 --> dst 1.1.1.2 with nexthop 2.2.2.2 ifindex 16 flow1 src 1.1.1.2 --> dst 2.2.2.1 nexthop not lookup or invalid flow0's next hop: 192.168.1.254 flow1's next hop: 2.2.2.2 crt_sess->revs_rres.gw: 192.168.1.254, crt_sess->forw_rres.gw 2.2.2.2 Calculate flow1 hash, srcip: 1.1.1.2, dstip: 2.2.2.1, lports: 50cce1, prot: 6, token: 1 in flow_firstprofile_merge ------sess:55,app :10 init in firstproc Initpak_cache_list in dp_trigger_app_init() Application 10 initfailed.ret=[-1] crt_sesspolicy_flag is 0000, session flag1 is 100000 HTTP: create session: atomic bit 0 SESS_LIMIT:Dropped: Session limit module limit reached dp_sess_sm_transtion:Do session state machine transtion, state: 1, event: 4! deny session:flow0 src 192.168.1.254 --> dst 1.1.1.2 Deny session installed successfully Dropped: Failed to create session -----------------------First path over (session not created) Droppped: failed to create session, drop the packet (action=0) Reason :Session limits configured at device, relevant session is blocked. Need to check the configuration of session-limit.

  20. Debug Parameter Instruction 4.1 Debug information If you want view the TCP information, please enable the “debug dp snoop”, below is the comparison between parameter of TCP three- way-handshake and packet information captured by Wireshark from PC

  21. Text (bullet point) SG-6000(config)# showlogging debug 2015-10-28 13:25:58, DEBUG@FLOW: core 1 (sys up 0x17615f ms): 1532: (i) len=66 0024.8c01.4056->001c.544d.f60a/800 192.168.1.254->1.1.1.2/6 vhl=45, tos=00, id=23316, frag=16384, ttl=64, tlen=52 tcp:ports 60164->80, seq=915561528, ack=0, flag=32770/SYN rx_handle_prepare: 0024.8c01.4056->001c.544d.f60a, size 66, type 0x800,vid 0, port ethernet0/0 dp_prepare_if_for_pak Switchid is 9(interface ethernet0/0) port ethernet0/0 Switchid is 9(interface ethernet0/0) port ethernet0/0 ,pakiif=ethernet0/0 rx_handle_preparei_if is ethernet0/0 rx_handle_prepare calling dp_sanity ethernet0/0 Start l3 forward Packet: 192.168.1.254 -> 1.1.1.2, id: 23316, ip size 52, prot: 6(TCP): 60164 -> 80 ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 6 dp_prepare_pak_lookupsrcip: 192.168.1.254, dstip: 1.1.1.2, src-port:60164,dst-port:80, prot 6 No session found, try to create session dp_first_crt_sess_init_flow0_from_pak_iif: set cpuid 0 -----------------First path creating new session----------------- dp_sess_sm_transtion: Do session state machine transtion, state: 0, event: 0!

  22. Text (bullet point) allocate pending session and install flow0 begin lookup predefine prot:6 port:80 Identified as app HTTP (prot=6). timeout 1800. --------VR:trust-vr start-------- 192.168.1.254:60164->1.1.1.2:80 Get nexthopif_id: 16, flags: 0, nexthop: 2.2.2.2 Found the reverse route for force or prefer revs-route setting : source port60164->port60164 --------VR:tMatched source NAT: snat rule id:1 Matched source NATrust-vr end-------- Start policy lookup. Pak src zone trust, dst zone trust, prot 6, dst-port 80. Policy 3 matches, ===PERMIT=== crt_sess->flow0_io_cpuid 0 flow0 src 192.168.1.254 --> dst 1.1.1.2 with nexthop 2.2.2.2 ifindex 16 flow1 src 1.1.1.2 --> dst 2.2.2.1 nexthop not lookup or invalid flow0's next hop: 192.168.1.254 flow1's next hop:2.2.2.2 crt_sess->revs_rres.gw: 192.168.1.254, crt_sess->forw_rres.gw 2.2.2.2 Calculate flow1 hash, srcip: 1.1.1.2, dstip: 2.2.2.1, lports: 50eb04, prot: 6, token: 1 in flow_firstprofile_merge

  23. Text (bullet point) ------sess:58,app :10 init in first proc Initpak_cache_list in dp_trigger_app_init() Application 10 initfailed.ret=[-1] crt_sesspolicy_flag is 0000, session flag1 is 100000 HTTP: create session: atomic bit 0 dp_sess_sm_transtion: Do session state machine transtion, state: 1, event: 3! The following session is installed session: id 58, prot 6, flag0 a,flag1 100000, created 1532, life 1800 flow0(if id: 9 flow id: 116 flag: 40000810):192.168.1.254:60164->1.1.1.2:80 flow1(if id: 16 flow id: 117 flag: 800): 1.1.1.2:80->2.2.2.1:60164 Session installed successfully -----------------------First path over--------------------- Found the session 58 session: id 58, prot 6, flag0 a,flag1 100000, created 1532, life 1800 flow0(if id: 9 flow id: 116 flag: 40000810):192.168.1.254:60164->1.1.1.2:80 flow1(if id: 16 flow id: 117 flag: 200810): 1.1.1.2:80->2.2.2.1:60164 TCP seqence handling

  24. Text (bullet point) flow mac_copy L3 forward, out if is ethernet0/7, mtu 1500 1532: (o) len=66 001c.544d.f611->001c.5418.9147/800 2.2.2.1->1.1.1.2/6 vhl=45, tos=00, id=23316, frag=16384, ttl=63, tlen=52 tcp:ports 60164->80, seq=915561528, ack=0, flag=32770/SYN 2015-10-28 13:25:58, DEBUG@FLOW: core 1 (sys up 0x17615f ms): 1532: (i) len=66 001c.5418.9147->001c.544d.f611/800 1.1.1.2->2.2.2.1/6 vhl=45, tos=00, id=0, frag=16384, ttl=127, tlen=52 tcp:ports 80->60164, seq=1662758881, ack=915561529, flag=32786/SYN/ACK rx_handle_prepare: 001c.5418.9147->001c.544d.f611, size 66, type 0x800, vid 0, port ethernet0/7 dp_prepare_if_for_pak Switchid is 16(interface ethernet0/7) port ethernet0/7 Switchid is 16(interface ethernet0/7) port ethernet0/7 ,pakiif=ethernet0/7 rx_handle_preparei_if is ethernet0/7 rx_handle_prepare calling dp_sanity ethernet0/7 Start l3 forward Packet: 1.1.1.2 -> 2.2.2.1, id: 0, ipsize 52, prot: 6(TCP): 80 -> 60164 ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 6

  25. Text (bullet point) dp_prepare_pak_lookupsrcip: 1.1.1.2, dstip: 2.2.2.1, src-port:80, dst-port:60164, prot 6 Found the session 58 session: id 58, prot 6, flag0 a,flag1 120000, created 1532, life 1800 flow0(if id: 9 flow id: 116 flag: 40000810):192.168.1.254:60164 ->1.1.1.2:80 flow1(if id: 16 flow id: 117 flag: 200810): 1.1.1.2:80 ->2.2.2.1:60164 TCP seqence handling flow0 arp resolved L3 forward, out if is ethernet0/0, mtu 1500 1532: (o) len=66 001c.544d.f60a->0024.8c01.4056/800 1.1.1.2->192.168.1.254/6 vhl=45, tos=00, id=0, frag=16384, ttl=126, tlen=52 tcp:ports 80->60164, seq=1662758881, ack=915561529, flag=32786/SYN/ACK

  26. Text (bullet point) 2015-10-28 13:25:58, DEBUG@FLOW: core 1 (sys up 0x17615f ms): 1532: (i) len=60 0024.8c01.4056->001c.544d.f60a/800 192.168.1.254->1.1.1.2/6 vhl=45, tos=00, id=23322, frag=16384, ttl=64, tlen=40 tcp:ports 60164->80, seq=915561529, ack=1662758882, flag=20496/ACK rx_handle_prepare: 0024.8c01.4056->001c.544d.f60a, size 60, type 0x800, vid 0, port ethernet0/0 dp_prepare_if_for_pak Switchid is 9(interface ethernet0/0) port ethernet0/0 Switchid is 9(interface ethernet0/0) port ethernet0/0 ,pakiif=ethernet0/0 rx_handle_preparei_if is ethernet0/0 rx_handle_prepare calling dp_sanity ethernet0/0 Start l3 forward Packet: 192.168.1.254 -> 1.1.1.2, id: 23322, ip size 40, prot: 6(TCP): 60164 -> 80 ad_vector_for_fast_flow: zonename trust, proto_flag[1] 0, proto 6 dp_prepare_pak_lookupsrcip: 192.168.1.254, dstip: 1.1.1.2, src-port:60164, dst-port:80, prot 6 Found the session 58 session: id 58, prot 6, flag0 a,flag1 120000, created 1532, life 1800

  27. Text (bullet point) flow0(if id: 9 flow id: 116 flag: 40200810):192.168.1.254:60164 ->1.1.1.2:80 flow1(if id: 16 flow id: 117 flag: 200810): 1.1.1.2:80 ->2.2.2.1:60164 TCP seqence handling flow mac_copy L3 forward, out if is ethernet0/7, mtu 1500 1532: (o) len=60 001c.544d.f611->001c.5418.9147/800 2.2.2.1->1.1.1.2/6 vhl=45, tos=00, id=23322, frag=16384, ttl=63, tlen=40 tcp:ports 60164->80, seq=915561529, ack=1662758882, flag=20496/ACK

More Related