1 / 67

Three scenarios who drives admin’s crazy… and how to handle them

Three scenarios who drives admin’s crazy… and how to handle them. Ing. Thomas Mitrovits, MSc Sr. Systems Engineer. covered topics. contaminated infrastructure t wo challenges for long distance considerations ( aka frame handling ) s low drainer. c ontaminated infrastructure.

jenaya
Télécharger la présentation

Three scenarios who drives admin’s crazy… and how to handle them

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Three scenarios who drives admin’s crazy…and how to handle them Ing. Thomas Mitrovits, MScSr. Systems Engineer

  2. coveredtopics • contaminatedinfrastructure • twochallengesforlongdistanceconsiderations(akaframehandling) • slowdrainer

  3. contaminated infrastructure

  4. ClearLink Diagnostics Functional Details Test Initiator (switch port) Test Responder (device port) • D_Port test consists of following four steps: • Electrical loopback test (E-WRAP) • Optical loopback test (O-WRAP) • Link traffic test • Link latency and distance measurement

  5. Validating Configurations • Use ClearLink diagnostic port (D_Port) mode to test all 16 Gbps-capable ISLs, ICLs, and Brocade HBA connections • Complete optical, electrical and link saturation testing to ensure reliable connections • Pre-test and validate the entire SAN fabric at full line rate and with full FOS features enabled using the integrated flow generator • Emulate a 16 Gbps SAN without having any 16 Gbps hosts, targets or SAN testers H1 H2 T1 T2

  6. D-Port Test Results via CLI sw0:root> portdporttest --show 10/39 D-Port Information: =================== Slot: 10 Port: 39 Remote WWNN: 10:00:00:05:33:7e:69:c4 Remote port: 24 Mode: Manual No. of test frames: 12 Million Duration of test (HH:MM): 00:01 Test frame size: 1024 Bytes Payload Pattern: JTSPAT FEC (enabled/option/active): Yes/No/No CR (enabled/option/active): No/No/No Start time: Mon Jan 16 05:57:51 2012 End time: Mon Jan 16 05:58:56 2012 Status: FAILED ================================================================================ Test Start time Result EST(HH:MM:SS) Comments ================================================================================ Electrical loopback 05:57:52 PASSED -------- ---------- Optical loopback 05:58:06 PASSED -------- ---------- Link traffic test 05:58:13 FAILED -------- See failure report ================================================================================ Roundtrip link latency: 934 nano-seconds Estimated cable distance: 1 meters Buffers required: 1 (for 1024 byte frames at 16Gbps speed) Failure report: Errors detected (local): CRC, Bad_EOF, Enc_out Errors detected (remote): CRC, Bad_EOF Please use portstatsshow and porterrshow for more details on the above errors. D-Port test results show pass/fail as well as reason for failure to accelerate troubleshooting

  7. Long Distance (aka Buffer-Credit-Handling)

  8. Important Numbers … Numbers … Numbers 5 µs latency per km fiber km maximumdistancewith 16 Gbit FC SFPs 25 m isthemaximumdistancewith16 Gbit/s and OM4 cabling 125 mlengthis a FC frame @16 Gbit/s 250

  9. 1st challenge: the physics

  10. Fiber Optics Transmission Window Attenuation (dB/km) 1300nm=0.5dB/km 1550nm=0.2dB/km 1550 850 1300 Wavelength (nm)

  11. Available SFP+ Optical Small Form-factor Pluggable (SFP+) transceivers are available in short- and long-wavelength types:

  12. Optical cable length for Multimode fiber Optical cable length for Fibre Channel OM1 OM2 OM3 OM4

  13. SFP specifications Possible Budget Real Budget --3dBm --5dBm -24dBm --9,5dBm -20,5dBm -15dBm Power Budget = (Worst Case Launch Power) – (Worst Case Receiver Sensitivity) + (Connector Attenuation)

  14. FCIP - extension without limits ? • use of existing IP wide area network (WAN) infrastructure toconnectFibre Channel SANs. • Noimplicitdistancelimit. • The TCP connections ensure in-order delivery of FC frames and lossless transmission. • All Fibre Channel targets and initiators are unaware of the presence of the IP WAN.

  15. 2nd challenge: Flow Control

  16. Flow Control Credit exchange at Fabric Login Host says, “I can receive 40 frames.” Storage says, “I can receive 16 frames.” Switch says, “I can receive 8 frames.”

  17. Buffer Credits Credit accounting after Fabric Login Switch thinks, “OK, I can send 40 frames that way and 16 frames this way, but I have to think about it.” Host thinks, “Good, I can send 8 frames without thinking about it.” Storage thinks, “Good, I can send 8 frames without thinking about it.” Credit Count 16 Credit Count 40 Credit Count 8 Credit Count 8

  18. Buffer Credits Frame 1 1km 1km 1km 1km 1km 1km 1km 1km Frame 1

  19. Buffer Credits Frame 1 Frame 2 1km 1km 1km 1km 1km 1km 1km 1km Frame 2 Frame 1

  20. Buffer Credits Frame 1 Frame 2 Frame 3 1km 1km 1km 1km 1km 1km 1km 1km Frame 3 Frame 2 Frame 1

  21. Buffer Credits Frame 1 Frame 2 Frame 3 Frame 4 1km 1km 1km 1km 1km 1km 1km 1km Frame 4 Frame 3 Frame 2 Frame 1

  22. Buffer Credits Frame 1 Frame 2 Frame 3 Frame 4 Frame 5 1km 1km 1km 1km 1km 1km 1km 1km Frame 5 Frame 4 Frame 3 Frame 2 Frame 1

  23. Buffer Credits Frame 1 Frame 2 Frame 3 Frame 4 Frame 5 Frame 6 1km 1km 1km 1km 1km 1km 1km 1km Frame 6 Frame 5 Frame 4 Frame 3 Frame 2 Frame 1

  24. Buffer Credits Frame 1 Frame 2 Frame 3 Frame 4 Frame 5 Frame 6 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 7 Frame 6 Frame 5 Frame 4 Frame 3 Frame 2 Frame 1

  25. Buffer Credits Frame 1 Frame 2 Frame 3 Frame 4 Frame 5 Frame 6 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame 8 Frame 7 Frame 6 Frame 5 Frame 4 Frame 3 Frame 2 Frame 1

  26. Buffer Credits Frame 1 Frame 1 Frame 2 Frame 3 Frame 4 Frame 5 Frame 6 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame 9 Frame 8 Frame 7 Frame 6 Frame 5 Frame 4 Frame 3 Frame 2 Frame 9 ACK 1

  27. Buffer Credits Frame 1 Frame 1 Frame 2 Frame 2 Frame 3 Frame 4 Frame 5 Frame 6 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame A Frame 9 Frame 8 Frame 7 Frame 6 Frame 5 Frame 4 Frame 3 Frame 9 Frame A ACK 1 ACK 2

  28. Buffer Credits Frame 1 Frame 1 Frame 2 Frame 2 Frame 3 Frame 3 Frame 4 Frame 5 Frame 6 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame A Frame B Frame 9 Frame 8 Frame 7 Frame 6 Frame 5 Frame 4 Frame 9 Frame A ACK 1 ACK 2 ACK 3 Frame B

  29. Buffer Credits Frame 1 Frame 1 Frame 2 Frame 2 Frame 3 Frame 3 Frame 4 Frame 4 Frame 5 Frame 6 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame B Frame C Frame A Frame 9 Frame 8 Frame 7 Frame 6 Frame 5 Frame 9 Frame A ACK 1 ACK 2 ACK 3 ACK 4 Frame B Frame C

  30. Buffer Credits Frame 1 Frame 1 Frame 2 Frame 2 Frame 3 Frame 3 Frame 4 Frame 4 Frame 5 Frame 5 Frame 6 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame C Frame D Frame B Frame A Frame 9 Frame 8 Frame 7 Frame 6 Frame 9 Frame A ACK 1 ACK 2 ACK 3 ACK 4 ACK 5 Frame B Frame C Frame D

  31. Buffer Credits Frame 1 Frame 1 Frame 2 Frame 2 Frame 3 Frame 3 Frame 4 Frame 4 Frame 5 Frame 5 Frame 6 Frame 6 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame D Frame E Frame C Frame B Frame A Frame 9 Frame 8 Frame 7 Frame 9 Frame A ACK 1 ACK 2 ACK 3 ACK 4 ACK 5 ACK 6 Frame B Frame C Frame D Frame E

  32. Buffer Credits Frame 1 Frame 1 Frame 2 Frame 2 Frame 3 Frame 3 Frame 4 Frame 4 Frame 5 Frame 5 Frame 6 Frame 6 Frame 7 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame E Frame F Frame D Frame C Frame B Frame A Frame 9 Frame 8 Frame 9 Frame A ACK 1 ACK 2 ACK 3 ACK 4 ACK 5 ACK 6 ACK 7 Frame B Frame C Frame D Frame E Frame F

  33. Buffer Credits Frame 1 Frame 1 Frame 2 Frame 2 Frame 3 Frame 3 Frame 4 Frame 4 Frame 5 Frame 5 Frame 6 Frame 6 Frame 7 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame 8 Frame F Frame G Frame E Frame D Frame C Frame B Frame A Frame 9 Frame 9 Frame A ACK 1 ACK 2 ACK 3 ACK 4 ACK 5 ACK 6 ACK 7 ACK 8 Frame B Frame C Frame D Frame E Frame F Frame G

  34. Buffer Credits Frame 1 Frame H Frame 2 Frame 2 Frame 3 Frame 3 Frame 4 Frame 4 Frame 5 Frame 5 Frame 6 Frame 6 Frame 7 Frame 7 1km 1km 1km 1km 1km 1km 1km 1km Frame 8 Frame 8 Frame G Frame H Frame F Frame E Frame D Frame C Frame B Frame A Frame 9 Frame A ACK 2 ACK 3 ACK 4 ACK 5 ACK 6 ACK 7 ACK 8 ACK 9 Frame B Frame C Frame D Frame E Frame F Frame G Buffer Credit Frame1 can be released now !

  35. write req (BB -1) write req (BB -1) data n (BB -1) data n (BB -1) data 1 (BB -1) r_rdy (BB +1) r_rdy (BB +1) r_rdy (BB +1) r_rdy (BB +1) data 1(BB -1) status good (BB-1) status good (BB-1) x_fer rdy (BB-1) x_fer rdy (BB-1) r_rdy (BB+1) r_rdy (BB+1) r_rdy (BB+1) r_rdy (BB+1) r_rdy (BB+1) r_rdy (BB+1) sequence sequence sequence sequence FCP frame control (i.e. SCSI-FCP Write Command) Initiator Switch Target

  36. Whathappensifack_framesget lost • if BB=0 (i.e. lost r_rdy frames) the link will be reseted by sending LinkCreditReset (LR) and LinkCreditResetResponse (LRR). • Automatically recovers flow control buffer credit loss at the VC level, improving availability lost ack_frames (akaperformance) • Buffer Credit Recovery t

  37. FCP frame control cont. • Data Droop

  38. Bandwidth – distance Extension • Remove Data droop – adding Buffer-to-Buffer Credits

  39. FCP frame control cont.

  40. Bandwidth – distance Extension cont. • Remove Data Droop => Terminating Buffer-to-Buffer Credits

  41. How “long” is a frame? Traveling at the speed of light = 300.000 km/s in vacuum (approx. 65% in fiber) a frame can be very short… @ 1G a frame is about 4Km in length @ 2G a frame is about 2Km in length @ 4G a frame is about 1Km in length @ 8G a frame is about 0.5Km in length @ 16G a frame is about 250 m in length

  42. How much credit do I need? Good “Rule of thumb” Number of credits needed = 1 + Link speed in Gb/s * Distance in Km Frame Size in K Example: 20 Km at 1 Gb/s 1 + 1 * 20 = 11 2 Example: 10 Km at 4 Gb/s 1 + 4 * 10 = 21 2

  43. Performance Optimization on FC Long Distance ISLs Optimize Performance • Allow end users to specify either the number of buffers or average frame size while configuring a long distance port • Provides more control to users to optimize performance on long distance links based on traffic pattern • Two new options for Portcfglongdistance CLI - one option to configure buffers, and another option to configure frame size for LD and LS modes •  In pre-FOS v7.1, user can configure only the “distance” for long-distance static and dynamic mode. Buffer estimation done based on distance, link speed and full_size frame buffers assumed, which can lead to suboptimal buffer allocation • With FOS v7.1, a user can directly configure the buffers required for a port of a long distance link • Users can also configure/specify the average frame size for a long distance port. Using the frame size option, number of buffers/credits required for a port will be automatically calculated

  44. Performance Optimization on FC Long Distance ISLs Optimize Performance • Enhancement to display the average buffer usage and average frame size in portbuffershow • Average buffer usage is the real time buffers used by the port while the traffic is in progress • Provides better insights into the traffic pattern and also lets users optimize performance on long distance links by specifying the average frame size • A new CLI portBufferCalc to calculate the number of buffers required per port given the distance, speed and frame size • If a user does not provide any of the options, then current port’s configuration will be considered to calculate the number of buffers required • This CLI will give the users an estimate on the number of buffers required for given distance, speed and frame size

  45. Portcfglongdistance Example • -distance & -framesize pluto_134:FID128:root> portcfglongdistance 1/3 LS 1 -distance 100 -framesize 1024 Reserved Buffers = 806 Warning: port (3) may be reserving more credits depending on port speed. pluto_134:FID128:root> portcfgshow 1/3 Speed Level: AUTO(HW) Fill Word(On Active) 1(Arbff-Arbff) Fill Word(Current) 1(Arbff-Arbff) AL_PA Offset 13: OFF Trunk Port ON Long Distance LS VC Link Init ON Desired Distance 100 Km Frame Size 1024 Bytes Reserved Buffers 806

  46. Portcfglongdistance Example • -buffers pluto_134:FID128:root> portcfglongdistance 1/3 LS 1 -buffers 400 Reserved Buffers = 406 Warning: port (3) may be reserving more credits depending on port speed. pluto_134:FID128:root> portcfgshow 1/3 Area Number: 3 Speed Level: AUTO(HW) Fill Word(On Active) 1(Arbff-Arbff) Fill Word(Current) 1(Arbff-Arbff) AL_PA Offset 13: OFF Trunk Port ON Long Distance LS VC Link Init ON Desired Buffers 400 Reserved Buffers 406

  47. Portbuffercalc – New CLI • This CLI is used as an assistance to configure the recommended buffers for a longdistance port. • It returns the buffers based on the distance/speed/framesize configured. • CLI • portBufferCalc [SlotNumber/]PortNumber <-distance distance> <-speed speed> <-framesize framesize> • Example pluto_134:FID128:root> portbuffercalc 1/3 -distance 100 406 buffers required for 100km at 8G and framesize of 2048bytes

  48. Buffer Credits

  49. Bottleneck Detection

  50. Server 1 Storage A FC Switch 2 FC Switch 1

More Related