Topics of Discussion

1. Troubleshooting Voice and Video Performance over IP using Measurement Tools…Prasad Calyam and Paul Schopis,ITEC-Ohio, OARnet and Ohio Supercomputer Center, Tools Tutorial,Joint Techs, Salt Lake City, Utah, February 2005

2. Topics of Discussion • Basics of Voice and Video over IP performance measurement • Measurement Tools we have experimented with… • “H.323 Beacon” Tool we have developed! • How is this tool different? • Tool Features • Two Case-studies • An Useful H.323 Beacon Tool Supplement!!! • Ethereal RTP Analysis Module • Conclusion

3. Basics of Voice and Video over IP performance measurement • Voice and Video Traffic are inherently different from Data Traffic • They use Signaling protocols such as H.323, SIP, … • Media (Voice and Video payload) is delivered over IP using RTP packets irrespective of which codec or end-point technology (PC-based Vs Appliance-based) • General ICMP and UDP based tools fail to totally capture the performance bottlenecks faced by actual voice and video traffic at the end-host and in the network • i.e.- ping, traceroute, Iperf, pathrate, … don’t suffice! • We need to measure end-user experience of Voice and Video over IP applications also!

4. Measuring End-user experience… • Two approaches to evaluating end-user experience of audiovisual quality • Subjective Measurements • Involve human participants to rate audiovisual quality Can you hear me now? • Mean Opinion Score (MOS) Ranking technique (ITU-T P.800) Not just “Good”! • Objective Measurements • Automated techniques to rate audiovisual quality • “E-Model” [ITU-T G.107] • Perceptual Evaluation of Speech Quality (PESQ) [ITU-T P.862]

5. Mean Opinion Score (MOS) Concept

6. What are the end-host and network problems? Scenario I: A Researcher and an Industry professional want to Videoconference

7. Case1:Researcher is unable to make a call!

8. There was a mis-configured firewall blocking necessary ports…

9. Case2: Industry professional is unable to make a call!

10. His LAN’s Internet connectivity was non-functional at that time…

11. Case3: They connected, but of them experienced bad audio & video!

12. There was congestion at one of the intermediate routers along the path…

13. There was congestion at one of the intermediate routers along the path…

14. There was congestion at one of the intermediate routers along the path…

15. The performance problem can be anywhere in the E2E Path!!!

16. What are the other common end-to-end performance problems? • Common problems involving endpoint devices • Failure of audio and video hardware, out dated or buggy end-point application software • Faulty connections and configurations of audio and video interfaces • Lack of lip-synchronization • Mis-configured jitter buffer sizes • Non inter-operable end-point application software • Lack of forward error correction and echo cancellation mechanisms in end-point clients • Lack of end-user training

17. Common End-to-End performance problems in VVoIP Systems (2) • Common problems involving network devices • Insufficient network capacity for handling multiple high data rate videoconferencing calls • Excessive delay, loss, jitter, out of order packets and re-ordered packets in the network • Duplex mis-match problems • Traffic congestion at peak-usage periods of the network • Mis-configured priorities for real-time audio and video traffic streams in the network • Asymmetric routing with excessive delays on one path • Lack of network engineering resource personnel

18. Common End-to-End performance problems in VVoIP Systems (3) • Common problems involving application service devices • Misconfigured firewalls that block required ports • Non-H.323 friendly NATs • Misconfigured devices • MCUs • Gatekeeper • Gateway devices • Outdated or buggy application-service device application software

19. Measurement Tools we have experimented with… • MCU-specific Performance Monitors • Per-call monitoring of many network and end-point parameters (delay, jitter, loss, reordering) • Telchemy VQMon • Based on E-Model Standard; Integrated into many vendor VoIP troubleshooting products (e.g., Brix Networks, Agilent, etc…) • PESQ • Based on PESQ Standard; Used in many products (e.g., Opticom, Acterna, etc…) • Hammer Call Analyzer • Packet Analyzer (Ethereal backend) with many features such as call-flow display, jitter buffer analysis, wave file analysis, etc… • Nettach • Developed originally by PictureTel; Similar to H.323 Beacon where you emulate Voice and Video traffic and measure performance • Many other good tools… (AppareNet, ViDeNet Scout (based on NetIQ Chariot), etc.)

20. MCU-specific Performance Monitors Increased Video Sync Loss!

21. Telchemy VQMon

22. PESQ SAMPLE RESULTS: REFERENCE DEGRADED PESQMOS PESQMOS original.wav degraded.wav SQValue=2.930 2.930

23. Hammer Call Analyzer Taken from- http://www.empirix.com

24. Nettach

25. H.323 Beacon Overview • An application-specific measurement tool • To monitor and qualify the performance of H.323 Videoconferencing sessions at the host and in the network (end-to-end) • Useful to an end-user/conference operator/network engineer • Uses OpenH323 and J323Engine libraries

26. Comparison with other existing tools… • Addresses problems due to H.323 protocol-specific idiosyncrasies • Can be generalized to RTP packets performance over the network • Many in-built tools that generate various kinds of measurement data for pre/during/post Videoconference troubleshooting! • An “easy to install and use” tool that is open source (FREE!) • Commercial tools are VERY expensive! • NetIQ Chariot, Spirent Smartbits, Telchemy VQMon,…

27. Initial call setup failures and haphazard disconnections… • Test Status • In-Session, Normal Close, Exception Close • Exception Close Alarms • “Possible Firewall/NAT presence obstruction” • “No Internet connectivity” • “Incompatible codec being used” • “Insufficient bandwidth” • “Remote H.323 Beacon Server not online”

28. Initial call setup failures and haphazard disconnections…

29. Network Health Status… • Delay, Jitter and Loss data • Real-time, offline raw data and test session summary

30. Network Health Plots… • Watermarks for “Good”, “Acceptable” and “Poor” grade of quality as experienced by end-user • Delay: (0-150)ms, (150-300)ms, > 300ms • Jitter: (0-20)ms, (20-50)ms, > 50ms • Loss: (0-0.5)%, (0.5-1.5)%, >1.5% • Levels obtained from our PAM 2004 paper Poor Acceptable Good

31. Audio and Video Quality Assessments • Audio and video loopback feature • E-Model-based objective MOS ranking • Slider-based subjective MOS ranking

32. Customization of tests… • Test results data folder, TCP/UDP/RTP port settings, H.225 and H.245 parameters, preferred codec, watermarks for delay, jitter, loss, …

33. Use-case I • Problem report • Intermittent frame freezing • Lot of pixilation • No significant audio problems • Sudden disconnections Effect of a mis-configured firewall on a video stream

34. Use-case I (2) • MCU Software • ~15 fps for 384Kbps call and ~0.6% packet loss • ~30 fps for 128Kbps call and ~0% packet loss • H.323 Beacon • Sluggish call-setup • Delayed packet-events • Initial jitter variations in poor range Jitter variations indicated by H.323 Beacon client-to-server test Increasing the rate-limit and firewall re-configuration solved the problem!

35. Use-case II • Problem report • Poor audio and video reception at the site of user • Problem only in the streams from Campus A to Campus B • MCU Software • Packet loss (~1%) • Packet re-ordering • H.323 Beacon • Jitter values consistently in poor range • Traceroute and Reverse-Traceroute showed an asymmetric route • E-Model results • Campus A to Campus B: 3.78 • Campus B to Campus A: 4.37

36. Use-case II (2) Fig. 1 Time-plot of Equipment Impairment Factor Fig. 2 Time-plot of Delay Fig. 3 Time-plot of Loss Fig. 4 Time-plot of Mean Opinion Score Ranking Problem was solved by contacting the neighboring ISP who was suspected to be experiencing problems!

37. How do you setup the H.323 Beacon in a network? Scenario I: H.323 Beacon used in a Network Measurement Infrastructure…

38. Scenario II: In a Multi-point setting…

39. Scenario II: In a Multi-point setting…

40. Scenario II: In a Multi-point setting…

41. Ethereal RTP Analysis Module • An useful H.323 Beacon Tool supplement!!! • Adds advantages of passive measurement to the H.323 Beacon • In general, to any RTP-based application! • Generates inter-packet jitter, packet-size distribution, packet loss and re-ordering information from input traffic traces • We have written simple parsing scripts that can process data in a format that facilitates analysis using Minitab or Excel • Use the OPENXTRA version of Ethereal! http://resource.intel.com/telecom/support/appnotes/9008/9008an.pdf

42. Ethereal RTP Analysis Module (2) • Steps for analyzing the Traces • Load the packet trace into Ethereal • Trace will contain both forward and reverse direction streams (Check “Source” and “Destination” IP addresses) • Decode streams as RTP (default is UDP) • This will mark all related packets as belonging to a specific audio and video codec streams • Analyze individual audio or video streams • Import various information fields as .csv file (“Save as CSV” option) • Also has wave file generation relating to an audio stream (“Save Payload” option) • Works only for G.711 Codec streams! • Good for PESQ where you want to compare original and degraded wave files to obtain Objective MOS information

43. Ethereal RTP Analysis Module (3) General UDP Stream decoded as an H.263 payload stream

44. Ethereal RTP Analysis Module (4) Audio Stream Video Stream

45. Ethereal RTP Analysis Module (5) Loss Re-ordering; 4541345415 45414(Observe Sequence #s; could also be 2 consecutive packet losses) Pink-marked packets relate to either lost or re-ordered packets!

46. Ethereal RTP Analysis Module (6) An Imported CSV File!

47. Ethereal RTP Analysis Module (6) Create interesting visualizations to understand various RTP packet characteristics; can do the same for both Voice and Video packets!!!

48. Questions? H.323 Beacon Reference: http://www.itecohio.org/beacon