How to Evaluate Exotic Wireless Routing Protocols?

1. How to Evaluate Exotic Wireless Routing Protocols? Dimitrios Koutsonikolas1, Y. Charlie Hu1, Konstantina Papagiannaki2 1Purdue University , 2Intel Research, Pittsburgh

2. Evolution of Wireless Routing Protocols • From the Ad Hoc Erato the Mesh Era • New design goals • High throughput vs. connectivity • New “exotic” optimization techniques • Cross – layer design AODV TORA ExOR ROMER DSDV Performance comparisons SOAR DSR MORE MC2 noCoCo ETX ETT COPE 1994 1996 1997 1998 2000 2003 2004 2005 2006 2007 Ad Hoc Networking Era Mesh Networking Era

3. In This Talk… • Review the evolution of wireless protocol design • Reveal challenges to evaluation methodology of new routing protocols • Discuss current practices • Weaknesses • Suggest guidelines for fair and meaningful evaluation

4. Ad Hoc Networking Era • Primary challenge • Deal with route breaks due to host mobility • Layering principle • Routing protocol discovers route • 802.11 unicast transmits packets to next hop • ACK/RETX, exponential backoff • Evaluation • PDR, control overhead, tradeoffs • Low constant offered load

5. Mesh Networking Era • Static routers • Mobility not a concern • Commercial applications • Compete with other internet technologies • New research focus • High Throughput

6. Towards High Throughput • Link-quality routing metrics • Examples: ETX, ETT • Still follow layering principle • “Exotic” optimization techniques • Examples: Opportunistic Routing, Network Coding • Abandon layering principle

7. Opportunistic Routing • First demonstrated in ExOR [SIGCOMM ‘05] • Packet broadcast at each hop, all neighbors can receive it • Neighbor closest to destination rebroadcasts • Coordination required A 0% 50% 50% 0% S A B C D S B D 0% 50% C

8. Intra-Flow Network Coding • First demonstrated in MORE [SIGCOMM ‘07] • Routers randomly mix packets • Benefits • Remove need for coordination • FEC-style reliability, no ACK/RETX α*p1+β*p2 p1, p2 A A p1, p2 p1, p2 Both forward Who forwards? S D S D B B p1, p2 γ*p1+δ*p2 Coordination Required! No Coordination!

9. Inter-Flow Network Coding • First demonstrated in COPE [SIGCOMM ‘06] • Routers mix packets from different flows • Increase network capacity! • Implied evaluation methodology • Subject network to congestion • Use network coding to eliminate congestion 1:p1 2:p2 1:p1 2:p2 4:p2 3:p1 3:p1+p2 3:p1+p2 Alice Router Bob Alice Router Bob Traditional Routing: 4 TX Network Coding: 3 TX

10. Implications of 802.11 Broadcast • 802.11 broadcast has no ACK/RETX, no exponential backoff • No reliability • Nodes can send faster than in unicast • Exotic techniques do not work well with TCP • Batching • Consequence • Reliability and rate control are brought to routing layer from lower or upper layers

11. Evolution of Protocol Stack Application Layer Application Layer End-to-end Rate Control Network Sublayer 3 End-to-end Rate Control End-to-end Reliability Transport Layer End-to-end Reliability Network Sublayer 2 Packet Forwarding Network Layer Packet Forwarding Network Coding Network Sublayer 1 Hop-by-hop Rate Control Hop-by-hop Reliability Hop-by-hop Reliability MAC Layer Medium Access MAC Layer Medium Access Physical Layer Physical Layer Traditional Network Stack New Network Stack

12. Implications on Protocol Evaluation • Evaluation becomes a much subtler task • Possible conflicts between new and old mechanisms • Inter-flow network coding vs. rate control • Current state • Diverse set of evaluation methodologies • Lack of clear guidelines

13. Evaluation of Unreliable Protocols

14. Practice 1: Making Both Protocols Reliable • Evaluation of ExOR, comparison with Srcr • ExOR guarantees delivery of 90% of the file • Srcr offers no guarantee • Methodology • Download a 1MB file • Send 1.1MB with ExOR to compensate for loss • Carry the whole file hop-by-hop with Srcr to avoid collisions • Problem • Removes spatial reuse from traditional routing

15. Practice 2: No Rate Control – Varying the Sending Rate • Evaluation of COPE, comparison with Srcr • COPE increases network capacity • Methodology • UDP traffic • Vary offered load • Exceed nominal capacity (6Mbps) • Problem • PDR drops quickly as network capacity is exceeded

16. Practice 3: A Protocol With Rate Control Against a Protocol Without Rate Control • Evaluation of SOAR, comparison with Shortest Path (SP) • SOAR applies rate control • SP has no rate control • Methodology • Saturate the network • Problem • Not clear what fraction of gain comes from opportunistic routing and what from rate control

17. Evaluation of Reliable Protocols

18. Practice 5: A Reliable Against an Unreliable Protocol • Evaluation of MORE, comparison with Srcr • MORE offers FEC-style e2e reliability • Srcr offers no reliability • Methodology • UDP sent at maximum possible rate • Problem • Srcr suffers losses due to congestion • Same amount of data sent by src, different amount delivered to dst

19. Practice 6: Running an Unreliable Protocol Under TCP • Evaluation of noCoCo, comparison with COPE • noCoCo applies backpressure-based congestion control/reliability • COPE has no congestion control, weak reliability • Methodology • Run COPE under TCP • Problem • TCP performs poorly in multihop wireless networks • Solution – Practice 7 • Modify COPE to use noCoCo’s congestion control/reliability

20. Use (or No Use) of Autorate Adaptation • Traditional routing uses 802.11 unicast • Exploits autorate adaptation • Exotic optimization techniques rely on 802.11 broadcast • Operates on single rate • Methodology • Evaluation of most exotic protocols disables autorate adaptation for traditional routing • For “fair”comparison • Problem • Methodology can be unfair to traditional routing

21. Recommendations for more consistent and meaningful evaluation

22. The Importance of Rate Control I Unreliable Protocols • Traditional routing under UDP has no rate control • Packets dropped beyond capacity • Throughput reduction • Exotic protocols w/o rate control • Increase throughput, may increase capacity • Packets still dropped beyond (new) capacity • Exotic protocols w/ rate control • Constant throughput beyond capacity • No need to increase offered load beyond capacity

23. The Importance of Rate Control II Reliable Protocols • FEC-style reliability provides no rate control • PDR remains 100%, rate control still needed • Exceeding capacity may lead to • Increased delays • Unfairness among flows • Related recommendation • Evaluate with multiple flows

24. Isolating the Benefit from Exotic Technique • Evaluation should quantify the gain from new exotic optimization technique • Tricky part • Adding an exotic technique may require old techniques to move to the routing layer • Recommendation • Old techniques should also be incorporated into traditional routing

25. Separating Rate Control from End-to-end Reliability • Running traditional routing under TCP + No modification to the protocol itself • TCP performs poorly in multihop wireless networks • TCP provides both rate control and reliability • If new protocol has only one mechanism, overkill to run old protocol under TCP • Recommendation • Incorporate reliability/rate control mechanism of new protocol to old protocol

26. How to Incorporate Reliability To Traditional Routing • Case 1: reliability component disjoint to exotic technique • Example: ARQ component in noCoCo • Method: add same component to traditional routing • Case 2: reliability component merged with exotic technique • Example: intra-flow NC in MORE • Method: add FEC to traditional routing?

27. MAC Autorate Adaptation • Exotic protocols should try to incorporate autorate adaptation • Not always feasible • Recommendation • Enable autorate adaptation for traditional routing • Show exotic protocol outperforms traditional routing both with and without autorate adaptation

28. Conclusions • Inconsistencies in evaluating wireless mesh routing protocols • Fundamental reason • No unified framework for understanding interactions among • MAC • Congestion • Reliability • Interference • Network coding • Real problem goes beyond how to evaluate exotic protocols

29. Thank You!