Wireless Mesh Networks

1. Wireless Mesh Networks Anatolij Zubow Link-level Measurements from 802.11 Mesh Networks

2. MIT Roofnet Case Study • Link-level Measurements from an 802.11b Mesh Network: • Paper analyzes the causes of packet loss in a 38-node urban multi-hop 802.11b network. • Distribution of inter-node loss rates is relatively uniform over the whole range of loss rates; there is no clear threshold separating “in range” and “out of range.” • Most links have relatively stable loss rates from one second to the next (non-bursty losses). • SNR and distance have little predictive value for loss rate. • The large number of links with intermediate loss rates is probably due to multi-path fading.

3. MIT Roofnet Case Study • Is a multihop wireless network. • Roofnet nodes are computers with 802.11b cards in apartments spread over six square kilometers of Cambridge. • Each node has a roof-mounted omni-directional antenna. • The network’s main purpose is to provide Internet access via a few wired gateways. • Three- and four-story houses; most antennas are mounted about two or three feet above the chimneys of these houses.

4. Link Abstraction • Many routing and link-layer protocols assume the validity of a “neighbor” abstraction that partitions all the pairs of nodes into pairs that can communicate directly, and pairs that cannot. • E.g. 802.11 assumes that a pair of nodes will either hear each other’s control packets (e.g. RTS/CTS), or will not interfere. • Neighbor abstraction is supported by typical assumptions about the (theoretical) relationship between SNR and BER (rapid transition from zero to high BER). • E.g. Intersil Prism HFA3873 baseband processor is about 3 dB. • Signal strength falls off rapidly with distance  we expect relatively few node pairs to lie in the transition zone  we expect almost all pairs of nodes to either be able to talk to each other with low loss, or not at all.

5. Conclusions for MAC/Routing Protocols • Links with intermediate loss rates are common, with no sharp transition between high and low packet loss rates. • Inter-node distance is not strongly correlated with whether nodes can communicate. • Most links have non-bursty loss patterns. • Links with very high signal strengths are likely to have low loss rates, but in general signal strength has little predictive value. • A link is likely to have a significant loss rate at its optimum 802.11b bit-rate. • Explanation: Multi-path fading greatly affects outdoor links.

6. Experimental Methodology • 38 nodes distributed over roughly six square kilometers of Cambridge. • All nodes use identical 802.11b cards (Prism 2.5 chip-set). • Cards transmit at 2.422 GHz (802.11b channel 3) with the tx power level set to +23 dBm (200 mW). • Omni-directional antennas. • Automatic bit-rate selection was disabled. • Roofnet routing was turned off (no user traffic). • All the experiments were executed in the early hours of the morning (maybe the effects of non-Roofnet radio activity is underestimated ?!?!).

7. Experimental Methodology (2) • Single Experiment • Each node in turn sends 1500-byte 802.11 broadcast packets as fast as it can, while the rest of the nodes passively listen. • Each sender sends for 90 seconds at each of the 802.11b bit-rates. • 802.11 broadcast packets (no Acks, no retries) • Each packet includes a unique sequence number. • The sender records the time at which it sends each packet, and all the other nodes record each received packet’s sequence number, arrival time, and the “RSSI” and “silence” values. • Signal Strength Measurements • Prism 2.5 chip-set provides per-frame measurements called RSSI (receive signal strength indication) and “silence value”. • RSSI reflects the total power observed by the radio hardware while receiving the frame, including signal, interference, and background noise. • The silence value reflects the total power observed just before the start of the frame. • Accuracy of the RSSI and silence readings was within 4 dB. • SNR values derived from the RSSI and silence values.

8. Distribution of Delivery Probabilities • Distribution of inter-node packet delivery probabilities at different bit-rates. • Data for each bit-rate is sorted separately! • @1, 2, and 5.5 Mbit/s - Distribution of loss rates is fairly uniform. • @11 Mbit/s - there is a more rapid fall-off in delivery probability, but there are still many links with intermediate probabilities.

9. Distribution of Delivery Probabilities (2) • Implications: • Neighbor abstraction does not apply well to Roofnet: most node pairs that can communicate have intermediate loss rates. • It would be difficult to find multi-hop routes through Roofnet that did not involve one or more hops with significant loss rates. • A routing protocol cannot ignore this problem by simply ignoring all but the very best links: • E.g. a one-hop route with 40% loss rate has better throughput than a two-hop route with loss-free links.

10. Spatial Distribution of Loss Rates • A potential explanation for the distribution of link delivery probabilities is the attenuation due to distance. • Figure (below) shows samples of how delivery probability varies with location. • Each map corresponds to a different sender; the size of each node’s disk indicates the fraction of packets that node received from the sender. These maps show the delivery probabilities from three senders to all other nodes. The sender is marked S, and each receiver is indicated by a circle with radius proportional to the fraction of packets it received. There is a correlation to distance but it is not always consistent.

11. Spatial Distribution of Loss Rates (2) • Since the three senders are close to each other, one might expect the three reception patterns to be similar. • This is true to the extent that very close nodes have high delivery probabilities for all three senders. • Other than that the three reception patterns are quite different. • The differences are likely caused by obstacles in the environment, different antenna heights, and multi-path fading.

12. Spatial Distribution of Loss Rates (3) • Both bit-rates (1 and 11) exhibit a cluster of short links with high delivery probabilities, a few remarkably long links, and a significant set of links with no discernible relationship between distance and delivery probability. Figure shows the relationship between distance and delivery probability for all node pairs, for 1 (left) and 11 Mbit/s (right).

13. Time Variation of Loss Rate • The significance of intermediate loss rates depends on the time scale at which loss and delivery alternate. • One way in which a link might exhibit a 50% loss rate would be to deliver or drop each packet in alternation. • At another extreme, a 50% link might alternate 10-second periods of total loss and total delivery. • Different route selection and error correction strategies are appropriate in the two different situations. • Most links are not bursty!! Delivery probability over time (in seconds) for four 1 Mbit/s links, a link with about 50% average loss rate. The send rate is about 80 1500-byte packets/second. Each point is an average over 200 milliseconds. The top graph shows one of Roofnet’s most bursty links, the bottom one of the least bursty.

14. Effect of SNR • One reason that many links have intermediate loss rates might be that many links have marginal SNRs. • Prism 2.5 spec. - range of SNR values for which the packet error rate would be between 10% and 90% is only 3 dB wide, assuming additive white Gaussian noise (AWGN). • This experiment confirms the Prism spec. - most SNR values result in either very high or very low loss rates; the intermediate range is only a few dB wide. Delivery probability versus SNR, measured using the emulator and two Prism 802.11b cards. Architecture of the hardware channel emulator.

15. Effect of SNR (2) • In order for marginal SNR with AWGN to explain why so many Roofnet links have intermediate loss rates, the majority of Roofnet links would have to have SNR ratios in a narrow 3 dB range. • This is not the case. The range of SNR values is much greater than 3 dB, even though most Roofnet links have intermediate loss rates. • While high SNR values correspond to high delivery probabilities, the range of SNR values for intermediate loss rates is much wider than 3 dB. Delivery probability at 1,2 ,5.5, and 11 Mbit/s versus the average S/N. Each data point represents an individual sender-receiver pair.

16. Effect of SNR (3) • Figure shows per-receiver versions of the 1 Mbit/s plot from previous Figure. • These plots show a better correlationbetween SNR and delivery probability. • The range of SNR values corresponding to intermediate loss rates is larger than 3 dB for three of the four receivers, suggesting that SNR is not the only factor determining delivery probability. • SNR does affect delivery probability, but one cannot expect to use SNR as a predictive tool. A scatter plot of average S/N vs average delivery probability at 1 Mbit/s. Each graph corresponds to a different receiver, each point shape corresponds to a different sender, and there is one point per one-second interval per sender.

17. Effect of TX Power • The data come from an experiment in which each sender transmitted at three different power levels. • The three curves show the delivery probabilities between the node pairs at 10, 40, and 200 mW. The effect of varying the transmit power level on the delivery probability, for 1 Mbit/s. For example, raising the power level from 10 to 40 mW almost doubles the number of nodes that have delivery probabilities of 40% or more.

18. Effect of Bit-Rate • The 802.11b transmit bit-rates differ in robustness. • E.g., there are about three times as many links at 1 Mbit/s as at 11. • Figure shows, for each pair of nodes, the throughput in 1500-byte packets/second at the different bit-rates (pairs are sorted by the throughput at 11 Mbit/s).

19. Effect of Bit-Rate (2) • Implications for 802.11b bitrate selection algorithms: • Wait until a high bit-rate is performing very badly (i.e. delivering only half the packets) before it reduces the bit-rate. • 11 Mbit/s often provides higher throughput than 5.5 Mbit/s even when the loss rate at 11 Mbit/s is higher than 50%. • Performance at a low bit-rate is not a good predictor of performance at higher rates: e.g., there are many links with high loss rates at 1 Mbit/s that would have a higher throughput at 11 Mbit/s. • Bit-rate selection must be based on explicit measurements of throughput at the different rates, rather than on indirect prediction.

20. Interference from 802.11 Sources • Another possible reason for the links with intermediate delivery probabilities could be interference from other 802.11 activity. • Packets could be lost due to interference from other 802.11 senders on the same channel, or from overlapping channels. • Roofnet itself produces no packets, so all packets in the table are from non-Roofnet sources. • Q.: Are the numbers of packets received from non-Roofnet sources consistent with the quantity of losses observed by Roofnetreceivers? Data and beacon packets per second received on each channel, averaged over all Roofnet nodes. These numbers include all frames recognized by the 802.11 hardware in “monitor” mode, including non-Roofnet traffic and damaged frames.

21. Interference from 802.11 Sources (2) • While the numbers of foreign packets are of the same order of magnitude as the numbers of lost packets, there does not seem to be any correlation between foreign packets received by each receiver and Roofnet packets lost by each receiver. • It does not seem likely that foreign 802.11 packets on channel 3 are causing Roofnet losses. Each point indicates one host pair. The x-axis shows the number of foreign packets received per second by the receiver in that pair (same data as Table 1). The y-axis shows the number of 1500- byte packets lost per second at 1 Mbit/s. The data for the two axes are from experiments performed within a few minutes of each other. There is no obvious correlation.

22. Effect of Multi-path • A receiver may hear not just the signal that travels directly from the sender, but also copies of the signal that reflect from objects such as buildings. • The reflected signals follow longer paths than the direct signal, so the receiver sees the combination of multiple copies of the signal at different time offsets. • The Intersil HFA3873 baseband processor in the Prism 2.5 chip-set has a RAKE receiver and equalizer capable of suppressing reflected copies with delays of up to 250 ns. • Studies of outdoor urban radio propagation find that delay spreads often exceed one microsecond. • Theoretical models demonstrate that high delay spreads significantly increase packet loss rates. • Roofnet was unable to characterize the reflective paths present in mesh (idea?!?!). • Evaluation of the impact of longer delay spreads on packet loss with the channel emulator.

23. Effect of Multi-path (2) • The emulator uses a two-ray channel model, in which a delayed copy of the transmitted radio signal is attenuated and mixed with the original before arriving at the receiving radio. • This emulates the original signal following a line-of-sight path and a single reflective signal which followed a longer path. • The parameters of the model are the delay between the two signals and their relative strengths. • Note: a real physical environment would produce many reflective rays.

24. Effect of Multi-path (3) • Sender transmitted batches of 200 broadcast packets at 1, 2, 5.5 and 11 Mbit/s. • Measurements were taken while varying both the delay and attenuation of the reflected ray in increments of 0.02 microseconds and 0.2dB respectively. • The original ray was not attenuated. At a delay of 1 microsecond, at least 10% of the packets are lost when the reflected ray is attenuated by 9.2dB or less, and at least 90% of the packets are lost when the reflected ray is attenuated by 7dB or less. The x-axis indicates how long the reflected ray is delayed relative to the direct ray. The y-axis indicates how much the reflected ray is attenuated relative to the direct ray. The tops of the gray and black bars at each delay indicate the attenuation levels which result in 10% and 90% packet loss respectively, so the height of the gray bar indicates the region of intermediate packet loss. Packet loss is common for delay spreads greater than a few hundred nanoseconds and occurs more often when the reflected ray is delayed by a multiple of the symbol time.

25. Effect of Multi-path (4)

26. Effect of Multi-path (5) • A delay spread of less than a few hundred nanoseconds has little effect on packet loss, regardless of the relative strength of the reflected ray (consistent with Intersil’sspec. for the RAKE receiver). • Packet loss rates increase for delays above a few hundred nanoseconds because the RAKE receiver has trouble distinguishing the original signals from the reflections. • Significant losses due to multi-path are only likely to occur if the inter-node distances in Roofnet are long enough that a reflected signal could be delayed on the order of a microsecond. • This delay corresponds to approximately 300 meters, so Roofnet would have to have direct paths significantly longer than that.

27. Effect of Multi-path (6) • CDF of the distances between all pairs of Roofnet nodes that have nonzero delivery probabilities at 1 Mbit/s. • The median is 500 m, and about a quarter of the links are longer than 1000 m. • Links of this length seem compatible with delay spreads of at least a few hundred nanoseconds. • The emulator experiment shows that multi-path interference could cause loss rates in a way that would be hard to predict from SNR alone. • Loss rate could vary widely depending on the exact length of the reflective path; if reflective path lengths are more or less uniformly distributed, then one might expect the loss rate caused by multi-path to be roughly uniformly distributed as well. A CDF of the distance between all pairs of Roofnet nodes that have non-zero delivery probabilities for 1500-byte packets at 1 Mbit/s.

28. FRACTEL Case Study • Current understanding is that wireless link performance is inherently unpredictable, due to multipath delay spread. • Focus here: mesh networks in rural locations. • Results from measurements show that 802.11b PHY layer is indeed stable and predictable. • Strong correlation between PER and SNR. • Interference, and not multipath fading, is the primary cause of unpredictable performance. • Link abstraction is valid in outdoor rural mesh.

29. Introduction • Predictable performance of community mesh networks is critical for real-time applications. • For rural areas in developing regions, video-conferencing based applications is the primary use of the network. • Problem: Multi-hop mesh networks are not really known for predictable performance. • Results from MIT Roofnet study: • Unpredictability in performance comes right from the PHY layer • Wireless links exhibit widely varying and unpredictable error rates due to large multipath-induced delay spreads in outdoor environments. • Little correlation between the received SNR and the observed packet error rate (PER). • Link abstraction is NOT valid.

30. Background • FRACTEL (wiFi-based Regional/Rural data Access and TELephony) project, our goal is to build WiFi mesh networks to support data and multimedia applications, in rural settings. • Most links are expected to be short (up to 500 m, similar to Roofnet) • Deployment target: • Few buildings, two or three storeys tall in rural environments or sometimes even in semi-urban residential communities. • Expected multipath behaviour is similar to Roofnet and DGP (WiFi long distance).

31. Background (2) • Comparison of the main measurement results from Roofnet [6] and DGP [7]:

32. Experimental Methodology • Environment: • Rural village & a residential university campus. • One location within the village and five locations within the campus. • For each location, the transmitter position was fixed and the receiver position was varied over six different choices. • Total number of 36 links. • Link lengths were in the range 150-400 m (Roofnet study had a significant number of links over 500 m). • Campus environment helps studying interference versus multipath as being the cause for packet error rates.

33. Experimental Methodology (2) • Hardware: • The same was used in the Roofnet study (Senao 2511CD plus ext2 PCMCIA) • Software: • Receiver side per packet information: RSSI, noise (silence) level, rate (1, 2, 5.5, or 11 Mbps), MAC header details, and whether or not the received packet passed the CRC check. • The receiver was set in monitor mode for all of the experiments. • The transmitter sent packets at a fixed transmit power and transmit rate, in adhoc, pseudo-ibss mode, with an inter-packet gap of 20 ms*. • In each experiment, the transmitter sent 6000 packets. *Inter-packet interval = 20 ms & per-packet transmit duration 12 ms at most imply that we will most likely capture some foreign packets (if any) in the gap ≥ 8ms between two successive packets.

34. Experimental Methodology (3) • Choice of transmitter/receiver positions: • The transmitter was placed at an elevation, atop a building terrace in all cases except Gnd. At Gnd, the transmitter mast was placed at a clearing, on a 1.5 m tall tripod. • For the receiver position • Good position to be one where there was clear line-of-sight, and the mean RSSI was about −70dBm. • Medium receiver position to be one which had some foliage or building in-between, and the RSSI thus calculated was about −75dBm. • Bad receiver position is one where the RSSI was about −80dBm or so, and there was no clear LOS. • At each location, we chose a combination of good, medium, and bad positions.

35. Error Rate Analysis • Primary characteristic for wireless links is the error rate (function of the RSSI). • The correlation between the two to a large degree determines whether or not the link abstraction holds. • For many transmitter-receiver position pairs, there was some variation (up to 4 dB in most cases) in the RSSI, across the 6000 packets. • The 6000 transmitted packets were separated into 60 bins of 100 packets each. • For each of the 60 bins, the average RSSI and the error rate is computed (for empty bins the RSSI was -100dBm). • Noise, or silence level in most of the experiments was −94dBm to −95dBm (error rate against the RSSI is equivalent to error rate against the SNR).

36. Error Rate Analysis (2) • Several points in the error rate versus RSSI graph showed a high error rate despite a high signal strength (−75dBm or above). • Looked into the receiver side logs for these cases reveals that there were lots of packets from external WiFi sources. • Two cases • Interference-prone cases (500 to over 90,000 foreign packets over the 2 min duration) • Interference-free cases Each point in each plot represents a 100 packet bin from a particular transmitter receiver position pair.

37. Error Rate Analysis (3) • At the interference-free positions, there is a very strong correlation between the signal strength and the observed error rate. • There is a threshold signal strength above which the error rate is more or less negligible: • For the 1 Mbps & 11 Mbps rates, this threshold is about −86dBm & −79dBm respectively. • Controlled experiment indoors when the transmitter and receiver cards were connected by an RF cable shows similar results: • Noise floor was between −95dBm and −94dBm, • Threshold RSSI values above which the error rate fell below 1% was −88dBm, and −81dBm • At the interference-prone positions, correlation between the error rate and the RSSI breaks down, and there are several cases of high error rate even in the presence of high signal strength: • Clusters of points for the interference-prone locations. • The different clusters correspond to different locations, and the clustering is due to the different levels of interference at these locations.

38. Error Rate Analysis (4) • For one of the positions, it was possible to see the dependence on external interference: • We initially ran our 2 min experiments and observed interference at this position, and high error rates. • We then identified the interference to be from a known WiFi source. • We then had the interference source temporarily shut down. • When we then re-ran the experiment, the error rates came down to close to zero.

39. Roofnet Data: A Fresh Analysis • Roofnet suggests that, since external interference does not seem to be a factor, multipath induced delay spread in excess of 1μs is the cause of high loss rates at high SNR. • This is contrary to the FRACTEL conclusion. • A strange pattern we observed was that the noise floor measurements in this data were not only much higher than ours in many cases, but also showed significant variation across the various packets. • Noise, or silence level as reported by the card, for a particular received packet, is the energy level as measured before the packet reception began. • The noise level should remain more or less constant for the various packets as with observed in the FRACTEL measurements (noise level of −94dBm or −95dBm whether or not there was multipath)

40. Roofnet Data: A Fresh Analysis (2) • Roofnet data shows very different behaviour: • We choose the subset of the transmitter-receiver pairs where the average RSSI was over −77dBm at the 11 Mbps data rate (well above the measured card sensitivity of −81dBm), and those which showed a loss rate between 20% and 80%. • That is, where we have high signal strength but still high error rate. For all points the average SNR too was very high (11 to 39 dB).

41. Roofnet Data: A Fresh Analysis (3) • 11 Mbps • The median noise levels are as high as −86dBm, with most median values at −90dBm or above. • The 95%-ile minus 5%-ile (band where 90% of the values lie) can be as high as 16 dB! • In contrast, the noise levels in the FRACTEL data was at most −94dBm, with a variation of only about 1 dB. • 1 Mbps • Shows similar behaviour.

42. Roofnet Data: A Fresh Analysis (4) • Fig. shows the variation of the RSSI and noise levels, as a function of the packet number, for the data point no: 13 (from prev. Fig.). • CDF: 50% of the packets show a noise level of about −90dBm or more. • Explanation for the high and variable noise levels is the presence of several 2.4 GHz sources (likely WiFi).

43. Understanding Interference • Controlled experiment: • Two transmitters A and B and one receiver R. • R’s card has two connectors for the external antenna (for diversity). • We make use of these two connectors to connect A to R and B to R respectively. • In this arrangement, A and B cannot hear one another, but R can hear both (i.e. a case of hidden nodes). • 11 Mbps rate, with an interpacket interval of 2 ms. • A and B act as interference to one another. • We fixed the attenuators such that the mean RSSI of B’s packets at R was about −75dBm. • The attenuator at A was varied across four experiments.

44. Understanding Interference (2) • Controlled experiment: • From R’s log various statistics were calculated. • As A’s RSSI increases, the loss rate of A’s packets decreases and that of B’s increases (Col-3/4).

45. Understanding Interference (3) • P1: The noise as well as the noise band are quite high in almost all of the cases (Col-5, Col-8). These are similar to the noise levels observed from Roofnet data. • So, interference does cause the noise level to be high and variable. • P2: Focusing on experiment no: 2 of Table, we see that so far as B’s packets are concerned, there is a loss of 18.3% (Col-4). But then, the number of A’s packets received are very few. With a loss rate of 99.2%, and a sending rate of 500 pkts/sec, A’s packets are received at an average rate of only 4 pkts/sec. But this is sufficient to cause an 18.3% loss rate, which amounts to 91.5 lost pkts/sec! • So, the packet loss rate can be high even when the number of observed interference packets is low.

46. Understanding Interference (4) • P3: It is easy to see why the packet loss rate can be low even when the number of interference packets seen is high. This can happen when the two transmitters can hear one another. So the interference simply backs off when transmission on the link of our interest starts. • P3 and P2 taken together: • Indicate how we may have no correlation between the foreign packet rate and the observed error rate, and still have all of the packet errors caused due to interference. • This, taken along with the high and variable noise levels in Roofnet, leads us to conclude that external interference did play a major role in causing their error rates. • Multipath delay spread seems not to be the main cause of packet errors. Cmp with Roofnet scatter plot.

47. Understanding Interference (5) • P5: Can one estimate the link performance based on the average measured noise floor for packets? • We see that for a given average noise floor, there is a wide range of error rates possible! • This means that we cannot really estimate the expected interference, or the resultant link behaviour, based on the average noise floor either.

48. Link Stability over Time • Q.: Is it possible to build links with predictable performance in interference free environments? • We need to investigate the stability at (a) small time scales (important for routing), as well as at (b) large time scales (important for link provisioning). • Short term variation in RSSI (interference free receiver positions, 2min) Variation in the RSSI (small time scale)

49. Link Stability over Time (2) • Small scale variations: • For most of the experiments, the band (90% of all values) is within 3-4 dB. • In three cases the band is 6-7 dB (NLOS, e.g. moving obstacles). • Three cases (pair numbers 42, 43, 44) where the band was 18 dB, 23 dB, and 24 dB respectively (hardware bug). • RSSI variation is within about 3-4 dB is most cases (<4 dB in LOS environments). • Interference-prone positions : Band was within 5 dB for most links. There were a few links in the Roofnet data which showed larger bands (6 to 11 dB).

50. FRACTEL Results • Link error rate is strongly correlated with RSSI (received signal strength indicator) or the SNR. • There is an identifiable RSSI threshold beyond which the error rate is close to 0%. • Strong evidence that external interference, and not multipath induced delay spread is the primary cause of unpredictable link behaviour and lack of dependence on the RSSI. • Conclusions in Roofnet are likely incorrect, not only for rural mesh networks, but also for urban mesh networks. • Long-running experiments: RSSI is stable, with only a small band of variation of 3-4 dB.