Simulations in wireless sensor and ad hoc networks

1. Simulations in wireless sensor and ad hoc networks Comparison, ‘proof of concept’, independent variable, simplicity issues … IEEE Communications Magazine 2008 Ivan Stojmenovic Ivan@site.uottawa.ca www.site.uottawa.ca/~ivan Ivan Stojmenovic

2. Sensor may measure Internet • Distance, Direction, Speed • Humidity, Soil makeup • Temperature, Chemicals • Light, Vibrations, Motion • Seismic data, Acoustic data • strain, torque, load, pressure • Self-configure into wireless multi-hop network Ivan Stojmenovic

3. Traditional wireless sensor networks Ivan Stojmenovic

4. Wireless networks: a taxonomy • Multi-hop self-organized networks • Conference, battlefield, rescue • Peer to peer networks • Ad hoc networks • Single hop networks • Cellular networks • Satellite networks Ivan Stojmenovic

5. Hybrid ad hoc wireless networks • Sensor networks • Cellular multi-hop networks • Mesh/rooftop networks: wireless fast Internet access • Vehicles on highway Ivan Stojmenovic

6. Multi-hop networks: Routing Sensor networks Position information Unit disk graphs radius • Routing: source destination Ivan Stojmenovic

7. Three approaches to simulation • Ignoring = mathematical • Light = computing • Heavy = engineering • Correlation between approach and background is high bit not perfect Ivan Stojmenovic

8. Mathematicians • strong theory weak or no simulation: • lot of theorems/proofs • Assumptions to suite theorem proving and not practice • not much practical relevance • not much support to computer scientists or engineers: • ‘your algorithm is too simple, there are no theorems..’ • worst case performance more important than average case, because • It is possible to prove theorems about worst case while • It is normally infeasible to prove nontrivial theorems about average case performance, thus simulations … Ivan Stojmenovic

9. Mathematician - contributions • But sometimes they do great job, when assumptions are accidentally right.. • Jie Wu et all: localized connected dominating sets • Li, Hou, Sha, and followers: LMST: Localized minimal spanning trees • BMSU: Applying Gabriel graph to route with guaranteed delivery: GFG (duplicated as GPSR) Ivan Stojmenovic

10. More mathematical contributions • Gupta, Kumar: capacity analysis for routing in ad hoc networks • XY Li et all: Game theoretical approach on selfish routing; topology • Penrose, Bettsetter,…: connectivity properties of random unit graphs Ivan Stojmenovic

11. Mathematical community isolated? • Despite lack of realism/assumptions in most papers • Many of them appear in top conferences/journals • Because • Reviewers are likely to be mathematician with similar philosophy • And it is difficult to survive with new/lone philosophy, good or bad.. Ivan Stojmenovic

12. Computing approach • Mix/balance of • Algorithms: what is really the new idea? • some simulations: how does it compare with others under assumptions used in its design, • (Sometimes) some theory, to support claims • algorithms are more important than simulations • simplified simulations to extract major properties • Evolution of assumptions and contributions • One problem at a time (for one paper), fully processed. Ivan Stojmenovic

13. Example: simplified approach • if routing A is significantly better than routing B • on a single routing task with ideal MAC layer and home-made simulator, • it is highly likely that 1000 simultaneous routing tasks A will be better than 1000 simultaneous routing tasks B • with 802.11 and NS-2, Glomosym, and other traffic… Ivan Stojmenovic

14. Engineering approach • weak theory strong simulation • Engineers dominated by 2000 with almost full control • Simulations with realistic physical layers although protocol design uses only simple model • MAC and transport layers added • Thus multiple variables and multiple problems studied together • Understanding and explanations are then more difficult • Hoping to be more useful for immediate applications.. • Unfortunately studies show huge mismatch between simulations and testbeds… Ivan Stojmenovic

15. Frequent ‘engineering’ approach • ‘I am best’ approach: compare with something expected to be worse, ignore existing better solutions • Literature review incomplete • Example: introduce power, delay etc. metric into DSR/AODV routing protocol, compare with DSR/AODV that uses hop count metric in decisions… • Use simulator that was used by others; hide many detail, • add transport and medium access layer immediately to network layer - complicated scenarios • Simulation diagrams to be impressive, choose parameter values showing good performance • Algorithmic description incomplete, vague, or given by pseudo-code Ivan Stojmenovic

16. More on ‘I am the best’.. • Assumptions look realistic, but often ‘forgetting’ to measure something, • e.g. not measuring communication overhead for collecting required global information in sensor or mobile ad hoc networks • when applying centralized solution! • And not mentioning existence of any localized algorithm • Where nodes make decision based on limited local knowledge, easy to gather Ivan Stojmenovic

17. Engineering ‘new’ solutions: painting • Introduce new vocabulary/terminology for well known problem statement, then show ‘new’ solution • Example: Directed diffusion (Estrin et all): • It is not a route discovery by DSR/AODV (never mentioned), but it is flooding with ‘interests’ and setting ‘gradient’ pointers backward • (other examples) • It is not routing but data gathering, reporting … • It is not clustering but network organization.. • Hide from literature review done by others.. Ivan Stojmenovic

18. Simulation - issues • Good compared to what ? • ‘proof of concept’ - what is that? • Independent variable • Simplicity • Parallel advance of useful modeling and protocol design Ivan Stojmenovic

19. Existing criticism of simulations • Takai, Martin, Bagrodia, Mobihoc 2001: • changing parameters and simulators leads to different comparative outcome • Perrone, Yuan, and Nicol 2003: • most published experimental data cannot be replicated because of insufficient information given • Kotz et al 2004: • simulation results differ significantly from experimental testbed results! Ivan Stojmenovic

20. More existing criticism • [AY] T.R. Andel, A. Yasinsac, On the credibility of manet simulations, IEEE Computer, July 2006, 48-54. • lack of statistical validity (no confidence intervals) • lack of sensitivity analysis (no analysis of variance over multiple variables) • inappropriate radio models, unrealistic application traffic and lack of real world implementation. • Finally, due to improper precision, [AY] advice to use simulations to provide proof of concept and general performance characteristics, not to directly compare multiple protocols against one another. • THUS PROMOTING ‘I AM the BEST’ APPROACH! Ivan Stojmenovic

21. Science vs Engineering • Science: new solution aims to be the best for given problem under given assumptions and scenarios • Literature review then needs to be thorough to identify proper competing solutions • Engineering: new solution should be good solution that works for given problem in a practical setting • Validation more important than competition • Less emphasis to originality and competitors • Modeling and assumptions in ‘practical’ problem ? • Validation by simulation may be questionable • Best solution does not mean a valid solution in practice - only under given model and assumptions Ivan Stojmenovic

22. What is the ‘proof of concept’? • [AY]: possible only after thorough simulations and even testbeds - thus impossible for a single paper! • alternative: basic simulation, simple model and scenario, • Matching model with assumptions used in protocol design • Simulations serve as evidence of average case performance in the absence of difficult math derivations • Repeatability, independent verification, no ‘realistic’ modeling on some simulators with hidden agenda Ivan Stojmenovic

23. More realism may defeat a protocol Flooding: simple, winning in ideal model without collisions • Poor in dense networks with collisions • More messages may be counterproductive - collisions and loss of information • Defeated when collisions are added, motivation to • Design broadcasting protocols based on backbones • But new protocols again evaluated starting from basic modeling without collisions, then add MAC • greedy routing designed with hop count but evaluated with expected hop count: change metric and redesign! Ivan Stojmenovic

24. Selecting independent variables • Transmission radius or neighbor density ? • Sparse, medium and dense graphs differ in routing performance, which may be hidden when TR is used • The real purpose of simulation should be to gain understanding and explain results, and provide stepping stone toward a better protocol (in the next article). Ivan Stojmenovic

25. Stepwise approach • Solve one problem at a time • Explain one variable at a time, fix others • Add one more realism at a time • (e.g. simple MAC layer) • Parallel advance of models and protocols • Every model is unrealistic but some of them are useful! Ivan Stojmenovic

26. Reviewing: easy to be negative • Simulations: never enough … • Model: never realistic … • Protocol is too simple or too complicated. • Simple solutions can be modified to address more realism and eventually adopted • No math/analytical results … but may be it is too hard.. • please identify contributions first… and then • check page limit.. Ivan Stojmenovic

27. Extracting more convincing data • Assume your algorithm A has cost 80 while competing algorithm B has cost 100. • You seem to be better by 20%. Is that enough for a good conference? • A and B are localized competing algorithms • Centralized algorithm C has cost 60. This is minimal ‘nonnegotiable’ price to pay. Deduct it from costs of A and B, so their costs are 20 and 40 • Now cost reduction achieved is 50%. Ivan Stojmenovic

28. Avoid parameters • Example: threshold based DSR/AODV • Eliminate ‘bad’ links or nodes, use good ones only • But one bad link may still contribute to acceptable overall route, or to only possible route for important traffic • Why not simply use best links/nodes at each step? Ivan Stojmenovic

29. Localized algorithms • Scalability: algorithms work well (or still work) on ad hoc networks with large number of nodes • Globalized algorithms: global network information or global structure required (e.g. for shortest path) • Localized algorithms: Decisions made based only on information from neighbors and natural additional information (e.g. destination for routing) • Local localized: Maintenance remains local • Quazi-local localized:Local changes may trigger global updates • Mobility or changes between active and sleep periods in ad hoc and sensor networks require localized algorithms, preferably local localized Ivan Stojmenovic

30. centralized algorithms? • ?? linear programming running on 20-30 nodes for problems with >100 nodes • However: applying centralized algorithms on local knowledge often leads to winning solutions: • Power aware routing: apply shortest weighted path toward selected neighbor • Minimal energy broadcasting: Node applies centralized algorithm on 2-hop knowledge - very close to performance of centralized solution! Ivan Stojmenovic

31. Memorization and message count ? • Avoid/reduce memorization at nodes, because that node may not be active or at expected place when the stored information is needed • Number of messages between neighbors: few ? O(degree) ? More ? • Number of messages between neighbors to run a protocol should be very limited (e.g. under five), possibly even zero after ‘hello’ messages for backbone construction, since • Bandwidth and power are limited, and • Impact of realistic physical layer: unreliable receptions Ivan Stojmenovic

32. Conclusion • Existing criticism on simulation practices are limited in their scope and often severely misleading and unfair • They provide more support for applying unfair reviews than to support sound research toward ultimate solution in stepwise approach • Current advices are like flooding toward sink with known position information: try everything in all directions, keep paid student busy for a while … • Why not instead making one greedy intelligent step in the right direction at a time? • I hope for more support on presented views. • Only 33 citations… Ivan Stojmenovic