The Design and Implementation of Declarative Networks

1. The Design and Implementation of Declarative Networks Boon Thau Loo University of Pennsylvania, University of California-Berkeley* *This dissertation was completed as a graduate student at the University of California- Berkeley

2. Declarative Networking • A declarative framework for networks: • Network protocols are declaratively specified using a database query language • Distributed query engine executes specifications to implement network protocols • Success of database research: • 70’s – today: Database research has revolutionized data management • Today: Similar opportunity to revolutionize the Internet architecture

3. Motivation • Internet faces many challenges today: • Unwanted, harmful traffic • Complexity/fragility in Internet routing • Proliferation of new applications and services • Efforts at improving the Internet: • Evolutionary: App-level “Overlay” networks • Revolutionary: Clean-slate designs • NSF GENI initiative, FIND program Opportunity: Software tools that can significantly accelerate network innovation

4. Recursive Query Execution Network protocol Distributed database Network State Distributed Dataflow Network messages A Declarative Network messages Dataflow Dataflow messages Dataflow messages Dataflow Dataflow Distributed recursive query Dataflow Declarative Networks Traditional Networks

5. The Case for Declarative Networking • Ease of programming: • Compact and high-level representation of protocols • Orders of magnitude reduction in code size • Declarative Chord DHT is 48 lines instead of 10,000. • Easy customization • Safety: • Queries are “sandboxed” within query processor • Potential for static analysis of safety • What about efficiency? • No fundamental overhead when executing standard routing protocols • Application of well-studied query optimizations • Note: Same question was asked of relational databases in the 70’s.

6. Main Contributions of Dissertation • Declarative Routing [SIGCOMM ’05]: • Extensible Routers: balance of flexibility, efficiency and safety • Declarative Overlays [SOSP ’05]: • Rapid prototyping of new overlay networks • Database Fundamentals [SIGMOD ‘06]: • Network specific query language and semantics • Distributed recursive query execution strategies • Query optimizations, classical and new

7. A Breadth of Use Cases • Example implementations to date: • Textbook routing protocols • Chord DHT • Narada mesh for end-system multicast • Distributed Gnutella/Web crawlers • Pastry DHT • Replication protocols • Lamport/Chandy snapshots • Paxos distributed consensus • Overlays for host mobility • Sensor network protocols • P2 declarative networking system • http://p2.cs.berkeley.edu

8. Outline • Background • The Connection: Routing as a Query • Execution Model • Path-Vector Protocol Example • Query specification  protocol implementation • Query Processing • Beyond routing: Declarative Overlays • Ongoing work @ Penn

9. Neighbor Table updates Forwarding Table updates Packets Packets Traditional Router Routing Protocol Control Plane Forwarding Plane Traditional Router

10. Declarative Queries Neighbor Table updates Forwarding Table updates Packets Packets Declarative Router Routing Protocol Query Engine Control Plane Forwarding Plane Declarative Router Traditional Router

11. All-Pairs Reachability R1: reachable(S,D)link(S,D) R2: reachable(S,D)link(S,Z),reachable(Z,D) “For all nodes S,D, If there is a link from S to D, then S can reach D”. link(a,b) – “there is a link from node a to node b” reachable(a,b) – “node a can reach node b” Input: link(source, destination) Output: reachable(source, destination)

12. All-Pairs Reachability R1: reachable(S,D)link(S,D) R2: reachable(S,D)link(S,Z),reachable(Z,D) “For all nodes S,D and Z, If there is a link from S to Z, AND Z can reach D, then S can reach D”. Input: link(source, destination) Output: reachable(source, destination)

13. Towards Network Datalog • Specify tuple placement • Value-based partitioning of tables • Tuples to be combined are co-located • Rule rewrite ensures body is always single-site • All communication is among neighbors • No multihop routing during basic rule execution • Link-restricted rules: Enforced via simple syntactic restrictions

14. Location Specifier “@S” All-Pairs Reachability Network Datalog R1: reachable(@S,D)link(@S,D) R2: reachable(@S,D)link(@S,Z), reachable(@Z,D) Query: reachable(@a,N) Query: reachable(@M,N) link link link link Input table: b c d a reachable reachable reachable reachable Output table: Query: reachable(@a,N)

15. Path Vector in Network Datalog R1: path(@S,D,P) link(@S,D), P=(S,D).  path(@Z,D,P2), P=SP2. path(@S,D,P) link(@Z,S), R2: Query: path(@S,D,P) Add S to front of P2 Input: link(@source, destination) Query output: path(@source, destination, pathVector)

16. path path path Query Execution R1: path(@S,D,P) link(@S,D), P=(S,D). R2: path(@S,D,P)  link(@Z,S), path(@Z,D,P2), P=SP2. Query: path(@a,d,P) link link link link Neighbor table: a b c d Forwarding table:

17. Matching variableZ= “Join” path path path Query Execution R1: path(@S,D,P)  link(@S,D), P=(S,D). R2: path(@S,D,P) link(@Z,S), path(@Z,D,P2), P=SP2. Query: path(@a,d,P) link link link link Neighbor table: Communication patterns are identical to those in the actual path vector protocol a b c d path(@a,d,[a,b,c,d]) path(@b,d,[b,c,d]) Forwarding table:

18. Other Routing Examples • Best-Path Routing • Distance Vector • Dynamic Source Routing (Wireless) • Policy Decisions • QoS-based Routing • Link-state • Multicast Overlays (Single-Source & CBT)

19. Outline • Background • The Connection: Routing as a Query • Query Processing • Beyond routing: Declarative Overlays • Sampling of ongoing work

20. 10 9 3-hop 8 7 6 5 2-hop 4 3 1-hop 2 1 Recursive Query Evaluation • Semi-naïve evaluation: • Iterations (rounds) of synchronous computation • Results from iteration ith used in (i+1)th 9 7 5 2 4 10 1 8 0 3 6 Link Table Path Table Network Problem: Unpredictable delays and failures

21. Pipelined Semi-naïve (PSN) • Fully-asynchronous evaluation: • Computed tuples in any iteration pipelined to next iteration • Natural for network protocols 9 10 7 9 5 6 2 4 10 1 3 Relaxation of semi-naïve 8 0 8 5 3 2 7 6 4 1 Link Table Path Table Network

22. p(x,z) :- p1(x,y), p2(y,z), …, pn(y,z), q(z,w) recursive w.r.t. p Pipelined Evaluation • Challenges: • Does PSN produce the correct answer? • Is PSN bandwidth efficient? • I.e. does it make the minimum number of inferences? • Proofs for • Basic technique: local timestamps

23. Execution Plan Strands Network Out Network In Messages Messages Single Node • Nodes in execution plan (“operators”): • Network operators (send/recv, cc, retry, rate limitation) • Relational operators (selects, projects, joins, aggregates) • Flow operators (mux, demux, queues)

24. Matching variable Z = “Join” Matching variable Z = “Join” Localization Rewrite • Rules may have body predicates at different locations: R2: path(@S,D,P)  link(@S,Z), path(@Z,D,P2), P=SP2. Rewritten rules: R2a: linkD(S,@D) link(@S,D) R2b: path(@S,D,P) linkD(S,@Z), path(@Z,D,P2), P=SP2.

25. Join path.Z = linkD.Z Project path(S,D,P) path Send to path.S linkD Join linkD.Z = path.Z Project path(S,D,P) linkD Send to path.S path Localized Rule Compilation R2b: path(@S,D,P)  linkD(S,@Z), path(@Z,D,P2), P=SP2. Execution Plan Network In Network Out

26. PV/DV (Wired)  DSR (Wireless) Hybridized protocols: Zone Routing Protocol Optimizations • Traditional: evaluate in the NW context • Aggregate Selections • Magic Sets rewrite • Predicate Reordering • New: motivated by NW context • Multi-query optimizations: • Query Results caching • Opportunistic message sharing • Cost-based optimizations • Neighborhood density function • Hybrid rewrites

27. Beyond Routing: Declarative Overlays • Language extensions to support events and soft-state predicates • Chord Routing, including: • Multiple successors • Stabilization • Optimized finger maintenance • Failure detection • 48 rules • 11 table definitions • MIT-Chord: x100 more code • Another example: • Narada mesh in 22 rules 10 pt font

28. Outline • Background • The Connection: Routing as a Query • Query Processing • Beyond routing: Declarative Overlays • Ongoing work @ Penn

29. Ongoing Work @ Penn • Declarative secure networking • Difficult to design/implement/reason about secure networks • Network Datalog + logic-based security languages [NetDB ’07] • Authenticated path vector protocol, DNSSEC, secure DHTs,… • Moving forward: • Exploit fine-grained control over networks and security policies • Data-centric querying and routing in heterogeneous networks • Internet: Wired infrastructure with wireless clouds at the edges • Flexible network support for mobility [ACM MobiArch ’07] • Declarative queries for addressing and naming mobile hosts • Session-aware customizable QoS routing • Moving forward: • Declarative wireless ad-hoc networks • Cost-based query optimizations to adapt protocols

30. Summary • Declarative networking: • Declarative Routing: • Extensible routing infrastructure • Declarative Overlays • Rapid prototyping overlay networks • Database fundamentals • Query language • New distributed query execution strategies and optimizations • Semantics in dynamic networks • P2 declarative networking system (http://p2.cs.berkeley.edu)

31. Many Thanks… • Advisors: Joseph M. Hellerstein, Ion Stoica • Collaborators: • UC Berkeley: Tyson Condie, Ryan Huebsch • Intel Research: David Gay, Petros Maniatis, Timothy Roscoe • Yahoo! Research: Minos Garofalakis, Raghu Ramakrishnan • Rice University: Atul Singh • Many others…