Global Overlay Network : PlanetLab

1. Global Overlay Network : PlanetLab Claudio E.Righetti 6 October, 2005 (some slides taken from Larry Peterson)

2. “PlanetLab: An Overlay Testbed for Broad-Coverage Services “ Bavier, Bowman, Chun, Culler, Peterson, Roscoe, Wawrzoniak . ACM SIGCOMM Computer Communications Review . Volume 33 Number 3 : July 2003 • “ Overcoming the Internet Impasse through Virtualization “ Anderson , Peterson , Shenker , Turner . IEEE Computer. April 2005 • “Towards a Comprehensive PlanetLab Architecture”, Larry Peterson, Andy Bavier, Marc Fiuczynski, Steve Muir, and Timothy Roscoe, June 2005 http://www.planet-lab.org/PDN/PDN-05-030

3. Overview • What is PlanetLab? • Architecture • Local: Nodes • Global: Network • Details • Virtual Machines • Maintenance

4. What Is PlanetLab? • Geographically distributed overlay network • Testbed for broad-coverage network services

5. PlanetLab Goal “…to support seamless migration of an application from an early prototype, through multiple design iterations, to a popular service that continues to evolve.”

6. Priorities • Diversity of Network • Geographic • Links • Edge-sites, co-location and routing centers, homes (DSL, cable-modem) • Flexibility • Allow experimenters maximal control over PlanetLab nodes • Securely and fairly

7. Architecture Overview • Slice : horizontal cut of global PlanetLab resources • Service : set of distributed and cooperating programs delivering some higher-level functionality • Each service runs in a slice of PlanetLab

8. Services Run in Slices PlanetLab Nodes

9. Services Run in Slices PlanetLab Nodes Virtual Machines Service / Slice A

10. Services Run in Slices PlanetLab Nodes Virtual Machines Service / Slice A Service / Slice B

11. Services Run in Slices PlanetLab Nodes Virtual Machines Service / Slice A Service / Slice B Service / Slice C

12. “… to view slice as a network of Virtual Machines, with a set of local resources bound to each VM .”

13. Virtual Machine Monitor ( VMM) • Multiple VMs run on each PlanetLab node • VMM arbitrates the nodes’s resources among them

14. PlanetLab Architecture • Node-level • Several virtual machines on each node, each running a different service • Resources distributed fairly • Services are isolated from each other • Network-level • Node managers, agents, brokers, and service managers provide interface and maintain PlanetLab

15. Per-Node View Node Mgr Local Admin VM1 VM2 VMn … Virtual Machine Monitor (VMM)

16. Node Architecture Goals • Provide a virtual machine for each service running on a node • Isolate virtual machines • Allow maximal control over virtual machines • Fair allocation of resources • Network, CPU, memory, disk

17. PlanetLab’s design philosophy • Application Programming Interface used by tipical services • Protection Interface implemented by the VMM PlanetLab node virtualization mechanisms are characterized by the these two interfaces are drawn

18. One Extreme: Software Runtimes (e.g., Java Virtual Machine, MS CLR) • Very High level API • Depend on OS to provide protection and resource allocation • Not flexible

19. Other Extreme: Complete Virtual Machine (e.g., VMware) • Very Low level API (hardware) • Maximum flexibility • Excellent protection • High CPU/Memory overhead • Cannot share common resources among virtual machines • OS, common filesystem • High-end commercial server , 10s VM

20. Mainstream Operating System • API and protection at same level (system calls) • Simple implementation (e.g., Slice = process group) • Efficient use of resources (shared memory, common OS) • Bad protection and isolation • Maximum Control and Security?

21. PlanetLab Virtualization: VServers • Kernel patch to mainstream OS (Linux) • Gives appearance of separate kernel for each virtual machine • Root privileges restricted to activities that do not affect other vservers • Some modification: resource control (e.g., File handles, port numbers) and protection facilities added

22. PlanetLab Network Architecture • Node manger (one per node) • Create slices for service managers • When service managers provide valid tickets • Allocate resources for vservers • Resource Monitor (one per node) • Track node’s available resources • Tell agents about available resources

23. PlanetLab Network Architecture • Agent (centralized) • Track nodes’ free resources • Advertise resources to resource brokers • Issue tickets to resource brokers • Tickets may be redeemed with node managers to obtain the resource

24. PlanetLab Network Architecture • Resource Broker (per service) • Obtain tickets from agents on behalf of service managers • Service Manager (per service) • Obtain tickets from broker • Redeem tickets with node managers to acquire resources • If resources can be acquired, start service

25. Obtaining a Slice Agent Broker Service Manager

26. Obtaining a Slice Agent Broker Resource Monitor Service Manager

27. Obtaining a Slice Agent Broker Resource Monitor Service Manager

28. Obtaining a Slice Agent ticket Broker Resource Monitor Service Manager

29. Obtaining a Slice Agent ticket Broker Service Manager Resource Monitor Resource Monitor

30. Obtaining a Slice Agent ticket Broker ticket ticket Service Manager Resource Monitor Resource Monitor

31. Obtaining a Slice Agent ticket Broker ticket ticket Service Manager

32. Obtaining a Slice Agent ticket Broker ticket ticket Service Manager

33. Obtaining a Slice Agent ticket Broker ticket ticket Service Manager

34. Obtaining a Slice Agent ticket Broker Service Manager ticket ticket

35. Obtaining a Slice Agent ticket Broker Service Manager Node Manager ticket Node Manager ticket

36. Obtaining a Slice Agent ticket Broker Service Manager

37. Obtaining a Slice Agent ticket Broker Service Manager

38. PlanetLab Today www.planet-lab.org

39. PlanetLab Today • Global distributed systems infrastructure • platform for long-running services • testbed for network experiments • 583 nodes around the world • 30 countries • 250+ institutions (universities, research labs, gov’t) • Standard PC servers • 150–200 users per server • 30–40 active per hour, 5–10 at any given time • memory, CPU both heavily over-utilised

40. Node Software • Linux Fedora Core 2 • kernel being upgraded to FC4 • always up-to-date with security-related patches • VServer patches provide security • each user gets own VM (‘slice’) • limited root capabilities • CKRM/VServer patches provide resource mgmt • proportional share CPU scheduling • hierarchical token bucket controls network Tx bandwidth • physical memory limits • disk quotas

41. Need to define the PlanetLab Architecture Issues • Multiple VM Types • Linux vservers, Xen domains • Federation • EU, Japan, China • Resource Allocation • Policy, markets • Infrastructure Services • Delegation

42. Key Architectural Ideas • Distributed virtualization • slice = set of virtual machines • Unbundled management • infrastructure services run in their own slice • Chain of responsibility • account for behavior of third-party software • manage trust relationships

43. Princeton Berkeley Washington MIT Brown CMU NYU ETH Harvard HP Labs Intel NEC Labs Purdue UCSD SICS Cambridge Cornell … princeton_codeen nyu_d cornell_beehive att_mcash cmu_esm harvard_ice hplabs_donutlab idsl_psepr irb_phi paris6_landmarks mit_dht mcgill_card huji_ender arizona_stork ucb_bamboo ucsd_share umd_scriptroute … Trust Relationships Trusted Intermediary (PLC) N x N

44. Principals • Node Owners • host one or more nodes (retain ultimate control) • selects an MA and approves of one or more SAs • Service Providers (Developers) • implements and deploys network services • responsible for the service’s behavior • Management Authority (MA) • installs an maintains software on nodes • creates VMs and monitors their behavior • Slice Authority (SA) • registers service providers • creates slices and binds them to responsible provider

45. 1 4 2 3 Trust Relationships MA (1) Owner trusts MA to map network activity to responsible slice (2) Owner trusts SA to map slice to responsible providers 6 (3) Provider trusts SA to create VMs on its behalf Owner Provider (4) Provider trusts MA to provide working VMs & not falsely accuse it (5) SA trusts provider to deploy responsible services (6) MA trusts owner to keep nodes physically secure 5 SA

46. node database MA Owner VM NM + VMM Node Owner Service Provider VM SCS slice database SA Architectural Elements

47. Narrow Waist • Name space for slices < slice_authority, slice_name > • Node Manager Interface rspec = < vm_type = linux_vserver, cpu_share = 32, mem_limit - 128MB, disk_quota = 5GB, base_rate = 1Kbps, burst_rate = 100Mbps, sustained_rate = 1.5Mbps >

48. Node Boot/Install Process Node Boot Manager PLC Boot Server 1. Boots from BootCD (Linux loaded) 2. Hardware initialized 3. Read network config . from floppy 4. Contact PLC (MA) 5. Send boot manager 6. Execute boot mgr 7. Node key read into memory from floppy 8. Invoke Boot API 9. Verify node key, send current node state 10. State = “install”, run installer 11. Update node state via Boot API 12. Verify node key, change state to “boot” 13. Chain-boot node (no restart) 14. Node booted

49. PlanetFlow • Logs every outbound IP flow on every node • accesses ulogd via Proper • retrieves packet headers, timestamps, context ids (batched) • used to audit traffic • Aggregated and archived at PLC

50. Network Activity Slice Responsible Users & PI Chain of Responsibility Join Request PI submits Consortium paperwork and requests to join PI Activated PLC verifies PI, activates account, enables site (logged) User Activated Users create accounts with keys, PI activates accounts (logged) PI creates slice and assigns users to it (logged) Slice Created Nodes Added to Slices Users add nodes to their slice (logged) Slice Traffic Logged Experiments run on nodes and generate traffic (logged by Netflow) Traffic Logs Centrally Stored PLC periodically pulls traffic logs from nodes