Fred Kuhns fredk@arl.wustl Applied Research Laboratory Washington University in St. Louis

1. Substrate Control:Resource Manager Proxy(See Types.ppt for a description of data types used in this presentation) Fred Kuhns fredk@arl.wustl.edu Applied Research Laboratory Washington University in St. Louis

2. Defining Slices: GNM → RMP • Message context ID = 0 (privileged operation) • retCode_t add_slice(plabID_t, name_t plName) Message context ID = 0 plabID: PLC Globally unique Slice ID. plName : unique string naming slice • must be received on a separate, privileged control socket • return 0 for success or an error code • retCode_t rem_slice(plabID_t) • delete any/all defined fast paths • remove all slice state from rmp • On startup RMP must read the passwd file for a list of currently active slices ... but this will not provide the plabID!!??

3. Reserving Fastpath Resources See Types.ppt for type definition and mapping to substrate API, message context ID = 0. Reserve resources for 0..N fastpaths (on NPEs) and 0..1 GPEs • retCode_t make_reservation(rsvRecord_t) Update an existing reservation, note because all operations need to be idempotent the update argument includes all required resources. • retCode_t update_reservation(rsvRecord_t) Cancel any reservation overlapping date_t • retCode_t cancel_reservation(date_t) Return any reservation (belonging to slice) that overlaps with date_t • rsvRecord_t? get_reservation(date_t) Return a list of all reservations associated with calling slice • rsvRecord_t* get_reservations()

4. XML For a Fastpath Reservation Record <rsvRecord> <rDate start=“Date” end=“Date” /> <fpRSpec> <fpName>name</fpName> <fpParams> <copt> code_option_tag </copt> <bwspec firm=“bw” soft=“bw” /> <rcnts fltrs=“N” queues=“N” buffs=“N” stats=“N” /> <mem sram=“N”, dram=“N” /> </fpParams> <ifParams> <ifRec bw=“N” ip=“ipAddr” /> ... </ifParams> </fpRSpec> <plRSpec> <ifParams> <ifRec bw=“N” ip = “ipAddr” /> ... </ifParams> </plRSpec> </rsvRecord>

5. Allocate Code Option • fpInfo_t alloc_fastpath(copt_t, bwspec_t, rcnts_t, mem_t) copt_t code option (see Types.h) bwspec_t: Aggregate BW requirements for fastpath across all interfaces rcnts_t: Max resource counts mem_t: Requested size of SRAM and DRAM in Bytes (currented fixed by code option) fpInfo_t: NP instance information • RMP must: • send alloc_fastpath request to SRM. SRM performs admissions control, allocates resources and notifies SCD of allocations and new fastpath instance number. Returns the fp parametersfp alloc_fastpath(copt, bwspec, rcnts, mem) • open logical interface and assign it the allocated fp VLAN id, then add NPE’s MAC address to the arp table vconfig add ethX vlan ifconfig ethX.vlan up fpIP netmask fpNM -arp route add -net fpNet netmask fpNM gw npeIP arp -i ethX.vlan -s npeIP npeMAC • send set_gpe_info command to scd, using the last two valid qid’s, set context id to fpid, retCode_t set_gpe_info(exPort, ldPort, exQID, ldQID) • open two UDP sockets using fpIP and two ephemeral ports for local delivery and exception packets. These sockets are passed back to the client as ancillary data. Deprecated

6. Removing a Fastpath • void free_fastpath(fpID_t) : • RMP must: • remove logical interface for fastpath route del -net fpNet netmask fpNM gw npeIP ifconfig ethX.vlan down fpIP netmask fpNM -arp vconfig rem ethX.vlan arp -i ethX.vlan -d npeIP • send set_gpe_info command to scd, setting entries to 0, set context id to fpid,retCode_t set_gpe_info(0, 0, 0, 0) • send free_fastpath to srm with context ID set to fpid retCode_t free_fastpath() • remove remaining fp state and send reply to client Deprecated

7. Instantiate Fastpath and/or Slowpath reservations • All resource allocation methods operate on the currently active reservation (i.e. must be used during the interval of time when a reservation is valid). • If free is not called then all resources are automatically freed when reservation expires. allocate any GPE/Slice resources reserved in the current interval • retCode_t alloc_plspec() instantiate the “current” fastpath reservation with name fpName • fpInfo_t alloc_fpspec(name_t fpName=“”) • if only one fastpath then the fpName parameter can be omitted. free any GPE specific resources currently allocated to slice. • retCode_t free_plspec() Destroy fastpath instance, releasing meta-interfaces and NP resources. • retCode_t free_fpspec(fpID_t) • rmp converts slices specific fpid to global fpid

8. System Interfaces, Address and Ports Reserve Aggregate Interface BW for slice (traffic to/from GPE). • retCode_t resrv_pl_ifbw(ifn_t, bw_t) • Request to reserve bw Kbps on interface ifn for the calling slice Release slice interface BW reservation for GPE • retCode_t reles_pl_ifbw(ifn_t, bw_t): release bw on ifn Reserve aggregate Interface BW for indicated fastpaths • retCode_t resrv_fpath_ifbw(fpID_t, ifn_t, bw_t) • Request to reserve bw Kbps on interface ifn for the calling slice’s fast path Release Aggregate Interface BW allocation for indicated fastpath • retCode_t reles_fpath_ifbw(fpID_t, ifn_t, bw_t) Deprecated

9. System Interfaces, Addresses and Ports Get list of available interfaces • if_list get_interfaces() • Message context ID = 0. if_list = {ifAttr_t, ...} ifAttr_t { ifn_t ifn; // Interface number ifType_t type; // {Multi-Access = 0, Point-to-Point = 1} ipAddr_t ipaddr; // Interface/Link IP address ipAddr_t linkBW; // Raw (Gross) interface (link) BW bw_t availBW; // Available (Net) interface (link) BW } • if type = point-to-point then may call get_ifpeer() Map interface IP address to interface number. • ifn_t get_ifn(ipAddr_t) Get updated interface attributes • ifAttr_t get_ifattrs(ifn_t) Get the IP address of point-to-point link’s peer node (SPP peer). • ipAddr_t get_ifpeer(ifn_t) • If interface ifn is type peering then this returns the IP address of the peer. Otherwise returns null.

10. Allocating EndPoints on the GPE • Message context ID set to 0 Allocate a local endpoint for slice (linecard directs matching traffic to the corresponding GPE) • epInfo_t alloc_endpoint(epInfo_t) • Allocate new endpoint, installs filter in line card to direct matching traffic to the GPE. • if proto is TCP or UDP and port == 0 then the substrate will allocate a port number. • If user requests a port and its not available then an error is returned. • If a slice has not reserved sufficient BW on an interface then an error is returned. • If bw is set to 0 then share BW. Release allocated endpoint (linecard filters removed and address added back to available pool of addresses tuples). • retCode_t free_endpoint(epoint_t) • Removes the endpoint’s filter and bandwidth allocation.

11. Adding Meta-Interfaces Allocate local Address for Slice’s fastpath (MI) and forward to NPE uint8_t tunnel_type : must be one of {UDP = 0} • [mi, ep] alloc_tunnel(fpid, bw, ipaddr, tunnel_type, params)Not implemented • [miID_t, epInfo_t] alloc_udp_tunnel(fpID_t, bw_t, ipAddr_t, ipPort_t) • Adds one end of a UDP tunnel, treated as a meta-interface for a fast path instance. • if port = 0 then one is allocated from the free pool • Returns the meta-interface number representing the new endpoint. • status free_tunnel(ipaddr, tunnel_type, params) Not implemented • retCode_t free_udp_tunnel(fpID_t, miID_t) • epoint_t get_endpoint(fpID_t, miID_t) • Only valid if mi is a valid meta-interface identifier returned from a call to alloc_udp_tunnel or alloc_tunnel Deprecated

12. Adding Meta-Interfaces Allocate local Address for Slice’s fastpath (MI) and forward to NPE, will also reserve sufficient aggregate BW on corresponding internal and external interfaces. • [miID_t, epInfo_t] alloc_udp_tunnel(fpID_t, bw_t, ipAddr_t, ipPort_t) • It is an error if the sum of requested BW exceeds the reserved fastpath BW. • Adds one end of a UDP tunnel, treated as a meta-interface for a fast path instance. • if port = 0 then one is allocated from the free pool • Returns the meta-interface number representing the new endpoint Release BW and Address associated with corresponding meta-interface • retCode_t free_udp_tunnel(fpID_t, miID_t) Map meta-interface ID to corresponding externally visible address tuple • epoint_t get_endpoint(fpID_t, miID_t) • mi must be a valid meta-interface ID returned from a call to alloc_udp_tunnel

13. Manipulating Queues and Attributes • All fastpath queues must have a bandwidth assigned and be associated with an interface. The model is that a user allocates BW on each meta-interface. They may then allocate this BW amongst their queues as they please; the substrate will enforce total BW restrictions. Associate one or more queues with a meta-interface • retCode_t bind_queue(fpID_t, miID_t, list_type, qid_t[]) • uint8_t list_type: explicit list (0) or range specification (1) • qid_t[]: list of queue identifiers to associate with meta-interface mi. In C or C++ this would be an array of integers {0, ..., n}. If using a range then the list has the first and one past the last id of the range. For example {4, 8} is the same as {4, 5, 6, 7}. Set Queue parameters (bandwidth in Kbps and threshold in packets) • bw_t set_queue_params(fpID_t, qid_t, cnt_t threshold, bw_t) Get the current queue parameters for qid. • [cnt_t threshold, bw_t] get_queue_params(fpID_t, qid_t) Get the current queue length in packets and bytes • {cnt_t pktCnt, cnt_t byteCnt} get_queue_len(fpID_t, qid_t)

14. Managing the Lookup table • Common definitions (N = 14 Bytes), Message context ID = 0. Add new filter with ID fid for fastpath fpid (smaller fids have higher priority) • retCode_t write_fltr(fpID_t, fid_t, fpFltr_t) • Can we overwrite existing valid entries?? Don’t think so. • OK to call with an invalid fid, returns an error. Update result vector for filter ID fid • retCode_t update_result(fpid, fid, fpFltrResult_t) • If fid is a valid entry then updates result vector. Otherwise no change to the DB Lookup filter by ID or Key returning the filter entry • fpFltr_t get_fltr_byfid(fpid, fid)fpFltr_t get_fltr_bykey(fpid, fpKeyWrap_t) • returns the filter or an error code. • returns error if called with an invalid FID or Key Lookup filter by key and return the result vector • fpFltrResult_t lookup_fltr(fpid, fpKeyWrap_t) • returns the result field if key/fid matches a valid entry, otherwise NULL Remove filter with indicated FID or Key • retCode_t rem_fltr_byfid(fpid, fid)retCode_t rem_fltr_bykey(fpid, fpKeyWrap_t) • returns success if the entry exists (and was invalidated) or an error code. • Returns error if called with an invalid fid or key.

15. X X X X X T L W uint8_t flags: W : {0 - Bytes, 1 - Packts} L : {0 - PreQ, 1 - PostQ} T : {0 - Push, 1 - Pull} X : Don’t Care 7 6 5 4 3 2 1 0 Stats (accessing FP instance specific counters) One-time read of one of the stats counters for fastpath FPID • statRec_tread_stats(fpID_t, statID_t, flags) • Uses flags L and W to indicate which counter • Returns the counter valus and a timestamp in NBO Set stats counter to 0 • retCode_tclear_stats(fpID_t, statID_t, flags) • result : success or error code Instruct system to create a periodic event to read a stats counter storing values in a circular buffer of size cnt. • handle_t create_periodic(fpID_t, statID_t, cnt_t P, len_t cnt, flags) • uint32_t P : periodic interval in milliseconds • uint16_t cnt : Number of samples to keep in a history buffer • flags : One of {push, pull}. In the push model the substrate will periodical send the requested stats data to a registered UDP port (see set_callback()) . The pull model requires the VM to explicitly issue the get_periodic command (though it need not be periodic). • uint32_t handle : 32-bit opaque handle associated with periodic operation. • returns the last cnt values read Delete period event allocated from a call to create_periodic • retCode_t delete_periodic(fpID_t, handle_t) • Deletes periodic event and removes callback state. Register a UDP port number for the RMP to call periodically (implements the push model). Results sent every cnt*P msecs. • retCode_t set_callback(fpID_t, handle_t, ipPort_t) • callback (push) model for stats collection. • statRec_t vector periodically sent to the UDP port xport using the client VM’s control IP address • If xport is 0 then the callback is canceled. • We need to verify that user has this port already open so we don’t risk flooding an unsuspecting vm. Returns the cnt element stats buffer created from a call to create_periodic (this implements the pull model) • statRec_t* get_periodic(fpID_t, handle_t) • Returns a list of 32-bit statRecs (count plus time stamp)

16. Accessing Code Option Data • Each code option is expected to have a block of SRAM allocated for its exclusive use. • VM control application may read/write to any location within this block • Specific code options may provide libraries to manipulate control structures located in SRAM. • Common types • offset : Byte offset within fastpath instance’s SRAM block. The substrate control software verifies offset. Type uint32_t. • uint8_t data[cnt] : byte array of data to read/write. • rmpCode_t mem_write(fpID_t, cnt_t offset, data) • [len]: Number of Bytes in data vector. A library function should include a length field, however one is not required in the substrate message since it may be determined from the overall message length • data mem_read(fpID_t, cnt_t offset, cnt_t) • len : Number of Bytes to read from SRAM. The substrate control software must verify the read does not access memory outside of the instances block. it is not an error to set len = 0, in this case no data is returned. • Returns a byte array of data (which may be of 0 length) or error.