Active Messages

1. Active Messages CS63995 - Topics in Software Engineering - Summer 01’ Raquel S. Whittlesey-Harris

2. Active Messages • A Low overhead communication primitive (LWP - Light Weight Protocol) • Developed to take advantage of advances in high speed networking • Increased bandwidths • Decreasing switch latencies • Decrease the software overhead • Dominating communication costs

3. AM - Original Implemenation • Originally developed for MPPs (Massively Parallel Processors) • Only sufficient for Single-Program Multiple-data (SPMD) parallel programs • All AM Comm was constrained within individual parallel processes • Inadequate for Client/Server Apps • Mutually trusting instances of parallel processes

4. AM - Original Implementation • Error and fault models followed the fail-stop semantics • any application error causes the entire process to hang or abort • Supports only single-threaded applications • Supports asynchronous communication-event handling with nonstandard interrupt facilities

5. AM - New Implementaion • Developed for the cluster with support for protected multiprogramming, MIMD style applications and scalability • Exploit hardware capabilities of a network (low latency & high throughput) for a variety of communication interfaces

6. AM - New Implementation • MPI - Message Passing Interface • DSM - Distributed Shared Memory • DFS - Distributed File System • CORBA - Distributed Object Sharing

7. Active Messages Interface • Provides an RPC styled Request/Reply programming model • Allows applications to communicate through communication ports called Endpoints

8. AM - Endpoints • Object representing user or kernel network “PORT” • Potentially, any two endpoints can communicate • Contingent upon protection and authentication • Traditional protection boundaries and domains can be crossed • Kernel, file-system endpoint can communicate with user-level, client endpoints

9. AM - Endpoints • Apps exchange two types of P2P messages • Requests • Replies • Handlers are associated with each message • Executed upon receipt of message

10. Unlike general RPC mechanism (perform computation on data) Handlers extract data from the network & integrate it into the ongoing computation A handler index is passed in the message structure handlers are executed immediately Messages are not buffered Exception of those required for network transport AM - Endpoints

11. AM - Endpoint Structure • Function AM_AllocateEndpoint() - returns an endpoint structure and name • Consists of • Send Pool - for sending messages from an endpoint • Not directly exposed to applications • Random access to entries (“pool” not FIFO) • May be dynamically resized

12. AM - Endpoint Structure • Receive Pool - for receiving messages to an endpoint • Not directly exposed to applications • Random access to entries • May be dynamically resized • Handler Table - for translating handler indices into functions • A form of indirection that affords a level of protection • Eliminates requirement that senders have a priori knowledge of addresses of handlers • Initialized with 256 entries; point to abort() function

13. AM - Endpoint Structure • Function AM_SetHandler() - registers a handler at a particular index • Virtual Memory Segment - for receiving bulk memory transfers • Base address and byte length specify a window into the receiver’s address space into which endpoints with valid tags can write • Base address initialized to zero; length = 0 bytes

14. AM - Endpoint Structure • Translation Table - for associating indices with global-endpoint names and tags • Array associating indices with global-endpoint names and tags • Operations are atomic • Initialized with 256 entries; marked as unused • Function AM_Map() - maps a local name to a global-endpoint name

15. AM - Endpoint Structure • Tag - for authenticating messages arriving at the endpoint • 64-bit integers that authenticate messages • Managed by applications • Initialized to AM_NONE • May be changed at any time • AM_ALL - accept all messages • AM_NONE - accept no messages • Function AM_SetTag() - sets the 64-bit integer endpoint tag • Special Values: never-match, wild-card

16. AM - Endpoint Structure

17. AM - Endpoints • Endpoints can send an AM to another endpoint if and only if • The tag in the sender’s translation table entry for that destination endpoint matches the destination’s endpoint’s tag at the time the message is received. • Applications may coordinate the management of their translations to produce a convenient virtual network • Require exchange of endpoint names and tags • Unaddressed by AM interface specification

18. AM - Endpoint Bundles • Set of endpoints created by one process • Treated as a single unit of communication, event management, and synchronization • Has unique event mask • Function AM_SetEventMask() - set an event

19. AM - Endpoint Bundles • Has unique synchronization variable • Binary Semaphore • Function AM_Wait() - block for an event • Address deadlock issues that arise when multiple independent software packages are composed in a single-threaded application

20. AM - Endpoint Bundle Structure • Function AM_AllocateBundle() - create an endpoint bundle of either • AM_SEQ - at most one thread can access the bundle at any time • AM_PAR - multiple application threads concurrently access bundle

21. AM - Endpoint Bundle Structure • Consists of: • Endpoints - collection of endpoints • An endpoint is a member of exactly one bundle at any given time • Thread Synchronization Variable - binary semaphore • Posted upon event from any endpoint in a bundle • Identity of endpoint responsible is unavailable

22. AM - Endpoint Bundle Structure • Event mask - event bit indicator • Selects which endpoint states or transitions generate events which posts the synchronization variable • Initialized to disable the generation of all events • Under control of application • Access Mode Flag - indicates if concurrent use of the bundle or its endpoints is expected • Informs the system if multiple, application threads will access the bundle or its components concurrently. • Defaulted to serialize concurrent access (AM_PAR)

23. AM - Endpoint Bundle Structure

24. AM - Endpoint Naming Model • Goals & Issues • Maintain independence between interface and implemenation • Allows multiple, higher-level naming systems to be constructed on top of the primitive naming system • Globally unique names • Retain the ability to refer to endpoints locally with small integer values

25. AM - Endpoint Naming Model • Maintain position-independent names • Facilitate migration of endpoints • The system assigns a name upon creation of an endpoint • Not convenient • Not easily manipulated

26. AM - Endpoint Naming Model • Treats global-endpoint names as opaque types • Manipulates either the named endpoint object or indices to the named endpoint objects • Endpoint translation table accomplishes through indirection • Interface is independent of external name servers and global operating systems • Dependent on external agent(s) to manage mappings of global endpoint names to physical resources

27. AM - Endpoint Naming Model • The Translation Table maps small, integer indices to arbitrary endpoint names and tags • Allows for the existence of multiple name managers and naming models • Supports group models • Endpoints are members of communication groups • Named with group name and number (similar to a bundle but in a virtual sense)

28. AM - Endpoint Protection Model • Tags implicitly identify sets of communicating endpoints • Applications use tags to identify aggregates of endpoints • Providing a simple message authentication model Invalid Tag! P58 AM_Request() Aggregate Invalid Tag! P1 Index P23 Name & Tag Get aggregate Valid Tag! P32 AM_Reply()

29. AM - Endpoint Protection Model • Upon receipt of a request, the destination endpoint must match the tag of the sending endpoint • Processes may change an owned endpoint’s tag at any time • Allows processes to revoke sending capabilities of other endpoints (atomic action)

30. AM - Endpoint Protection Model • Special Tag Values • Never-match • Wild card - matches all except never-match • An endpoint with a wild-card tag may have the memory segment corrupted by other endpoints • May copy endpoint VM segments into private memory and inspect before corrupting • Non-bulk messages pass by value therefore are not a threat for corruption

31. AM - Endpoint Synchronization • Upon an event • The system atomically posts the bundle’s binary semaphore and clears the corresponding event mask bit • If Threads are blocked on the semaphore • One is unblocked

32. AM - Transport Operations • P2P interface • Matching requests and reply message pairs • Upon receipt of a request • Request Handler is invoked • Reply exactly once to a request • Error to do other wise • May or may not be detected and reported • Upon receipt of a reply • Reply Handler is invoked • Cannot perform request or reply operations

33. AM - Transport Operations • Transfer Types • Small • Messages composed entirely within a processor’s register set • Pass a small number of words by value (4 or 8) • AM_RequestM() - carries M integers as payload • E.G. AM_Request4() - carries 4 integers as payload

34. AM - Transport Operations • Medium • Typically up to 128 words by value • AM_MaxMedium() - return the maximum transfer size for medium transfers • Large • Source memory regions are copied to user-specified destination endpoint memory segments • Done before message is delivered • AM_MaxLong() - returns the maximum transfer size for large transfers

35. AM - Transport Operations • Tokens • Opaque pointer to data that identifies endpoints • Passed to handler functions • AM_GetSourceEndpoint() - translates a token into the globally-unique endpoint name of sender endpoint • AM_GetDestEndpoint() - translates a token into the globally-unique endpoint name of destination endpoint • AM_GetMsgTag() - retrieves the message tag from a token

36. AM - Error Model • AM are considered sent • The source storage (registers or memory) can be reused • AM are considered delivered • It is entirely written into its destination endpoint’s receive pool

37. AM - Error Model • AM are considered received • Handler function is invoked in receiving process • AM are delivered exactly once barring • Catastrophic network or system errors • persistent congestion at the destination endpoint

38. AM - Error Model • Active Messages are never dropped • Messages can be deemed undeliverable • Returned to sender • Behave like ordinary AM • Undeliverable Message Handler • First entry in every Handler (Handler0) • Notifies applications of undeliverable messages • Default handler is abort()

39. AM - Error Model • void handler (int status, op_t opcode, void *argblock) • Status - details why a message was undeliverable • opcode - identifies type of returned message • argblock (request) - points to structure with original arguments • argblock (reply) - points to structure containing token and original arguments

40. AM - Error Model • For all error conditions except EUNREACHABLE(the destination endpoint was unreachable due to a serious likely catastrophic, system or network error), the system guarantees no events were produced at the destination endpoint • Possibility exists that the message was received by the destination endpoint

41. References • A. Mainwaring and David Culler, “Active Message Applications Programming Interface and Communication Subsystem Organization”, Computer Science Division, University of California at Berkeley, Draft Technical Report, 1995. • N. Parab and M. Raghvendran, “Active Messages”, Center for Development of Advanced Computing, Bangalore, India.