Communication in Distributed Systems

1. Communication in Distributed Systems

2. Communication in Distributed Systems based on low level message passing offered by underlying network • Three popular models of communication: • Remote Procedure Calls (RPC) • Message-oriented Middleware (MOM) • Data Streaming, discussion of SCTP, 3GPP • Sending data to multiple receivers or Multicasting

3. Interprocess Communication is part of the Client/Server Computing Model • A client is an application or a process that requests a service from some other application or process. • A server is an application or a process that responds to a client request. • Both client/server applications perform specialized functions in a distributed computing environment. • Intraprocess communication used between homogeneous systems. Example : using named pipes, named queues, shared memory etc • Focus of this presentation is Interprocess Communication

4. In between the end users and large pool of computing resources, many applications act as both a client and a server, depending on the situation. • Interprocess communication is necessary to communicate between heterogeneous systems • Protocols govern the format, contents and meaning of messages

5. Two main types of Protocols: • Connection-oriented: Sender and receiver establish a connection and negotiate the protocol to use before exchanging data. At the end of commiunication, terminate or release the connection. An example: TCP • Connectionless: No connection is set up in advance. Sender transmits message when ready. An example: UDP

6. Main differences between connection-oriented and connectionless protocols: • Connection-oriented is reliable, while the other is not. • Connection-oriented is fully duplex • Connection-oriented is byte-stream service with no structure • Error checking an essential component of unreliable connectionless protocol • Connectionless protocol transfers packets of data and uses a protocol port address to specify receiver process

8. Lower-Level Protocols • Implemented in physical layer and data link layer of the stack. Groups data bits into frames and adds a pattern called checksum at either end of frame • Network layer chooses best path from sender to receiver by routing • Transport Protocols • TCP • UDP • Higher Level Protocols • FTP • HTTP

9. Middleware Protocols: • Examples are Authentication and Authorization protocols, commit protocols in transaction databases • Middleware protocols support high level communication services • Protocols that allow a process to call a procedure or invoke an object on a remote machine transparently. An example RPC/RMI • Protocols that support the setting up and synchronizing of streams for transferring real-time data such as multimedia applications. An example: SCTP • Protocols that support reliable multicast services to a WAN

10. Protocol Stack Application Protocol Application 6 Middleware Middleware Protocol 5 Transport Protocol Transport 4 Network Protocol Network 3 Data Link Protocol 2 Data Link Physical Protocol 1 Physical Network

11. Types of Communication • Persistent communication: Message submitted for transmission stored by communication middleware as long as it takes to deliver it to the receiver. Neither sending application nor receiving application need to be executing. • Transient communication: Message stored by communication system only as long as sending and receiving application are executing. • Asynchronous communication: Sender continues immediately after submitting message for transmission. Message temporarily stored by middleware on submission. • Synchronous communication: Sender blocks until message received and processed and receiver returns acknowledgement.

12. Remote Procedure Calls • Remote procedure calls can simplify the way IPCs are conducted through the network. • Client applications use the client side network API to call RPCs. • Server side of the network API turns the RPCs into local function calls. • Return value being transmitted back to the client application again through the network. • The OSI layers are transparent to the client application, as if it is making a local function call. • The RPC provided by Windows enable applications that use RPC to communicate with applications running with other operating systems that support DCE (Distributed Computing Environment). RPC automatically supports data conversion to account for different hardware architectures and for byte-ordering between dissimilar environments.

13. Synchronous RPC Operation: • Process on machine A calls procedure on machine B, the calling process on A is suspended and execution of the called procedure takes place on B. • Information is transported from caller to callee in the parameters and comes back in the procedure result • No message passing visible to programmer • Client and Server stubs are used • Client stub takes its parameters and packs them into a message (parameter marshalling) and sends them to the server stub

14. Parameter Passing: • Passing Value Parameters • Client stub takes parameters and puts them in the message. It also puts the name or number of the procedure to be called in the message • When message arrives at server, server stub examines the message to see which procedure is needed and then makes appropriate call. Stub takes the result and packs it into a message. Message is sent back to client stub. • Client stub unpacks the message to extract the result and returns it to waiting client procedure

15. Passing Reference Parameters • Pointers and references passed by copying the data structure such as array into message and sent to server • Server stub calls server with a pointer to this array • Server makes changes using this pointer that also affects the message buffer inside server stub • Server finishes its work, original message sent back to client stub which copies it back to the client. This is similar to copy/restore.

16. Asynchronous RPC: • Client continues immediately after issuing RPC request and receiving acknowledgement from server, it is not blocked. • Server sends immediately a reply or acknowledgement to the client the moment RPC request is received • Server then calls requested procedure • One-way RPC: Client does not wait for even an acknowledgement from server. Reliability not guaranteed as client has no acknowledgement from server. • Deferred synchronous RPC is a combination of two asynchronous RPCs, where client polls the server periodically to see whether results are available yet rather than server calling back the client.

17. Message-Oriented Communication • RPC assumes that receiver process is executing at the time a request is issued • Message-oriented communication such as message-queuing systems is therefore needed to allow processes to exchange information even if one party is not executing at the time the communication is initiated. • Message-oriented-model (MOM) offered by transport layer, as part of middleware solution

18. Berkeley Sockets • Sockets interface introduced in 1970s in Berkeley Unix • Standardizes the interface of the transport layer to allow programmers the use of messaging protocols through simple set of primitives • Another interface XTI stands for X/Open Transport Interface, also formerly called Transport Layer Interface (TLI) developed by AT&T • Sockets and XTI similar in their model of network programming but differ in their set of primitives • Socket forms an abstraction over the communication end point to which an application can write data that are sent over the underlying network and from which incoming data can be read. • Servers execute the first four primitives in table • When calling the socket primitive, the caller creates a new communication end point for a specific transport protocol. Internally, the OS reserves resources to accommodate sending and receiving messages for the specific protocol.

19. Bind primitive associates a local address with a newly-created socket. For example, server should bind the IP address of its machine together with a known port # to a socket • Binding tells the OS that the server wants to receive messages only on the specified address and port • Listen primitive is called only in the case of connection-oriented communication. It is a nonblocking call that allows the local OS to reserve enough buffers for a specified # of connections that the caller is willing to accept • A call to accept primitive blocks the caller until a connection request arrives. When the request arrives, the local OS creates a new socket with same properties as the original one and returns it to caller. Server can wait for another connection request on the original socket • Connect primitive on client requires that the caller specifies the transport level address to which a connection request is to be sent • Client is blocked until a connection is set up, after which info can be exchanged using send/receive • Closing the connection is symmetric as both server and client call the close primitive

21. Message-Passing Interface • A standard defined for message passing • Hardware and platform independent • Designed for parallel applications and transient communication. • Makes use of underlying network

22. Assumes that serious failures such as process crashes or network partitions are fatal and do not require automatic recovery • MPI assumes communication takes place within a known group of processes • Each group is assigned an identifier • Each process within a group is also assigned a local identifier • A (groupID, processID) pair uniquely identifies the source or destination of a message, and is used instead of a transport-level address • Several possibly overlapping groups of processes involved in a computation, executing at the same time • MPI has messaging primitives to support transient communication shown in next table

24. Transient asynchronous communication is supported by MPI_bsend primitive • Sender submits a message for transmission which is copied to a local buffer in MPI runtime system. Sender continues after message is copied. Local MPI runtime system will remove the message from its local buffer and transmit as soon as a receiver has called a receive primitive • MPI_send is a blocking send operation with implementation dependent semantics • It may either block the caller until the specified message has been copied to the MPI runtime system at the sender’s side, or until the receiver has initiated a receive operation • MPI_ssend implements synchronous communication by which the sender blocks until its request is accepted for further processing • MPI_sendrecv when called by sender, sends a request to the receiver and blocks until the latter returns a reply. This corresponds to normal RPC.

25. MPI_isend allows the sender to pass a pointer to the message and the MPI runtime system takes care of the communication. Sender continues. • MPI_issend allows the sender to pass a pointer to MPI runtime system. When runtime system indicates it has processed the message, sender knows that receiver has accepted the message. • Caller is blocked until message arrives when MPI_recv is called to receive a message • Asynchronous variant MPI_irecv called by receiver indicates it is prepared to accept message

26. Message-Oriented Persistent Communication • Message Oriented Middleware(MOM) or message queuing systems provide support for persistent asynchronous communication • Intermediate-term storage capacity for messages. Does not require sender/receiver to be active during message transmission • Slower than Berkeley sockets and MPI

27. Message-Queuing Model • Applications communicate by inserting messages in specific queues • Messages are forwarded over a series of communication servers and delivered to the destination, even if it was down when message was sent • Each application has its own private queue to which other applications can send messages

28. Sender is informed that the message will be eventually inserted in the receiver’s queue. No time is specified. • Neither sender nor receiver need to be executing when message is placed in the queue • Loosely coupled communication with sender and receiver executing independent of each other.

30. Architecture of Message Queuing System • Source queue to put message is local to sender • Message can be read only from local queue • Message put into a queue will contain the specification of a destination queue to which it should be transferred • Message queuing system responsible to provide the queues to sender/receiver and transfer messages from source to destination queue.

31. Message queuing system maintains mapping of queues distributed across multiple machines to network locations – a distributed database of queue names to network locations, similar to DNS. • Queues managed by Queue Managers, who interact with the application that is sending or receiving a message. Special queue managers operate as routers/relays that forward incoming messages to other queue managers

32. Message-queuing system may grow into a complete application level overlay network on top of existing computer network • Relays/routers help build scalable messaging systems • Relays allow secondary processing of messages, for example, a message may need to be logged for reasons of fault tolerance or security • Relays can be used for multicasting purposes. Incoming message put into each send queue.

33. E-mail Systems • E-mail systems use underlying transport services. Example, mail protocol for the Internet-SMTP, a message is transferred by setting up a direct TCP connection to the destination mail server. Generally no routing is used. • Provide direct support for end users when compared to message-queuing system • E-mail systems have specific requirements such as automatic message filtering, support for advanced messaging databases to retrieve old messages etc • Message queuing system enables persistent communication between processes. Wide range of applications including e-mail.

34. Stream-oriented Communication • Communication discussed till now dealt with independent, complete units of information and time had no effect on correctness of the communication • Stream communication such as audio or video stream in contrast to above are time-dependent • Information is represented in different formats such as GIF/JPEG for images, audio streams are encoded by taking 16-bit samples using PCM

35. In continuous representation media, temporal relationship between data is retained in order to correctly interpret the data. For example, motion can be represented by a series of images with successive images displayed at uniform spacing T in time (about 30-40 msec per image) • In discrete representation media, temporal relationships between data is not fundamental to correctly interpreting the data. Examples are text, still images etc

36. Data stream is a sequence of data units that can be applied to discrete as well as continuous media. Examples are UNIX pipes, TCP/IP connections of byte-oriented discrete data streams. Playing an audio file requires setting up a continuous data stream between the file and the audio device • Time is crucial to continuous data streams

37. Three types of transmission modes exist for data streams • Asynchronous transmission mode where data items are transmitted one after the other but no time constraints as to when the transmission of each item takes place. Example for discrete data streams : file transmission • Synchronous transmission mode where there is a maximum end-to-end delay defined for each data unit in stream. Example is Temperature sampled by sensors and passed over network • Isochronous transmission mode where time constraint is rigid and data units are transferred subject to maximum and minimum end-end delay (bounded (delay) jitter). Example are distributed multimedia systems such as audio/video

38. Focus in the presentation is on continuous data streams (streams) using isochronous transmission • Simple stream consists of single sequence of data • Complex stream consists of several related simple streams called substreams that are interdependent on each other based on time. Example is a video stream such as movie where two substreams continuously synchronized to transmit audio for the movie, a video substream, and a substream containing subtitles for the deaf or different language translation. All substreams are synchronized.

40. Architecture in the figure reveals some issues such as compression needed to reduce required storage and network capacity especially more for video than audio • Quality of transmission and synchronization to be controlled • Timing requirements expressed by Quality of Service(QoS) QoS for continuous data streams concerns the timeliness, volume and reliability of transmission.

41. QoS Specification • The required bit rate at which data is transported • Maximum delay until a session is set up(i.e., when an application can start sending the data). • Maximum end-to-end delay • Maximum delay variance or jitter

42. Enforcing QoS • Use buffers to reduce jitter • Use forward error correction to compensate for lost packets- encode the outgoing packets such that any k out of n received packets is enough to reconstruct k correct packets • Many distributed systems for stream-oriented communication are built on top of Internet protocol stack. Internet provides differentiating classes for data using differentiated services. Sending host can mark outgoing packets as belonging to one of several classes. Expedited forwarding class specifies that the packet should be forwarded by the router with absolute priority. With assured forwarding class, traffic is divided into 4 subclasses along with three ways to drop packets if the network gets congested. This means a range of priorities can be assigned to the packets to differentiate time-critical packets from non-critical ones.

43. Stream Synchronization • Maintains temporal relationships between streams/substreams , for example between discrete data stream and continuous data stream, or between continuous data streams • Synchronization takes place at the level of data units • Synchronization mechanisms concerned with synchronizing data streams and distribution of the mechanism in a networked environment

44. Multimedia middleware offers a collection of interfaces for controlling audio and video streams including interfaces for controlling devices such as monitors, cameras, microphones etc. • Each device and stream has its own high level interface including interfaces for notifying an application when some event occurred. Latter used for writing handlers for synchronizing streams. • Distribution of synchronization mechanism - receiving side has to have complete synchronization specification locally available

45. This approach is followed by MPEG streams • MPEG standards form a collection of algorithms for compressing video and audio • MPEG -2 designed for compressing broadcast quality video into 4 to 6 Mbps. Unlimited number of continuous and discrete streams merged into single stream. Input stream turned into a stream of packets that carry timestamp based on 90kHz clock. These streams multiplexed into a program stream consisting of variable length packets with common time base. Receiving side demultiplexes the stream using timestamp for interstream synchronization. Better to do synchronization at the sender rather than at receiver.

46. Stream Control Transmission Protocol (SCTP) • SCTP is a reliable transport protocol operating on top of a connectionless packet network such as IP. It is described in RFC 4960, RFC 3286. It offers the following services to its users: -- acknowledged error-free non-duplicated transfer of user data -- data fragmentation to conform to discovered path size -- sequenced delivery of user messages within multiple streams, with an option for order-of-arrival delivery of individual user messages -- optional bundling of multiple user messages into a single SCTP packet -- network-level fault tolerance through supporting of multi-homing at either or both ends of an association • The design of SCTP includes appropriate congestion avoidance behavior and resistance to flooding and masquerade attacks.

47. The Stream Control Transmission Protocol (SCTP) is a new IP transport protocol, existing at an equivalent level with UDP (User DatagramProtocol) and TCP (Transmission Control Protocol), which provide transport layer functions to many Internet applications. • SCTP has been approved by the IETF as a Proposed Standard • Like TCP, SCTP provides a reliable transport service, ensuring that data is transported across the network without error and in sequence. • Like TCP, SCTP is a session-oriented mechanism, meaning that a relationship is created between the endpoints of an SCTP association prior to data being transmitted, and this relationship is maintained until all data transmission has been successfully completed. • Unlike TCP, SCTP provides a number of functions that are critical for telephony signaling transport, and at the same time can potentially benefit other applications needing transport with additional performance and reliability. The original framework for the SCTP definition is described in [3].

48. SCTP User Application SCTP User Application SCTP Transport Service SCTP Transport Service One or more IP Address Appearances One or more IP Address Appearances IP Network Service IP Network Service SCTP Node B SCTP Node A Network Transport An SCTP Association

49. Message Format Common Header Chunk No: 1 … Chunk No: n

50. Chunk ID : 0 through 255. Each ID has Chunk Type defined as follows: • 0 - Payload Data (DATA) • 1 - Initiation (INIT) • 2 - Initiation Acknowledgement (INIT ACK) • 3 - Selective Acknowledgement (SACK) • 4 - Heartbeat Request (HEARTBEAT) • 5 - Heartbeat Acknowledgement (HEARTBEAT ACK) • 6 - Abort (ABORT) • 7 - Shutdown (SHUTDOWN) • 8 - Shutdown Acknowledgement (SHUTDOWN ACK) • 9 - Operation Error (ERROR) • Etc …