Communication in Distributed Systems

1. Communication in Distributed Systems Cheng-Zhong Xu

2. Taxonomy Communication Point-to-point Multicast group comm Two-way comm One-way comm Socket prog RPC/RMI Message-oriented

3. Outline • TCP/UDP socket programming • RPC/RMI programming • RPC: Remote Procedure Call • RMI: Remote Method Invocation • Distribute Objects • Local object vs remote object • Message-oriented comm

4. Limitations of Socket Programming • Limited support for data structure in comm • In Java TCP/UDP programming, data in transmission is defined as a stream of bytes • client and server are coupled in development: they need to know how the data are interpreted at other side • Limited support for heterogeneity • Network protocols: Internet protocol masks the differences between networks, but • computer hardware: e.g. data types such as integers can be represented differently • operating systems, e.g. the API to IP differs from one OS to another • programming languages, data structures (arrays, records) can be represented differently • implementations by different developers, • they need agreed standards so as to be able to inter-work

5. Architectural heterogeneity: an example Representations of integer 5 and String “JILL” on different machines: • Original message on the Pentium (little endian) • The message after receipt on the SPARC (big endian) • The message after being inverted. The little numbers in boxes indicate the address of each byte • Integers are reversed by byte ordering, but strings are not!!

6. Remote Procedure Call (RPC)Remote Method Invocation (RMI) • Middleware provides a simple programming abstraction and masks heterogeneity of client and server systems • Similar interface as in procedure call or method invocation between client and service • client sends command and arguments • server computes • Server sends back reply data • Implementation of the procedures or methods are on remote servers

7. Conventional Procedure Call For example, count = read(fd, buf, nbytes); • Parameter passing in a local procedure call: the stack before the call to read • The stack while the called procedure is active

8. Remote Procedure Call (RPC) • Build a mechanism, which allows clients • call procedures over a network as if the code was stored and executed locally • The mechanism automates the writing of data-handling and formatting code • Advantages: fewer errors, faster to code and evolve interfaces

9. client program server program client stub server skeleton network call return return call RPC Structure: Client/Server Proxies (stub, skeleton) • Client procedure calls client stub in normal way • Client stub builds message, calls local OS • Client's OS sends message to remote OS • Remote OS gives message to server stub/skel • Server stub unpacks parameters, calls server • Server does work, returns result to the stub • Server stub packs it in message, calls local OS • Server's OS sends message to client's OS • Client's OS gives message to client stub • Stub unpacks result, returns to client

10. Passing Parameters • Steps involved in doing remote computation through RPC 2-8

11. Outline • TCP/UDP socket programming • RPC/RMI programming • RPC: Remote Procedure Call • RMI: Remote Method Invocation • Distribute Objects • Local object vs remote object • Message-oriented comm

12. Distributed Objects • An object is an individual unit of run-time data storage used as the basic building block of programs. • Each object is capable of receiving msgs, processing data, and sending msgs to other objects; methods together form the object’s interface with the outside world or other objects. • A class is the blueprint from which individual objects are created. • Distributed objects have a separation between interfaces and objects that implement the interfaces; interfaces and the objects are in different machines. • Allow programs to treat remote objects just like local objects • remote references • remote invocation • First good impl: DEC SRC network objects for Modula-3 • Recent implementation: Java Remote Method Invocation

13. Organization of Distributed Objects • Separate proxy for each remote object that is referenced • proxy looks just like real object. but loaded at run-time when remote object is to be bound. • implements the same interface • proxy’s methods do RPC to real object • handle arguments, return values, exceptions correctly • proxy code generated by RMI compiler

14. Remote Object References • Naming a remote object (“global address”) • machine (IP address) where actual object resides • port number, identifying the server that manages the object • (boot time) • unique object ID number (generated by server) • Object references can be passed between processes, as parameters to method invocations • Before invocation, object ref. must first bind to its object. Binding leads to a placement of corresponding proxy • Programs referenced a remote object by using stand-in proxy object

15. Remote Invocation • when a proxy method is invoked: • invocation message sent to remote process • contains global address of object, method ID, arguments • remote process invokes real object • return message sent back to proxy object • contains return value or exception • proxy method call returns correct value, or throws correct exception • Under the Hood • each process keeps a table of all proxy objects • maps global address to proxy object • use table to maintain one-to-one mapping from proxy object to remote object • interactions with garbage collection

16. server interface server impl server skeleton client stub client app server app client code Server code stub generator Building an RMI App compiler/ linker compiler/ linker

17. RMI Binding (rmi registry) • How does a client find the service code • Registry to keep track of distributed objects • simply a name repository • provides a limited central management point for RMI • Server binds methods to names in the reg • Client looks up names in the reg for availability of an object

18. Interface Definition • All remote objects are referenced through interfaces • It extends the Remote interface as all RMI interfaces must • Methods must throw RemoteException import java.rmi.*; public interface myServer extends Remote { public void doSomething( … ) throws RemoteException; }

19. // myClient // myServer object has registered itself as “xuObject” import java.rmi.*; import java.rmi.server.*; public class myClient { public static void main( String[] args ) { System.setSecurityManager( new RMISecurityManager() ); try { myServer ro= (myServer)Naming.lookup(“xuObject”); ro.doSomething(); } catch (Exception e) {} } }

20. Notes • All RMI-based app need to import • java.rmi and java.rmi.server packages • Inside the main method, the first thing is set the Java security manager, which grant or deny permissions on the operations • create a try/catch block that performs the remote method invocation

21. // myServer implementation import java.rmi.*; import java.rmi.server.*; import java.rmi.registry.*; public class myServerImpl extends UnicastRemoteObject implements myServer { public myServerImpl() throws RemoteException{ } public void doSomething(… ) throws RemoteException{… …} public static void main( String[] args) { System.setSecurityManager( new RMISecurityManager()); try { myServerImpl obj = new myServerImpl(); Naming.rebind(“xuObject”, obj); System.out.println(“doSomething bound in registry”); }catch (Exception e) {} } }

22. Notes • myServerImpl extends • java.rmi.server.UnicastRemoteObject class, which defines a non-replicated remote object whose references are valid only while the myServer process is alive (Transient Objects). • It supports point-to-point active object reference via TCP streams. • Bind vs rebind • A remote implementation must have a zero-argument constructor

23. Deployment • Generate stub and skeleton • rmic myServerImpl, which generates • myServerImpl_Stub.class, myServerImpl_Skel.class • Since JDK 5.0, stub classes can be generated on-demand at runtime • rmiregistry • starts the registry service in the background • java ServerImpl • starts the server, which registers the remote object with the registry • java Client

24. Passing Arguments • Passing primitive types (int, boolean, etc.) • Remote object vs. non-remote objects • remote object is an instance of RemoteObject • remote object refers to an object whose method can be invoked from another JVM. • Remote objects passed by reference, while non-remote objects passed by copy • A difference between RPC and RMI • passing objects, • passed object may interact with receipt itself

25. Parameter Passing • Remote object vs Local Object • Pass-by-value is ok for all distributed object references, but not efficient • The situation when passing an object by reference or by value • E.g. Client on machine A calls a method of remote object in machine C, which contains two parameters: O1 in machine A and O2 in machine B

26. Passing objects • Object Serialization • To be passed between client/server, objects have to be serialized • For example, • CloudServer, which do jobs for clients in clouds • Different clients may ask CloudServer do different jobs • Each job is specified in arguments passed from a client to the server

27. //Task.java import java.io.*; public class Task implements Serializable{ public Object execute() { return null; } } //foo.java public class foo extends Task{ int n; public foo (int m) {n=m;} public Object execute() { return new Integer( n*n ); } }

28. // CloudServer.java import java.rmi.*; import java.util.*; public interface CloudServer extends Remote { Object execute( Task o) throws RemoteException }

29. // CloudServerImpl.java import java.rmi.*; import java.io.*; import java.rmi.server.*; public class CloudServerImpl implements CloudServer { public CloudServerImpl() throws RemoteException{} public Object execute( Task t) throws RemoteException { return t.execute(); } public static void main (String args[]) { System.setSecurityManager( new RMISecurityManager()); try { CloudServer csr = new CloudServerImpl(); Naming.rebind(“CloudServer”, csr); } catch (IOException ioe) {} }

30. // example.java // java example registry-host import java.rmi.*; import java.io.*; public class example{ public static void main( String[] args ) throws RemoteException { System.setSecurityManager( … ); new example(args[0]); } public example( String host ) { try { CloudServer csr = (CloudServer) Naming.lookup(“rmi://”+host+“/CloudServer”); System.out.println( csr.execute( new foo(10) ); catch (IOException ioe) {} … … }

31. Server site CloudServer.class CloudServerImpl.class CloudServerImpl_Stub.class CloudServerImpl_Skel.class Task.class java CloudServerImpl Client site CloudServer.class example.class CloudServerImpl_Stub.class Task.class foo.class Running on different machines At client site java -Djava.rmi.server.codebase=‘…’ example server

32. Remote Object References • Asynchronous Client/Server Comm • Client calls CloudServer’s asynExecute(…) • On the completion of the work, the CloudServer notifies the client of the result • Client itself is declared as a Remote Object, which reference is passed to CloudServer • CloudServer calls workCompleted(…) of the remote object.

33. // CloudServer.java import java.rmi.*; import java.util.*; public interface CloudServer extends Remote { Object execute( Task work ) throws RemoteException void asynExecute( Task work, WorkListener listener) throws RemoteException }

34. Public class CloudServerImpl extends UnicastRemoteObject implements CloudServer { … … public void asynExecute(Task work, WorkListener listener) throws RemoteException { Object result = work.execute(); listener.workCompleted( work, result ); } }

35. WorkListener Remote Interface // WorkListener.java public interface WorkListener extends Remote { public void workCompleted( Task request, Object result ) throws RemoteException; } Example.java will be modified to implement the remote interface. This turns example into a remote object

36. import java.rmi.*; import java.io.*; public class example extends UnicastRemoteObject implements WorkListener { public static void main( ) throws RemoteException {…} public void workCompleted( Task t, Object result) throws RemoteException { System.out.println(“Async work result = “ + result); } public example ( String host ) throws RemoteException { try { CloudServer csr = (CloudServer) Naming.lookup(“rmi://”+host+“/CloudServer”); csr.asynExecute( new foo(10), this ); catch (IOException ioe) {} }

37. Remarks • Asynchronous RPC: A client and server interacting through two asynchronous RPCs Asynchronous RMI ??

38. RPC vs RMI • RMI supports system-wide object references • RMI can have object-specific stubs, not necessarily have general-purpose client-side and server-side stubs • Pass objects as arguments (object migration) • Migrated objects can interact with both original and new hosts

39. Mobile Objects • Objects be passed as arguments • Single-hop vs multi-hop migration • Dispatched by its host, or migrated according to its own itinerary • Mobile objects can be performed on hosts (cloud servers) according to their own agenda, under a security sandbox • access to local computing and data resources • Mobile objects can communicate back to their home hosts. Weak mobility Stop before migration, and restart on arrival at a new server; running status not transferred Strong mobility Suspend before migration, and resume on arrival at a new server; running status carried over

40. Naplet: A mobile object middleware system • Structured Itinerary Mechanism • Sequential/parallel/alternative visit of a group of servers • Naplet directory based location + Forward Pointer • Message-oriented communication • Persistent comm between mobile objects • Un/secure execution • Multithreaded objects • Secure access to local resource • Java based Naplet 0.2 source code available http://www.ece.eng.wayne.edu/~czxu/software/naplet.html

41. Taxonomy Communication Point-to-point Multicast group comm Two-way comm One-way comm Socket prog RPC/RMI Message-oriented

42. Message-Oriented Persistent Communication

43. Organization of Comm System Examples: Email --- Client/Outbound Mail Server/Inbound Server MPI --- Message Passing Layer Difference??

44. Another Example of Persistent Comm • Persistent communication of letters back in the days of the Pony Express. Although the means of transportation and the means by which letters are sorted have changed, the mail principle remains the same

45. Persistence and Synchronicity • Persistent vs Transient Communication • Persistent: submitted messages are stored by the comm. system as long as it takes to deliver it the receiver • Transient: messages are stored by the comm system as long as the sending and receiving app. are executing. • Asynchronous vs synchronous • Sync: a client is blocked until its message is stored in a local buffer at the receiving host, or actually delivered to the receiver • Async: A sender continues immediately after it has submitted its message for transmission. (The message is stored in a local buffer at the sending host, or otherwise at the first communication server.

46. Persistent asynchronous communication E.g. Email • Persistent synchronous communication (??) (a weaker form: when the sender is blocked until its message is stored at receiver-side comm server )

47. Transient asynchronous communication (e.g. UDP, asynchronous RPC). Transmission fails if the receiver is not executing at the time the msg arrives. • Receipt-based transient synchronous communication . Sender is blocked until the msg is stored in a local buffer at the receiving host. (Example?) 2-22.2

48. Delivery-based transient synchronous communication at message delivery. (Sender is blocked until the msg is delivered to the receiver) • Response-based transient synchronous communication (e.g. RPC and RMI) (Sender is blocked until it receives a reply msg from the receiver)

49. Different Forms of Communication • Persistent Asynchronous • Persistent Synchronous • Transient Asynchronous • Transient Synchronous • Receipt-based (weakest, e.g. async. RPC) • Delivery-based • Response-based (strongest, e.g. RPC) • Need for persistent comm. in middleware, large-scale system • Components of a large scale distributed system may not always be immediately accessible • Failure should be masked with a recovery procedure • But, persistent comm. incurs long delays

50. Message-Oriented Transient Comm. • Socket is a communication endpoint to which an appl can write data to and from which incoming data can be read • Socket primitives for TCP/IP.