1 / 45

Iosif Legrand California Institute of Technology

DISTRIBUTED SERVICES. Iosif Legrand California Institute of Technology. Distributed Dynamic Services Architecture. Hierarchical structure of loosely coupled services which are independent & autonomous entities able to cooperate using a dynamic set of proxies or self describing protocols.

marah-weiss
Télécharger la présentation

Iosif Legrand California Institute of Technology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DISTRIBUTED SERVICES Iosif Legrand California Institute of Technology LISHEP 2004 Iosif Legrand

  2. Distributed Dynamic Services Architecture • Hierarchical structure of loosely coupled services which are independent & autonomous entities able to cooperate using a dynamic set of proxies or self describing protocols. • They need a dynamic registration and discovery & subscription mechanism • For an effective use of distributed resources, these services should provide adaptability and self-organization (aggregation and hierarchical orchestration) • Reliable on a large scale network distributed environment • Avoid single points of failure • Automatic re-activation of components and services • Scalable & Flexible for adding dynamically new services and automatically replicate existing ones to cope with time dependent load LISHEP 2004 Iosif Legrand

  3. Lookup Stub Service Skeleton Lookup Service Distributed Object Systems CORBA , RMI, DCOM “Traditional” Distributed Object Models (CORBA, DCOM) The Stub is linked to the Client. The Client must know about the service from the beginning and needs the right stub for it Server CLIENT “IDL” Compiler The Server and the client code must be created together !! LISHEP 2004 Iosif Legrand

  4. Interface Lookup Service WSDL Lookup Service Web Services WSDL/SOAP SOAP Server CLIENT The client can dynamically generate the data structures and the interfaces for using remote objects based on WSDL Platform independent LISHEP 2004 Iosif Legrand

  5. Lookup Proxy Proxy Lookup Service Service Any well suited protocol for the application Service CLIENT Dynamic Code Loading Services can be used dynamically • Remote Services Proxy == RMI Stub • Mobile Agents Proxy == Entire Service • “Smart Proxies” Proxy adjusts to the client Distributed Services Act as a true dynamic service and provide the necessary functionally to be used by any other services that require such information (Jini, interface to WSDL / SOAP) • mechanism to dynamically discover all the “Service Units" • remote event notification for changes in the any system • lease mechanism for each registered unit LISHEP 2004 Iosif Legrand

  6. CLIENT Lookup Lookup Service Service Lookup Lookup Service Service Register Service ID Register with ID Service Ask for a lease Get a lease for DT JINI – Network Services A Service Registers with at least one Lookup Service using the same ID. It provides information about its functionality and the URL addressed from where interested clients may get the dynamic code to use it. The Service must ask each Lookup Service for a lease and periodically renew it. If a Service fails to renew the lease, it is removed form the Lookup Service Directory. When problems are solved, it can re-register. jar jar Web Web Server Server Publish the Publish the “Interface” jar “Interface” jar The lease mechanism allows the Lookup Service to keep an up to date directory of services and correctly handle network problems. LISHEP 2004 Iosif Legrand

  7. Mobile Agents • Mobile Agents are automatons entities able to migrate between AgentStations. • Each AgentStation offers the runtime environment, priority and controls the access permission for the hosted agents. • Mobile Agents interact with the AgentStations network to get access to the necessary information to migrate and to perform a certain task. • The agent migration is done as a weak migration ( e.g. the agent prepare itself before it is moved and save its state into structured which a serialized) • Mobile Agents should remain small and simple entities. To achieve this, and provide complex functionality they must use and access “static” services hosted by the AgentStation network. LISHEP 2004 Iosif Legrand

  8. The Framework for Mobile Agents LISHEP 2004 Iosif Legrand

  9. AgentStation provides the platform on which agent services can be hosted and allowed to execute. Agents are authenticated by the station using the security mechanism of the framework before they are allowed to execute. Once an agent has been authenticated, it is allocated a new thread for execution. The Components of the Agent Station LISHEP 2004 Iosif Legrand

  10. Effective File Replication • Identify the efficient way to replicate a data base file to all the interested Regional Centers. Find the right pass and order based on currently available bandwidth and additional constrains of the producer. • Use Remote Events for Global Notification • Allow interested parties to “cooperate” and improve the resource utilization and the global performance and the response time. • e.g. Avoid sending the same file twice over the same transatlantic internet connection if two Regional Centers in U.S. want the same file. • Based on currently available resources to find the optimal way to distribute the file from a global perspective. LISHEP 2004 Iosif Legrand

  11. Java Space Tier1 B Remote Event Notification Tier2 A Tier0 Tier1 A Tier2 B Effective File Replication using Mobile Agents • Example: • Remote Notification for creating a new file • Interested Centers deploy an agent in a time window. • Agents can get from the home Station Sever connectivity parameters • The agent objective is to bring the file home cooperating with other agents • Interested parties may join later and get the information from a JavaSpace LISHEP 2004 Iosif Legrand

  12. LISHEP 2004 Iosif Legrand

  13. Monitoring: Data Collection Dynamic Thread Pool Other tools (Ganglia, MRT…) PULL SNMP get & walk rsh | ssh remote scripts End-To-End measurements Farm Monitor • Configuration Control Trap Listener PUSH snmp trap WEB Server Dynamic loading of modules or agents Trap Agent (ucd – snmp) perl LISHEP 2004 Iosif Legrand

  14. The Muti-Threaded Execution Architecture • Each request is done in an independent thread • A slow agent / busy node does not perturb the measurements of an entire system • Ex: Monitor 300 nodes @ 30 seconds interval 10-15 Threads are running in parallel LISHEP 2004 Iosif Legrand

  15. Data Handling Data Model • Configuration Farm , Function (Cluster), Node, Module • Monitored Values{Parameter, Value} • (Automatic) Mapping of the Data Model in : XML, SQL, SOAP, … • Configuration & Results objects are store in a DB (dynamically configurable for InstanDB, Postgres, MySQl, Oracle … ) • Subscription to results objects matching a template / predicate • Clients can load filter objects into the Data Cache service and generate any derived (or aggregate) data structures and register to receive them. • Monitored parameters have a life time TIME LISHEP 2004 Iosif Legrand

  16. MonALISA Service MonALISA Service MonALISA Service Registration / Discovery / Remote Notification Registration Discovery Client (other service) Lookup Service Services Proxy Multiplexer Data Filters & Agents Services Proxy Multiplexer Lookup Service Client (other service) LISHEP 2004 Iosif Legrand

  17. Service Monitor UNIT & Data Handling Lookup Service Monitor Data Stores Lookup Service Client (other service) Web client WEB Service WSDL SOAP Farm Monitor Data Cache Service & DB Discovery Registration Client (other service) Java data McKoi DB MySQL Predicates & Agents Other tools • Configuration Control (SSL) UDP MySQL • Predicates & Agents User defined loadable Modules to write /sent data MDS LISHEP 2004 Iosif Legrand

  18. Key store Key store Key Secure – Automatic Update Mechanismfor Services and Clients Lookup Service MonaLisa Service Web Server Sign Distribution Discovery • download • Update Signal SSL • download Admin Client • Update Signal SSL MonaLisa Service • All running services are update using the discovery mechanism • At startup each service check if it an update is done at a set of Web Servers • Clients use the Web Start mechanism Lookup Service LISHEP 2004 Iosif Legrand

  19. @ CALTECH Global Client for Farms and Network Connectivity DataTAG - - LISHEP 2004 Iosif Legrand

  20. Real-time Data for Large Systems“lxshare” cluster at cern ~ 600 ndoes LISHEP 2004 Iosif Legrand

  21. Access to historical and real-time values Past values are presented and the GUI remains a registered listener and the new vales are added Real Time Histograms for various parameters LISHEP 2004 Iosif Legrand

  22. Mobile Agents and Filters Simple “Global Load” filter agent From FNAL to all Maximum Flow Data Replication Path Agent Deployed to each RC and evaluates the best path for real-time data replication From CERN to all LISHEP 2004 Iosif Legrand

  23. Lookup Service MonaLisa Service Discovery WAP • Filter Agents / Data TOMCAT JSP/servelts Pseudo Client WEB MySQL IDB MySQL IDB MySQL • Filter Agents / Data MonaLisa Service Lookup Service Pseudo – Clients & Dedicated Repositories LISHEP 2004 Iosif Legrand

  24. MonALISA repositories LISHEP 2004 Iosif Legrand

  25. PDA 3 D MonALISA Client Developed by Research Lab LISHEP 2004 Iosif Legrand

  26. VRVS Architecture pub cal- tech cor- nell funet Reflectors are hosts that interconnect users by permanent IP tunnels. vrvs 5 star- light vrvs us vrvs eu The active IP tunnels must be selected so that there is no cycle formed. usf sinica inet 2 Tree usp The selection is made according to the assumed network links performance. kek triumf LISHEP 2004 Iosif Legrand

  27. Minimum Spanning Tree Algorithms Finding a tree T that contains all the vertices of a graph G spanning tree and has the least total weight over all such trees minimum-spanning tree(MST) Input: A weighted connected graph G = (V,E) with n vertices and m edges Output: A minimum- spanning tree T LISHEP 2004 Iosif Legrand

  28. Kruskal‘s Algorithm • Each vertex is in its own cluster 2. Take the edge e with the smallest weight - if e connects two vertices in different clusters, then e is added to the MST and the two clusters, which are connected by e, are merged into a single cluster - if e connects two vertices, which are already in the same cluster, ignore it 3. Continue until n-1 edges were selected LISHEP 2004 Iosif Legrand

  29. Kruskal‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  30. Kruskal‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  31. Kruskal‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  32. Kruskal‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  33. Kruskal‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  34. Kruskal‘s Algorithm 5 A B 4 6 2 2 D cycle!! C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  35. Kruskal‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  36. Kruskal‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  37. Kruskal‘s Algorithm minimum- spanning tree A B 2 2 D C 1 2 3 E F LISHEP 2004 Iosif Legrand

  38. Barůvka‘s Algorithm • For all vertices search the edge with the smallest weight of this vertex and mark these edges • Search connected vertices (clusters) and replace them by a “new“ vertex (cluster) • Remove the cycles and, if two vertices are connected by more than one edge, delete all edges except the “cheapest“ LISHEP 2004 Iosif Legrand

  39. Barůvka‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  40. Barůvka‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  41. Barůvka‘s Algorithm 5 A B 4 6 2 2 D C 3 1 2 3 E F 4 LISHEP 2004 Iosif Legrand

  42. Monitoring VRVS Reflectors LISHEP 2004 Iosif Legrand

  43. Monitoring VRVS Reflectors LISHEP 2004 Iosif Legrand

  44. Global Client / Dynamic Discovery LISHEP 2004 Iosif Legrand

  45. SUMMARY • MonALISA is able to dynamically discover all the “Service Units" used by a community and through the remote event notification mechanism keeps an update state for the entire system • Automatic & secure code update (services and clients) . • Dynamic configuration for services. Secure Admin interface. • Access to aggregate farm values and all the details for each node • Selected real time / historical data for any subscribed listeners • Active filter agents to process the data and provided dedicated / customized information to other services or clients. • Mobile Agents for decision support and global optimization. • Dynamic proxies and WSDL & WAP pages for services. • Embedded SNMP support and interfaces with other tools ( LSF, PBS, Ganglia, Hawkeye, IEPM-BW…) • Dedicate pseudo-clients for repository, WAP access or decision making units • It proved to be a stable and reliable distributed service system. It is currently running at ~ 100 sites • http://monalisa.cacr.caltech.edu LISHEP 2004 Iosif Legrand

More Related