1 / 55

SBIR Review Grid of Grids for Information Management

SBIR Review Grid of Grids for Information Management. Anabas May 4, 2007. Outline. Revisit Problem Statement Revisit Research Goals Revisit Research Methodology Part 1: Grid of Grids System And Tools Architecture, Implementation and Status Demo Scenario Summary

shae
Télécharger la présentation

SBIR Review Grid of Grids for Information Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SBIR ReviewGrid of Grids for Information Management Anabas May 4, 2007

  2. Outline • Revisit Problem Statement • Revisit Research Goals • Revisit Research Methodology • Part 1: Grid of Grids System And Tools • Architecture, Implementation and Status • Demo Scenario • Summary • Part 2: Additional Collaboration Grid Services • I. Collaborative Group Design, Implementation and Status • II. Hybrid Shared Display Design, Implementation and Status

  3. Problem Statement • Information and communication have played increasingly critical roles in our nation’s security • The GiG is not one global seamless construct • Different pieces have different stakeholders with different missions • Each has own name and unique vision of network-centric operations • Many operations have been done independently • Unable to satisfy interoperability, scalability, and security information management requirements for Net-Centric Operations without an advanced grid-based scalable service oriented framework

  4. Challenges • Operational tasks (network management, enterprise services management, information staging, and dissemination management) need to be done across the network using common tactics, techniques and procedures • To integrate global grid technology with collaboration technology to provide a framework for network-centric operations to examine and derive warfighter requirements on the GiG

  5. General Goals • Build Net Centric Core Enterprise Services in fashion compatible with GGF/OGF and Industry • Add key additional services including advance collaboration services and those for sensors and GIS • Support Systems of Systems by federating Grids of Grids supporting a heterogeneous software production model allowing greater sustainability and choice of vendors • Build tool to allow easy construction of Grids of Grids • Demonstrate the Grid of Grids capability through one or more relevant applications

  6. Research Methodology • Our solution builds upon existing technology and infrastructure currently being developed across the grid and web services communities • The major innovation is a systematic mapping between NCOW Core Enterprise Services and Grid and Web Service Architectures

  7. Approach • Develop the Grid of Grids architecture and information management system to address federation of legacy and new DoD enterprise systems with service oriented mediation between component collaboration, sensor, information, and computing Grids • Develop static and dynamic Grid Builder Tool compatible with Web service workflow standards • Demonstrate for Earthquake Science and DoD applications

  8. Part 1: Grid of Grids System and Tools

  9. Template Grids Grid Middleware High-level Architecture of the Grid of Grids Prototype System Metadata Service Grid Builder Retrieve Query Output Workflow designer Management System Interface for specific resource Deploy a grid via Condor Discover Input Ready WIP Resources

  10. Current Status • Key Accomplishment • Our work has led to the realization that one can combine static tools like the Grid Builder with dynamic systems management • Individual modules that are ready for use: • Template grids (function complete) • BPEL Workflow designer (function complete) • Resource viewer (function complete) • Management system (development near completion) • Work in progress • The systems management metadata service configuration for application-specific functions (integration in progress) • The integration of management component with Grid Builder (integration in progress) • Customizable management interface for specific resource (being designed) • Integrate workflow designer with workflow execution engine (engine available from OMII but not tested by us yet)

  11. Grid Builder • Grid Builder provides a graphical user interface with functionalities • Template Grid – Grid Builder takes abstract specifications of Grids (i.e., Template Grids) and instantiates them on real infrastructure • Management System - It provides needed management/monitoring, mediation (between sub Grids and Services), and configuration • Workflow Designer - It uses Business Process Execution Language (BPEL) to link services/grids in a workflow and deploy them through a workflow engine

  12. Grids linked as a workflow Grid of Grids (e.g., Earthquake Grid) Portal SOAP messages

  13. Template Grids • Grid Builder can instantiate template grids taking into account scale of events and currently available resources • Template Grids corresponding to anticipated scenarios • It can copy (with perhaps no changes) a Grid from one deployment to another • It can generate the needed management (fault tolerance, monitoring, firewall strategies)

  14. Extended WS-BPEL 2.0 to include the information below, to describe grids : WS-BPEL 2.0 specifies the workflow of services that are federated to complete a common task BPEL workflow Template Grids

  15. <Grid name="Earthquake" targetNamespace="http://cgl.com/ws/earthquakegrid" xmlns="http://schemas.xmlsoap.org/ws/earthquake-process/" xmlns:lns="http://manufacturing.org/wsdl/earthquake"> <services> <applications_services> name="GISGrid" targetNamespace="http://acme.com/ws-bp/gisgrid" myRole="GISInfo"/> <application_services> name="SensorGrid" targetNamespace="http://cgl.com/ws /sensorgrid" myRole="SensorInfo"/> <system_services> name="Security" targetNamespace="http://acme.com/ws-bp/security" myRole="SSH"/> <system_services> name="Registry" targetNamespace="http://acme.com/ws-bp/registry" myRole="UDDI"/> <system_services> name="Management" targetNamespace="http://acme.com/ws-bp/management" myRole="QoS"/> </services> <variables> <variable name="OS" Type="Linux"/> <variable name="Container" Type="tomcat"/> <variable name="Message" Type="SOAP"/> <variable name="Database" Type="ODBC"/> <variable name="Client" Type="Matrix"/> </variables> <contexts> <firewalls Type="" Status = "on"/> <nats type ="" Status = "enabled" <message type="SOAP"/> </contexts> <bpel_workflow> <faultHandlers> <catch faultName="lns:cannotComplete" faultVariable="Fault"> <reply partnerLink="administrator" portType="lns:fault" operation="sendError" variable="POFault" faultName="cannotComplet"/> </catch> </faultHandlers> <sequence> <receive partnerLink="GIS" portType="lns:Filter1" operation="CheckGISInfo" variable="PO"> </receive> <links> <link name="switch_to_Sensor"/> </links> <assign name="Sensor_Info"> </assign> <invoke partnerLink="switching" operation = "submitJob" …. </invoke> <receive partnerLink="switching" portType="lns:earthquakeinfo" operation="send_location_of_earthquakes" variable="filter"> </receive> </sequence> <bpel_workflow> </Grid> An Example of Template Grid in WS-BPEL 2.0 Specification

  16. BPEL Workflow Designer • Define or edit template grids • A GEF-based editor to design a workflow of grids/services • As a precursor to workflow systems. • Fill gap between installing a grid and collecting things into a workflow.

  17. BPEL Workflow Designer Interface A workflow describing the Servo Grid

  18. BPEL Workflow Designer Interface The workflow is associated with resource view

  19. Management System Key functions: • Configuration and lifecycle operations (function complete) • A global view of all accessible resources including their links (work-in-progress) • Resource status monitor (e.g., tracking logs) (work-in-progress) • System status maintenance (recovery, fault tolerance etc) (function complete) • Resource-specific features (being designed) • Input & output interfaces • Unique functionalities • Others

  20. The Integration of the Management System and Grid Builder (1) • Service wrapper for a manageable resource (function complete) • Need to set up a metadata service for handling different types of resources (integration in progress) • Predictable input and output interfaces are defined by metadata (integration in progress)

  21. The Integration of the Management System and Grid Builder (2) • Two schemas for each resource type • Essential info, such as name, type, operating system, security, availability, etc • Non-essential but useful info, such as location, ownership, etc • Metadata specifies both generic and non-generic features of a resource instance • Generic features: life-cycle operations, heartbeat, communication, etc • Non-generic features: specific interface, unique functions, etc

  22. User Interface The current interface is designed for specific resources, which are messaging nodes (brokers)

  23. Grid of Grids Demo Scenario • Building Earthquake Grid - QuickSim2. It provides services such as: • AnalyzeTseri Service (Time Series Filter) • AntVisco Service (Workflow) • Gnuplot Service (Plot) • RDAHMM Service (Datamining by HMM) • STFilter Service (Time Series Filter)

  24. Grid of Grids Demo Scenario • Building Earthquake Grid - QuickSim2. It provides services such as: • AnalyzeTseri Service (Time Series Filter) • AntVisco Service (Workflow) • Gnuplot Service (Plot) • RDAHMM Service (Datamining by HMM) • STFilter Service (Time Series Filter)

  25. Grid of Grids Demo Scenario • These form component Grids • User Assistance (Portal) • Sensors • Filter (Datamining, Compute) • Geographical Information System • Management (being added as part of project)

  26. QuakeSim2 • A representative Web Service Grid application, which includes • Web services: provide access to data and codes • Portlet: acts as an aggregation of client interfaces • We can build web services from the sketch or use those built by others • Data can be retrieved from archives or from real-time filters

  27. Typical use of Grid Messaging in NASA Sensor Grid GIS Grid Grid Eventing Datamining Grid

  28. SERVOGrid has a portal The Portal is built from portlets – providing user interface fragments for each service that are composed into the full interface – uses OGCE technology as does planetary science VLAB portal with University of Minnesota

  29. Streaming Data Support Archival Transformations Data Checking Hidden MarkovDatamining (JPL) Real Time Display (GIS) Grid Workflow Datamining in Earth Science NASA GPS • Work with Scripps Institute • Grid services controlled by workflow process real time data from ~70 GPS Sensors in Southern California Earthquake

  30. QuakeSim Services

  31. QuakeSim Services Continued

  32. Key GIS and Related Services

  33. Electricity CIGrid Security Notification Workflow Messaging DoD CIGrid Earthquake Grid … … Command & Control Services Earthquake Data & Simulation Services Portals Visualization Grid Collaboration Grid Sensor Grid GIS Grid Compute Grid Data Access/Storage Registry Metadata Core Grid Services Physical Network Critical Infrastructure (CI) Grids built as Grids of Grids

  34. Limitations • Lack of management of services such as: • Monitoring of machines and services • Recovery • Hot deployment • Fault tolerance, etc • More flexible approaches to access services are expected • Allowing users to compose collections of services and sub-Grids • The sub-Grids and services can be mapped to resources appropriately

  35. Our Approach • We have developed an extensible framework for managing resources (services included) • A demo of monitoring resources • Further useful extensions will be addressed • Based on WS-Management (WS-Enumeration, WS-Transfer, WS-Eventing, etc) • We use BPEL to represent system and demonstrate a BPEL workflow designer • A workflow engine will be integrated for executing workflows • The management system can provide useful information such as load balancing in the deployment • The prototype system gives an overview of the earthquake grid and allows the user to select services/grids based on the situation assessment

  36. QuakeSim2 Demo

  37. Management System Demo

  38. Workflow Designer Demo

  39. The Infrastructure for the Demo • (Windows XP) • Grid Builder tool • - Grid template • Workflow • Resource monitor (Windows XP) Management System for - Broker nodes - Simulated services Network/Grid Middleware (Linux) - Other Services • (Linux) • QuakeSim2 Portals • ServoGrid Services

  40. QuickSim2 Results Plot of X values *X, Y, Z are 3D coordinates of the station spot Plot of Y values Plot of Z values

  41. Lessons Learned (1) • We demonstrated that one could combine static tools like the Grid Builder with dynamic systems management • The end user, who serves as the administrator, can select a grid template based on the current situation • The extended workflow designer’s Graphical User Interface • enables the user to edit the template and resource requirement • enables accuracy and data integrity improvement of user input • The selected grid will be deployed on available resources automatically and dynamically • saves time and resources

  42. Lessons Learned (2) • The management system keeps monitoring resource status automatically and dynamically • saves time and resources • enables performance improvement • The user can access services through portals which are customizable • enables information access from different devices • Distributed, different services/grids are federated and interoperable in a seamless way

  43. Summary • What we have done • A management system for monitoring resources • A visual workflow editor • Portals with a set of web services • The design of a more complete framework • Why they are important for achieving our objectives • Provide the foundation for integrating for further capabilities of Grid Builder • The demonstration of the prototype system shows the potentials of achieving our objectives successfully • The scope • We leverage the technology actively developed and supported in the grid community • Not aimed to compete for specific systems such as Condor, etc.

  44. Goals and Plan (1) • Short-term goals • Management system for more general resources (in 1 ~ 2 months) • Recovery • Hot deployment • Fault-tolerance, etc • BPEL workflow (in 3 months) • A workflow engine to execute BPEL scripts • Mapping services in a workflow to resources, based on the status info from the management system.

  45. Goals and Plan (2) • Long-term goals • Management system (in 7~9 months) • Metadata service for different types of resources • Predictable input and output interfaces • BPEL workflow (in 7~9 months) • Enhance the workflow execution engine with the capability of match making to optimize the resource allocation

  46. Part 2: Key Additional Services- Advances in Collaboration Grid ServicesI: Collaborative GroupII: Hybrid Shared Display

  47. Advances in Collaboration Grid ServicesI: Collaborative Group – Design, Implementation and Status

  48. Advances in Collaboration Grid ServicesII: Hybrid Shared Display – Design, Implementation and Status

  49. Collaborative Groups Illustrated In Anabas Impromptu • Examples of applications: • private discussions in conference/lecture • simultaneous breakout groups • Multiple broadcasting in the same session (e.g. audio/voice or video/TV channels for user-defined, such as particular need-to-know, groups)

  50. Group & Sharedlets • An Anabas Sharedlet is a shared application, e.g. TextChat, VoIP, Video Conferencing, Shared Applications, Whiteboard • GroupManager provides preliminary Group information to each sharedlet, include joined sessions, active session, session participants, participant privileges (e.g. host, presenter) in each session • Each Sharedlet has its own specific method to handle Group. E.g. • Text Sharedlet stores all conversations in every sessions • Video Sharedlet displays the videos in the active session only • Audio Sharedlet plays the audio in the active session only • Shared Display Sharedlet may store data in every sessions or in the active session only • The Sharedlet specific method depends on network bandwidth requirement (e.g. Is the network bandwidth sufficient?) and usage difference (e.g. Can past data be disposed? Who can share information?)

More Related