A Concept of a Monitoring Infrastructure for Workflow-Based Grid Applications

1. A Concept of a Monitoring Infrastructure for Workflow-Based Grid Applications Bartosz Baliś, Marian Bubak, Włodzimierz Funika, Tomasz Szepieniec, Marcin Radecki, Roland Wismüller, Tomasz Arodź, Marcin Kurdziel

2. Plan • Workflow-based applications • Application Monitoring • Architecture of the monitoring infrastructure • Performance analysis • Monitoring of multilingual applications • Summary

3. Workflow-based Grid Applications • Applications composed of a workflow of components • Components are independent Grid services

4. Application Monitoring • Goal: provide monitoring services to • Obtain debug / performance-related information • Possibly enable manipulations on the target application • Detect events and program actions to be executed when they occur • Consumers • Tools for performance analysis / debugging • Other tools and systems, e.g., for fault-tolerance, load balancing, etc.

5. Monitoring infrastructure for Wf-based apps • Additional monitoring interface for each grid service (component) to be monitored • Obtain monitoring info for that component • Global monitoring service • Separate grid service • One per application • Obtain collective monitoring info

6. Monitoring Wf-based Applications Client Global monitoring (grid) service. Collective monitoring information. Additional monitoring interface (monitoring info related to the component) Monitoring service

7. Startup • User submits a workflow application and requests monitoring, possibly subsequently • Monitoring service is created • It must discover the workflow components • Workflow subsystem must provide a mechanism for this • Workflow registry?

8. Component monitoring interface – how? • Monitoring functionality inherent part of each component • How to provide the additional monitoring interface? • Take monitoring interface into account at the design stage • Component developers involved? • This approach (hopefully) enables instrumentation of grid services

9. Performance analysis service • Combined with the monitoring service • Supports performance analysis of • Grid infrastructure • Application • Intra-component • Inter-component

10. Infrastructure and intra-component analysis • Grid infrastructure • Uses existing tools to monitor the status of grid environment of the components • Intra-component operations • Measure some quantities describing the status and performance of a service used in the application workflow • Depend on the monitoring interface defined by the component developers • Reuse existing approaches to monitoring and analysis

11. Inter-component performance monitoring • Adds a new level to the performance analysis of grid applications • Accounts for the concept of application as a workflow of services • Monitors the state and performance of the cooperation of the services to capture the state and performance of the whole application • Two modes: • Structural, semantic monitoring mode • Activity monitoring mode

12. Structural, semantic monitoring • Captures information on workflow components: • Component status • Component usage • Custom component properties • Uses the intra-component monitoring data on: • Functions • Code regions • Synchronisation objects • Combines the information on various components to describe the status of the whole workflow based-application • Proposed techniques: • Source/byte code instrumentation • Dynamic instrumentation

13. Activity monitoring • Captures information on the behaviour of the workflow • Analysed activity properties: • Caller/callee relationships • End-to-end response time for activity invocation • Success rate of invocation • Synchronisation time in activity • Volume of data exchanged in activity • Proposed techniques: • Distributed instrumentation of workflow components • Insertion of probes

14. Event Service • Automatically sends notificationsto other grid services when some performance conditions are met. • Allows for a reaction on undesired changes of grid environment status as well as in the activity of the application. • Will be integrated with performance data provider in a uniform framework. • This integration will enable the notified service to inspect a wide range of performance information before deciding on the actions that should be taken.

15. Two levels of performance information • A proven concept form the G-PM tool. • Used in both application performance data provider and event service. • Lower level performance information: • a consistent, well-defined set of metrics, • used, by system administrators, application developers,users andother parties. • Higher level performance information: • more abstract, numerical, quantitative metrics, • defined on the basis of the lower-level metrics with a special definition language, • allowing for creation of various metrics suitable for a particular purpose.

16. Monitoring of multi-lingual applications • Services can be implemented using multi-lingual approach e.g. as a computational kernels written in Fortran and wrapped by Java code. • A higher level abstraction of metrics must be worked out which could handle different programming approaches and languages. • The low-level monitoring system must be designed in a way that enables a simultaneous monitoring of Java, C, C++ and other language-based service parts using an uniform approach: • the specification of monitoring services must be the same for all languages, • the implementations of the monitoring services may vary.

17. Summary • Workflow-based approach to applications still immature • Architecture for a monitoring infrastructure proposed • Integration of monitoring services into components • Global grid monitoring service • Performance analysis • Resuse of existing approaches for infrastructure and intra-component analysis • New level of inter-component analysis