Workflow Management System based on Service Oriented Components for Grid Applications.

1. Workflow Management System based on Service Oriented Components for Grid Applications. Ju-Ho Choi Korea University, Seoul, Rep. of Korea

2. Contents • Introduction • Previous Work • Requirements • Grid Workflow System • Implementation • Conclusion Distributed Supercomputing Lab in Korea University

3. Introduction Workflow Control/Data flow Coordinated execution of multiple applications job1 job4 control job2 job5 init Input Data Divide job Output Proc.. display job3 job6 Abstract activity Composition of activities We need a PSE which providesa transparent view in the aspect of Grid user Requirements ? Grid Unreliable, Distributed, heterogeneous, huge-scaled computational and data resources Distributed Supercomputing Lab in Korea University

4. Previous Work – Grid/PSE Need a familiar, easy, and, transparent PSE for user Specific domain tool User domain application : Cactus, SCIRun, … Simple job, Not easy forcomplex job User Environment : MPI, NetSolve, Nimrod/G, …Grid Portal, GridPort, CoG, … Hotpage, ASC portal… Unfamiliar User environment Simple job request text-based output Platform : Virtualhomogeneous environment Grid middleware : Globus, OGSA, … Distributed resource management, information service, data management, single sign-on, Now beingChanging & Improved Grid Very Complicate !! Distributed, heterogeneous, unreliable Various hardware, software,different administration … Distributed Supercomputing Lab in Korea University

5. Previous Work – Workflow Model • Workflow patterns in Business Workflow • Basic Patterns • AND-Split/Merge, OR-Split/Merge, XOR-Split/Merge • Other Patterns • N out of M join, deferred choice, arbitrary cycle • Previous Grid Workflow System • GWSL, Triana, GridAnt, myGrid, Petrinet-based Grid Workflow (FhRG) • Execution of multiple App. on multiple resources • Composing of the existing applications • Providing Graphic User Interfaces • Simple Link element for control flow • Common patterns : AND, XOR, LOOP • arbitrary cycle (Petrinet-based) • Data link Node for File transfer Distributed Supercomputing Lab in Korea University

6. Requirements • Transparency with abstract workflow model • Complicate Grid infrastructure to user • Various Grid middleware, their change and standardization • Execution of the existing applications and making a new application without suffering from the details of Grid • Parallel/distributed programming, data transfer, fault handling • Advanced distributed/parallel workflow pattern • Previous patterns are too simple to describe Grid applications • Insufficient expressiveness of simple link elements • New patterns to support various parallelism for easy parallel and distributed programming • Integration of Control and Data flow Distributed Supercomputing Lab in Korea University

7. Requirements • Fault handling method • Faults from unreliable Grid resources • Three categories of fault • Hardware fault : network or host problems • Software fault : illegal access, user defined error, … • Resource utilization fault • Allocation or reservation problem without hardware fault caused by dynamic property of Grid • Efficient execution of workflow • Static/Dynamic information and Prediction • CPU speed, memory size, file system size, … • Load, available memory, usable bandwidth, … • Current Insufficient and unsatisfying dynamic information • semi-automatically or manually adapting dynamic Grid • Fault handling, dynamic reallocation, runtime modification Distributed Supercomputing Lab in Korea University

8. Grid Abstract workflow Activity Controller GUI Editing & running workflows Data Transfer Manager A workflow instance A workflow instance Workflow instance Activity Activity Activity Dynamic Manager Grid Workflow : Architecture Workflow Engine Resource Mapper Realizer Information Service Parser Activity Listener Monitor Generallizer Workflow Controller Resource management Allocator GWDL (Grid WorkflowDescription Language) Activity Repository File transfer Distributed Supercomputing Lab in Korea University

9. Grid Workflow : Abstract Workflow • Grid Workflow • Activity • Control/Data flow • Resource • exception • Activity • Function • Input/output data • Exception Data2 B2 EUserDefinedA Data1 A B C S S alloc Data3 EHardwareCrash EUserDefinedB (a) T Data2 A end A start B start B end C start C end Data1 (b) Data3 Distributed Supercomputing Lab in Korea University

10. GWDL • Grid Workflow Description Language Data Model Data type Exception type Fail Allocation Memory NotExistData Time NotAvailable Storage UserDefined Resource Model Execution environment Resource specification Activity Model PredefinedActivity Function Userdefined Activity Loose-Coupled allocation Input/output Data Exception Tight-Coupled notify timer Flow Model Data flow Nodes Control flow Group Node Life cycle Exception handling Activity Node Distributed Supercomputing Lab in Korea University

11. Grid Workflow : Exceptions • Predefined Exceptions • Efail : unknown Fail • ENotAvaliable : hardware fault • EAllocation : allocation problem • ETimeExpired : timer • ENoData : no usable input or output data • EMemory : insufficient memory • EStorage : insufficient storage • User defined Exceptions • For notification of software fault Distributed Supercomputing Lab in Korea University

12. Sequential flows (one thread) (m) conditions … X T X (a) (b) (c) (d) Parallel flows(multi threads) (n) n1 n1 M A M A P (g) (h) (e) (f) (i) (o) (j) (k) (l) m n1 * m n C W Q (i') ni A faultHandling Group Grid Workflow : Patterns (a) Sequence (b) XOR-Split (c) XOR-Join (d) Loop (e) Multi-Split(f) Multi-Join (g) AND-Split (h) AND-Join (i) AND-Loop symbol(i') AND-Loop description (j) Queue (k) Wait (l) Node copy (m) Explicit termination (n) Pause/Continue (o) Group node Distributed Supercomputing Lab in Korea University

13. n W Q n C n m Q C Workflow Model: Patterns • Composition of basic patterns • Pipeline Distributed Supercomputing Lab in Korea University

14. Workflow Model: Patterns • Composition of basic patterns • Static/Dynamic Data distribution • Static: AND-Split and AND-Join • Dynamic: AND-Loop, Multi-Split and Queue A M A Q … … static dynamic Distributed Supercomputing Lab in Korea University

15. Grid Workflow : Workflow Execution • Creating Concrete workflow • Mapping activities to resources • Statically or dynamically • Automatically or manually • Activity parameters • LC (Loose-coupled) type • For the existing applications • Executable file, addresses for input/output file transfer • TC (Tight-coupled) type • For making new application • Executable file, identification number • Addresses for Input/output data: file, variables • Address for communication with workflow engine • Runtime Modification • Pause and continue a workflow instance • To handle fault, unexpected situation, or manual reallocation. Distributed Supercomputing Lab in Korea University

16. Implementation • Our prototype • Workflow engine and graphic workflow editor : Java • XML parsing : Apache Xerces • TC type activity : C++ • Making TC type activity template codes: Apache Xalan • Fault detection • Throwing user defined exception • State notifications of TC type activity • Polling with Java CoG’s GRAM.ping() function or Timeout • Grid middleware • Globus Toolkit 3.2 : GRAM, MDS, GridFTP • Client library of Grid • Java CoG (Commodity of Grid) Distributed Supercomputing Lab in Korea University

17. Conclusion and Future work • Analysis of Requirements for Grid Applications • We proposed a new Grid workflow system • Abstract workflow model • Advanced workflow pattern for parallelism • Exception for handling fault on Grid • Run time modification • Future works • Scalability experiment of our Grid workflow system • Appling our system to various Grid applications Distributed Supercomputing Lab in Korea University

18. Thanks for your attention jhchoi@snoopy.korea.ac.kr Distributed Supercomputing Lab in Korea University