1. Support for Parameter Study applications in the P-GRADE Portal Cevat Şener Dept. Of Computer Engineering, METU

2. General structure of a Parameter Study(PS) applications Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 SPMD parallelism: Single Program Multiple Data

3. AdvancedPS applications Algorithm 1 Cut inputintosmaller pieces Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2

4. AdvancedPS applications Algorithm 1 Cut inputintosmaller pieces Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 3 Aggregate result

5. PS applications inP-GRADE Portal 2.5 Complete workflow Files in the same LFC catalog (e.g. /grid/gilda/sipos/myinputs) Results produced in the same catalog

6. Initial input data Generator component(s) Generate orcut input intosmaller pieces Collector component(s) Aggregate result Advanced PS applications inP-GRADE Portal 2.5 Complete workflow Files in the same LFC catalog (e.g. /grid/gilda/sipos/myinputs) Results produced in the same catalog

7. Parallel execution inside a workflow node (SIMD/MIMD/MISD) • Parallel execution among workflow nodes(SIMD/MIMD/MISD) Multiple jobs run parallel Each job can be a parallel program Third level of parallelism Multiple instances of the same workflow process different data files • Parameter study execution of the workflow(SIMD)

8. Turning a WF into a PS By turning at least of the open input ports into a “PS Input port” the WF is turned into a Parameter Study

9. Turning a WF into a PS /grid/gilda/sipos/InputImages Image.0 Image.1 /grid/gilda/sipos/XCoordinates XCoordinate.0 XCoordinate.1 /grid/gilda/sipos/YCoordinates YCoordinate.0 YCoordinate.1 2 x 2 x 2 = 8 execution of the whole workflow /grid/gilda/sipos/Output ImagePart.0 ImagePart.1 . . .

10. PS workflows 4 x 3 normal workflow execution 1 PS workflow execution PS port: 4 instances of the input file PS port: 3 instances of the input file = This provides the 3rd level of parallelism resulting a very large demand for Grid resources

11. PS workflow execution

12. PS workflow detailed vieweWorkflows Total = Init + Submitted + Rescue + Error + Finished

13. Auto generator Pre defined program logic (static binary) Generates text files User can control text file content by templates and patterns Custom generator User provides generator program logic Useful to generate binary content (e.g. image files, audio files, …) Generators • Generate input files for parameter study workflows • Saves these files on SEs, register them with LFNs into the LFC catalog

14. Collector • Collects output files and perform collective operation on them. E.g. • Standard deviation • Average • Statistics • Evaluation and find the “best” result • … • User provides the program logic • Portal provides data transfer • Refer in your code to input files as local files • No need to use any Grid API in your code

15. Workflow management window showing Generator and Collector Collector phase is inactive until each eWorkflow has terminated

16. Workflow management window showing Generator and Collector The Output(s) of the collector(s) can be downloaded.

17. The generic PS workflow application First phase: Generators executed Second phase: eWorkflows executed Third phase: collectors executed

18. Introduction to the practicals

19. 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 15 Hands-on exerciseMatrix multiplication Multiplication job 1 2 3 4 5 6 7 8 12 2 1 3 1 1 1 3 3 3 Matrix2

20. Hands-on exerciseMatrix multiplication 1 2 3 4 5 6 7 8 X Auto generator 9 <= X <=15,step 3 Auto generator Multiplication job 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 12 1 2 3 4 5 6 7 8 15 2 1 3 1 1 1 3 3 3 Input files registered in the LFC catalog Matrix2 Output files registered in your the LFC catalog

21. Hands-on exerciseTurn the matrix operations WF into a PS AB[*,0]Tx AB[*,1] A B A * B A * B [ *, 0 ] A * B [ *, 1 ] A * B [ *, 0 ]T ( A * B [ *, 0 ]T ) * ( A * B [ *, 1 ] ) B