Improving Separation of Concerns in the Development of Scientific Applications

1. Improving Separation of Concerns in the Development of Scientific Applications S. Masoud Sadjadi, Juan Martinez, Tatiana Soldo, Luis Atencio Rosa M. Badia and Jorge Ejarque Florida International University Miami, Florida, U.S.A. Barcelona Supercomputing Center Barcelona, Spain

2. Outline • Motivation • Background • GRID superscalar • TRAP/J • Case Study: Matmul • Transparent Grid Enablement • Results • Related Work • Conclusions • Future Work

3. Motivation • High performance computing (HPC) is gaining popularity in solving complex scientific applications. • The current HPC programming standards (e.g., MPI, Open MP, and Grid Computing toolkits) are not targeted for scientists to develop their scientific applications • For example, Weather Research and Forecast is 200,000+ lines of code in FORTRAN 90 that uses MPI and Open MP • This lack of separation of concerns has resulted in scientific applications with rigid code, which entangles non-functional concerns (e.g., the parallel code and the platform-specific code) into functional concerns (i.e., the core business logic). • Effectively, this tangled code hinders the maintenance and evolution of these applications.

4. Transparent Grid Enablement: Goals To separates the task of developing the business logic of a scientific application from the task of improving its performance. To increase the level of modularity of code by separating crosscutting parallel programming related code from the business logic of the scientific application. To develop an automatic (or semi-automatic) Grid enablement process that requires no manual modifications to the business logic of the scientific application and hence “transparent” to the scientists and their sequential code. TGE achieves this goal by integrating two existing software tools, namely, TRAP/J and GRID superscalar.

5. Background: GRID superscalar • Inspired by the superscalar processors, GRID superscalar provides an easy programming paradigm for developing parallel programs. • Similar to superscalar processors that provide out-of-order and parallel execution of machine instructions by bookkeeping their dependencies, GRID superscalar provides parallelism to the functions of a program written in a high-level programming language such as Java. • GRID superscalar enables the development of applications for a computational Grid by hiding details of job deployment, scheduling, and dependencies and enables the exploitation of the concurrency of these applications at runtime. • In TGE, actual gridificationof the application is obtained through GRID superscalar and the GRID superscalar calls are woven transparently into the scientific application using TRAP/J.

6. Background: GRID superscalar

7. Transparent Shaping is a programming model that enables software adaptation through interception and redirection of interactions among different part of a software system without the need to manually modify the code. TRAP (Transparent Reflective Aspect Programming) is an extension of Transparent Shaping for object-oriented programming languages. TRAP/J is a realization of TRAP in Java that enables static and dynamic adaptation in Java programs at startup and runtime, respectively. Other realizations: TRAP/C++, TRAP/BPEL, and TRAP.NET. Background: TS, TRAP, and TRAP/J

8. Background: TRAP/J • Using TRAP/J, we can insert generic hooks/interceptors at important/sensitive points in a Java program. • Later, we can use these hooks to intercept and redirect the flow of control to a new code. Flow of Control in the Original Application Flow of Control in the Adapt-Ready Application Invoke Original Task Invoke Original Task TRAP/J Adapt? Yes No Execute the Original Task Execute the Original Task Execute the New Task

9. Background: TRAP/J • TRAP/J allows crosscutting concerns to be separated from the functional logic not only at development time, but also at run time. Before TRAP/J After TRAP/J

10. Outline Motivation Background GRID superscalar TRAP/J Case Study: Matmul Transparent Grid Enablement Results Related Work Conclusions Future Work

11. Case Study: Matmul Matmul is a simple matrix multiplication program written in Java. It uses a sequential matrix multiplication algorithm, which computes C = A.B, where A, B, and C are matrices of size NxN. This typical “row by column” sequential algorithm involves O(N3) operations. C B A = X

12. Transparent Grid Enablement C00=C00+A00*B00 C00=C00+A01*B10 C01=C01+A00*B01 C01=C01+A01*B11 C10=C10+A10*B00 C10=C10+A11*B10 C11=C11+A10*B01 C11=C11+A11*B11

13. A B C = X Transparent Grid Enablement Matmul Sequential Application New parallel approach HPC Expert Scientist TRAP/J GRID Superscalar Adapt-Ready/Grid-Enabled Application Startup-time adaptation B C A = X Finer-Grain Parallelism: Adaptive code for maximum parallelism of 9. Coarser-Grain Parallelism: Adaptive code for maximum parallelism of 4.

14. Transparent Grid Enablement MultiplyMatrices delegate MatmulApplication TRAP/J Adapt-ready Matmul Application Matmul IDLmultiply_acc() GRID superscalar multiply_acc() multiply_acc() multiply_acc()

15. Transparent Grid Enablement Sequential matrix multiplication public static void main(String[] args) { . . . Multiply_Matrices(size, args[1], args[2], args[3]); } public static void Multiply_Matrices(int size, fileC, fileA, fileB) { Block A = new Block(fileA, size); Block B = new Block(fileB, size); Block C = new Block(size); C.Multiply(A,B); C.blockToDisk(fileC); } Matmul IDL interface MATMUL { void multiply_acc(inout File f3, in File f1, in File f2, in int size); };

16. Transparent Grid Enablement Multiply Matrices delegate class • public class Matmul_Del implements DelegateInterface • { public static void Multiply_Matrices(int size, fileC, fileA, fileB) • { • GSMaster.On(); • for(int i=0;i<num_of_pieces;i++) • { • for(int j=0; j<num_of_pieces;j++) • { • for(int k=0; k<num_of_pieces;k++) • { //Method sent to each node in grid • Matmul.multiply_acc(C[i][j], A[i][k],B[k][j],…); • } • } • } • GSMaster.Off(); • MergeFiles(); //Merge files after computation …

17. Outline Motivation Background GRID superscalar TRAP/J Case Study: Matmul Transparent Grid Enablement Results Related Work Conclusions Future Work

18. Results Initially we got:

19. Results More disappointment!

20. Results Initially we got:

21. Results: Optimization • Problem in GS: GS_Off() • freeing resources and deleting temporary files after finishing the calls to the grid methods. • Since all the data is distributed along the nodes, there will be the need for cleanup that wastes extra time. • Solution: Avoiding the cleanup! ;) • Optimizing the use of GridFTP • TCP has a slow start • You can instruct GridFTP to open more TCP connections with bigger starting window to compensate for the slow start of TCP.

22. <?xml version="1.0" encoding="UTF-8"?><project isSimple="yes" masterBandwidth="100000" masterBuildScript="" masterInstallDir="/home/lion-e/globus2/matmul_java_master" masterName="la-blade-01.cs.fiu.edu" masterSourceDir="/a/lion.cs.fiu.edu./disk/216/e/globus2/matmul_java_master" name="Matmul" workerBuildScript="" workerSourceDir="/a/lion.cs.fiu.edu./disk/216/e/globus2/matmul_java_worker"><disks><disk name="_MasterDisk_"/><disk name="_WorkingDisk_la-blade-02_cs_fiu_edu_"/><disk name="_WorkingDisk_la-blade-03_cs_fiu_edu_"/></disks><directories><directory disk="_MasterDisk_" isWorkingPath="yes" path="/home/lion-e/globus2/matmul_java_master"/></directories><workers><worker Arch="" GFlops="1.0" LimitOfJobs="1" Mem="16" NCPUs="1" NetKbps="100000" OpSys="" Queue="none" Quota="0" deploymentStatus="deployed" installDir="/home/lion-e/globus2/matmul_java_worker" name="la-blade-02.cs.fiu.edu"><directories><directory disk="_WorkingDisk_la-blade-02_cs_fiu_edu_" isWorkingPath="yes" path="/home/lion-e/globus2/matmul_java_worker"/></directories></worker> <?xml version="1.0" encoding="UTF-8"?> <project isSimple="yes" masterBandwidth="100000" masterBuildScript="" masterInstallDir="/home/lion-e/globus2/matmul_java_master" masterName="la-blade-01.cs.fiu.edu" masterSourceDir="/a/lion.cs.fiu.edu./disk/216/e/globus2/matmul_java_master" name="Matmul" workerBuildScript="" workerSourceDir="/a/lion.cs.fiu.edu./disk/216/e/globus2/matmul_java_worker"> <disks> <disk name="_MasterDisk_"/> <disk name="_WorkingDisk_la-blade"/> <disk name="_sharedDisk_la-blade"/> </disks> <directories> <directory disk="_MasterDisk_" isWorkingPath="yes" path="/home/lion-e/globus2/matmul_java_master"/> </directories> <workers> <worker Arch="" GFlops="1.0" LimitOfJobs="1" Mem="16" NCPUs="1" NetKbps="100000" OpSys="" Queue="none" Quota="0" deploymentStatus="deployed" installDir="/home/lion-e/globus2/matmul_java_worker" name="la-blade-01.cs.fiu.edu"> <directories> <directory disk="_WorkingDisk_la-blade" isWorkingPath="yes" path="/home/lion-e/globus2/matmul_java_worker"/> <directory path="shared_path" disk="_SharedDisk_la-blade" isWorkingPath="no"/> </directories> </worker> Results: Optimization Using Network File System

23. Results After the optimizations we got:

24. Results In the Speedup graph shown below, we see that our approach performs almost twice better than the sequential one.

25. Outline Motivation Background GRID superscalar TRAP/J Case Study: Matmul Transparent Grid Enablement Results Related Work Conclusions Future Work

26. Related Work • Satin • is a Java based programming model for the Grid which allows explicit expression of divide-and-conquer parallelism. • Satin uses marker interfaces to indicate that certain invocation methods need to be considered for potentially parallel (spawned) execution. • Synchronization is also explicitly marked whenever it is required to wait for the results of parallel method invocations. • Higher-Order Components (HOCs) • is a component-oriented approach based on a master-worker schema. • HOCs express recurring patterns of parallelism that are provided to the user as program building blocks, pre-packaged with distributed implementations.

27. Related Work • ASSIST • is a programming environment aimed at providing parallel programmers with user-friendly, efficient, portable, fast ways of implementing parallel applications. • It includes a skeleton based parallel programming language (ASSISTcl, cl stands for coordination language) and a set of compiling tools and run time libraries. • The ensemble allows parallel programs written using ASSISTcl to be seamlessly run on top of workstation networks supporting POSIX and ACE (the Adaptive Communication Environment, which is an extern, open source library used within the ASSISTcl run time support).

28. ProActive • is a Java GRID middleware library for parallel, distributed and multi-threaded computing. • With a reduced set of simple primitives, ProActive provides a comprehensive API to simplify the programming of Grid Computing applications: distributed on Local Area Network (LAN), on clusters of workstations, or on Internet GRIDs. • ProActive is only made of standard Java classes, and requires no changes to the Java Virtual Machine, no preprocessing or compiler modification, leaving programmers to write standard Java code. • Architected with interception and reflection, the library is itself extensible, making the system open for adaptations and optimizations. • Current implementation is focusing of the CoreGRID NoE specification of the Grid Component Model (GCM).

29. Related Work • None of the above mentioned approaches provide an explicit separation of concerns identifying separate tasks for scientist developers and HPC expert developers. • TGE can be extended to use these works instead or in complement to GRID superscalar and can be used as an enabler for supporting interoperation among the above mentioned approaches.

30. Outline Motivation Background GRID superscalar TRAP/J Case Study: Matmul Transparent Grid Enablement Results Related Work Conclusions Future Work

31. Conclusions In this work, we have presented an innovative approach to transparent grid-enablement of scientific applications. We achieved this goal by combining two of our previously developed toolkits, namely, GRID superscalar and TRAP/J. Although this work is still in its preliminary stage, we were able to show its effectiveness through a simple case study. We acknowledge that it may not be easy (and even may be impossible) in some applications to separate the code parallelism from the business logic of the application; however, there are many existing applications that can benefit from TGE.

32. Future Work • We are applying TGE to a real case study, namely, hurricane mitigaiton simulation and visualization applications. • Currently, TGE support static adaptation at startup time. We plan to extend it to support dynamic adaptation. • Currently, TGE supports self-configuration and self-optimization. We plan to extend TGE to support other autonomic behavior including self-healing and self-protection. • We plan to extend the self-optimization of self-configuration of TGE so that it can take advantage of more worker nodes becoming available during runtime.

33. Acknowledgements • This work was supported in part by IBM (SUR and Student Support awards), the National Science Foundation (grants OCI-0636031, REU-0552555, and HRD-0317692), the Spanish CICYT (contract TIN2004-07739-CO2-01), and the BSC-IBM Master R&D Collaboration agreement. This work is part of the Latin American Grid (LA Grid) project.

34. References • [1] S. Masoud Sadjadi, J. Martinez, T. Soldo, L. Atencio, R. M. Badia, and J. Ejarque. Improving separation of concerns in the development of scientific applications. In Proceedings of The Nineteenth International Conference on Software Engineering and Knowledge Engineering (SEKE'2007), Boston, USA, July 2007. • [2] S. Masoud Sadjadi, Philip K. McKinley, and Betty H.C. Cheng. Transparent shaping of existing software to support pervasive and autonomic computing. In Proceedings of the first Workshop on the Design and Evolution of Autonomic Application Software 2005 (DEAS'05), in conjunction with ICSE 2005, St. Louis, Missouri, May 2005. • [3] S. Masoud Sadjadi. Transparent Shaping of Existing Software to Support Pervasive and Autonomic Computing. A Dissertation submitted to Michigan State University, 2004. • [4] S. Masoud Sadjadi, Philip K. McKinley, Betty H.C. Cheng, and R.E. Kurt Stirewalt. TRAP/J: Transparent generation of adaptable Java programs. In Proceedings of the International Symposium on Distributed Objects and Applications (DOA'04), Agia Napa, Cyprus, October 2004. • [5] Rosa M. Badia, Raül Sirvent, Jesus Labarta, and Josep M. Perez. Programming the GRID: An Imperative Language Based Approach. book chapter in Engineering the Grid, Section 4, Chapter 12 , January 2006. • [6] Philip K. McKinley, S. Masoud Sadjadi, Eric P. Kasten and Betty H.C. Chen. Composing Adaptive Software. Computer. July 2004, pages 56-64.

35. Questions/Comments • Contact Information: S. Masoud Sadjadi (sadjadi@cs.fiu.edu) Autonomic Computing Research Lab. (ACRL) School of Computing and Information Sciences (SCIS) Florida International University (FIU) • TGE, TRAP/J, TRAP.NET, TRAP/BPEL, ACT/J, and other Transparent Shaping tools can be downloaded from http://acrl.cis.fiu.edu/