Main steps to gridify an Application Miklos Kozlovszky MTA SZTAKI m.kozlovszky@sztaki.hu

1. Main steps to gridify an ApplicationMiklos KozlovszkyMTA SZTAKIm.kozlovszky@sztaki.hu

2. Outline • This talk gives a high-level view of application development in Grids • Contents • Review of concepts: grids and grid applications • Types of Grid applications • Challenges to researchers who write applications • General steps of application gridification • Practical: Preparing and submitting a job based on a simple non-grid application. • Acknowledgements • Gergely Sipos, SZTAKI, Hungary • Mike Mineter, University of Edinburgh, UK, “Application Development and Aspects of gLite 3.0”, 2006 • Vladimir Dimitrov, IPP-BAS, Bulgaria, “Practical: Porting applications to the GILDA grid”, Introduction to Grid Computing, EGEE and Bulgarian Grid Initiatives Plovdiv, 2006 • Douglas Thain, The University of Notre Dame • GILDA team

3. What is a grid application? • Definition Software that interacts with grid services to achieve requirements that are specific to a particular VO or user.

4. Grids: a foundation for e-Research computers software Grid sensor nets instruments Shared data archives Diagram derived from Ian Foster’s slide colleagues Enabling a whole-system approach Collaborative research / engineering / public service …

5. The vital layer Where computer science meets the application communities! VO-specific developments built on higher-level tools and core services Makes Grid services useable by non-specialists Grids provide the compute and data storage resources Application Application toolkits, ….. Higher-level grid services (brokering,…) Basic Grid services:AA, job submission, info, … Production grids provide these core services.

6. Types of grid applications

7. Complexities of grid applications • Simple jobs – submitted to WMS to run in batch mode • Job invokes grid services • To read & write files on SE • Monitoring • For outbound connectivity (interactive jobs) • To manage metadata • … • Complex jobs • An environment controls multiple jobs on users’ behalf • High-level services • Portals with workflow • Software written for the VO (or by the user) • …

8. Invocation of applications • From the UI • Command Line Interfaces / Scripts • APIs • Higher level tools • From desktop Windows applications • Use Grids without awareness of them! • But gLite not (yet) fully supporting Windows (more info: http://jessica.trigrid.it/grid2win/) • From portals • For recurring tasks: “core grid services” as well as application layer • Accessible from any browser • Tailored to applications • Different portal solutions, and wide range of capabilities. Second part of this course: P-GRADE Portal

9. Multi-Grid P-GRADE Portal Different jobs of a workflow can be executed in different grids EGEE Gride.g. VOCE P-GRADE-Portal The portal can be connected to multiple grids UK NGS London Rome Athens

10. Characteristics of VOs • What is being shared? • resources of storage and/or compute cycles • software and/or data • Distinct groups of developers and of users? • Some VOs have distinct groups of developers and users… • Biomedical applications used by clinicians,…. • …. Some don’t • Physics application developers who share data but write own analyses • Effect: need to • hide complexity from the 1st type of VOs • expose functionality to 2nd type of Vos • many security issues

11. Different Goals for App. Development Engineering challenges increasing • I need resources for my research • I need richer functionality • MPI, parametric sweeps,… • Data and compute services together… • I provide an application for (y)our research • How!? • Pre-install executables ? • Hosting environment? • Share data • Use it via portal? • We provide applications for (y)our research • Also need: • Coordination of development • Standards • …

12. Challenges to researchers who write grid applications

13. Challenges Research software is often Created for one user: the developer Familiarity makes it useable Short-term goals: Used until papers are written and then discarded Grid applications are often used by a VO Without support from developer In new contexts and workflows • Grid application developers are • In a research environment • Yet their s/w must have: • Stability • Documentation • Usability • Extendability • i.e. Production quality • Need expertise in: • software engineering • application domain • grid computing

14. Consequences • Team work! • Engaged in world-wide initiatives – reuse, don’t make your own! Cross disciplines for solutions. • From research to production software: ~5 times the effort. • “80% of the time for last 10% of the functionality & reliability” • Standardisation is key • For re-use, for dynamic configuration of services,.. • Both for middleware and domain specific • Need to follow a deliberate development process • Waterfall? Rapid prototyping? • Requirements engineering, design, implementation, validation, deployment • Engaged with the user community

15. More about gLite services

16. gLite Grid Middleware Services Access API CLI Security Services Authorization Information & Monitoring Services Application Monitoring Information &Monitoring Auditing Authentication Data Management Workload Mgmt Services JobProvenance PackageManager MetadataCatalog File & ReplicaCatalog Accounting StorageElement DataMovement WorkloadManagement ComputingElement Connectivity

17. More about gLite services • During today the focus is on: • New functionality in gLite 3.0 Workload Management • Accessing data on SEs • Can have massive files, too big to copy • How to access these? • Management of metadata • May have many thousands of files • Need to access and re-use based on characteristics… more than by their logical file names. • Monitoring of applications • May be running many long jobs • What’s happening?!

18. Workload Management System • Helps the user accessing computing resources • resource brokering • management of input and output • management of complex workflows • Support for MPI job even if the file system is not shared between CE and Worker Nodes (WN) – easy JDL extensions • Web Service interface via WMProxy

19. WMProxy WMProxy is a SOAP Web service providing access to the Workload Management System (WMS) Job characteristics specified via JDL jobRegister create id map to local user and create job dir register to L&B return id to user input files transfer jobStart register sub-jobs to L&B map to local user and create sub-job dir’s unpack sub-job files deliver jobs to WM

20. Complex Workflows Direct Acyclic Graph (DAG) is a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs A Collection is a group of jobs with no dependencies basically a collection of JDL’s A Parametric job is a job having one or more attributes in the JDL that vary their values according to parameters Using compound jobs it is possible to have one shot submission of a (possibly very large, up to thousands) group of jobs Submission time reduction Single call to WMProxy server Single Authentication and Authorization process Sharing of files between jobs Availability of both a single Job Id to manage the group as a whole and an Id for each single job in the group nodeA nodeB nodeC nodeE nodeD

21. Basic taskswhile Porting applications to the Grid • Developing a non-grid application (or inheriting and updating an ancient one); • Go/no-Go decision about gridification • Is it suitable for the Grid environment? • “Cost/profit” analysis / feasibility study Typical Questions Groups: • Current structure of the application • Dependencies of the application • Available resources (manpower, knowledge, etc.) • Requirements for the gridified application • Expected impact of gridification • Requirements for the grid infrastructure More info: Application Description Template http://www.lpds.sztaki.hu/gasuc/?m=4

22. Basic taskswhile Porting applications to the Grid (contd.) • Grid environment access • Requesting Certificates / VO membership • Accessing Grid environment • Appropriate VO UI machine account for command line • Portal GUI account; • Executing, Testing and Debugging the application; • Testing the non-grid application (debugging in Grid environment is a hard task), creating use cases for single (non-grid) runs; • Constructing the job suite – JDL (Job Description Language) files, executables, auxiliary scripts and input/output data files; • Submitting the job to the Grid as small-scale pilot application;

23. Basic tasks while Portingapplicationsto the Grid (contd.) • Executing, Testing and Debugging the pilot application; • IF something goes wrong THEN GOTO4; • IF everything seems to work THEN increase the scale of the application (increase problem size, amount of used resources); • Optimizing the grid application;

24. Porting legacy code applications Code from the past, maintained because it works Often supports business critical functions Not Grid enabled What to do with legacy codes in service Grids? • Bin them and reimplement them as grid services • Reengineer them  who knows the source code? • Port them onto the Grid with minimum user effort

25. Porting legacy code applications with GEMLCA GEMLCA – Grid Execution Management for Legacy Code Architecture Objectives • To deploy legacy code applications as Grid services without reengineering the original code and minimal user effort • To support the development and execution of “legacy code” grid service workflows • To make these functions available from a Web Portal GEMLCA GEMLCAP-GRADE Portal

26. The GEMLCA-view of a legacy code Any code that correspond to the following model can be exposed as Grid service by GEMLCA: Legacy code Input data (files, command line params, env. vars) Output data (files) . . . Perform computation Call shared libraries Communicate over the network Query databases

27. Legacy Application example Workflow to analyse road traffic Manhattan road network generator Traffic simulators Analyser

28. Another Legacy Application example Molecular Dynamics Study of Water Penetration in Staphylococcal Nuclease using CHARMm • Analysis of several production runs with different parameters following a common heating and equilibrium phase

29. Practical tools @ gridification On-Line Monitoring • Mercury monitor • to debug/optimize and visualize parallel jobs. • both at the workflow and job levels

30. Practical tools @ gridification (contd.) Hiding Grid remote storage system from a legacy application • Parrot • a handy tool for attaching old programs to new storage systems • EGEE (gLite module) Data Access: GFAL, LFN, GUID, SRM, RFIO, DCAP, and LFC • does not require any special privileges, any recompiling to existing programs • For example, an anonymous FTP service is made available to vi like so: parrot vi /anonftp/ftp.cs.wisc.edu/RoadMap Or example with gsiftp: $./parrot app_name /gsiftp/<gsiftp usr without gsiftp://> (More info: http://www.cse.nd.edu/~ccl/software/manuals/parrot.html)

31. Common problems and obstacles • The candidate-applications for porting usually are huge and complex. • Some of them use low-level network functions and/or parallel execution features of a specific non-grid environment. • Usage of non-standard or proprietary communication protocols. • The complete source code might not be available, might not be well documented or its “out-of-host” usage is restricted by a license agreement.

32. Common obstacles (continued) • The application might be written in many different programming languages – C, C++, C#, Java, FORTRAN etc. or even mixture of them. • Applications may depend on third-party libraries or executables which are not available by default on some Grid worker nodes. • Some application features could cause unintentional violation of Grid Acceptable Use Policies (Grid AUP). • Furthermore, the application can have hidden security weakness which will be very dangerous in case of remote Grid job execution.

33. Common obstacles (continued) • Some applications are pre-compiled or optimized for using on a machine with particular processor(s) only – Intel, AMD, in 32-bit or 64-bit mode, etc. But the Grid is heterogeneous! • The application may contain serious bugs, which have never been detected while running in a non-grid environment. • Finally, the formal procedure for accepting a new application to be ported to a Grid for production or even experimental purpose is not simple. Therefore, the porting of an arbitrary application to Grid could be very long, difficult and expensive process!

34. GASUC Grid Application Support Centre • AIM of the Center • Provides assistance to current and future grid users and grid application developers to port legacy algorithms and applications onto large-scale grid infrastructures. • Creates a bridge between Academy/Industry and Grid • Supports and develops Grid applications for Grid communities • Disseminates knowledge of concepts and tools used on Globus Toolkit 2, Globus Toolkit 4, LCG-2 and gLite grid environments. • Established in January 2007 • Full EGEE support • Announced officially in EGEE • Follow up in EGEE3 • Cost model • For non-commercial applications: free http://www.lpds.sztaki.hu/gasuc/

35. Grid Application Development Support Model • 8 steps support model • Contact phase • Pre-selection phase • Analysis phase • Planning phase • Prototyping phase • Testing phase • Execution phase • Dissemination and feedback phase

36. In the practical session • The application called MatrixDemo will be ported and executed in SGDEMO grid environment. (The program is borrowed from the “EGEE summer school” at MTA SZTAKI, 2006.) MatrixDemo is written in C programming language SGDEMO environment (gLite based) is supporting C, so porting the C or C++ programs is easy … hopefully

37. MatrixDemo program • MatrixDemo program performs some matrix operations – inverting, multiplying, etc. • Usage: • MatrixDemo has command line interface which accepts several arguments. Starting the program without any argument will display a short help. • Example: MatrixDemo I V This will Invert (I) the matrix defined in the file named INPUT1 and will store the result in the file OUTPUT with verbose details (V).

38. MatrixDemo program (continued) • Prerequisites: • File MatrixDemo.c– the source code of the program. • Files INPUT1 and INPUT2– they contain matrix data in the following text format: rows, columns, cell1, cell2, cell3 … Where rows is an integer representing the number of rows. columns represents number of columns, and cell1, cell2 etc. are the cells of the matrix, floating point numbers separated by commas (,). • A standard C compiler and linker. In this case we will use GNU C (gcc) already installed. • File MatrixDemo.jdl– a prepared JDL (Job Description Language) file.