WS-VLAM Tutorial: Part III (part3): Learn how to WS-VLAM JAVA API

WS-VLAM Tutorial: Part III (part3): Learn how to WS-VLAM JAVA API Adam Belloum

Outline • Part I: Hands on the User Graphical Interface • Part III: Port a Legacy application to WS-VLAM • Part III: Learn how to use WS-VLAM API • Part I: Hands on the User Graphical Interface • Part III: Port a Legacy application to WS-VLAM • Part III: Learn how to use WS-VLAM API • Notation (Title background): • Generic step • Tutorial specific step

Outline (Part III): Learn how to use WS-VLAM API Step 1: Define an Interface for your component Step 2: Create a description for your component Step 3: Define your component dependencies Step 4: JAVA Tutorial (create a WS-VLAM JAVA components) Step 6: C/C++ Tutorial (create a WS-VLAM C/C++ components) Step 5: PYTHON Tutorial (create a WS-VLAM PYTHON components) Step 6: Deploy the component/dependencies lib Step 7: Compose a workflow using the component Step 8: execute the workflow / monitor

Step 1: Define the component interface • The Interface defines how WS-VLAM will interact with your components • Parameter Interface: is use to interact at runtime with your module • Port interface: is used to stage in/out the data consumed/produced by your module • Port in allows to stage in the consumed data • Port out allows to stage out the produced data

The file reader “fileReader”opens a file specified as a URI using the parameter Interface “filename” and sends the content through an output port “port_out”. • fileReader support many protocols: GSIFTP, ftp, http, file Step 1: Example File Reader

To create a description of your component: use CLAM workflow component configuration tool Step 2.1: Create a description • Action: • Click on CLAM Button in the WS-VLAM menubar

CLAM workflow component configuration tool allows you to define: the name of the component, default host, ports, parameters, classification, and category Step 2.2: Create a description • Action: • Enter the name of the component • Entre the name of the default host “fs2.das3.science.uva.nl”

CLAM workflow component configuration tool allows you to define: the name of the component, default host, ports, parameters, classification, and category Step 2.3: Create a description • Action: • Click on ports tab • Click + to add new input or output port • Type the name of the ports exactly as you will define them the source code.

To add a port a pop up window appears “Add input Port” or “Add output port” Step 2.4: Create a description • Action: • Type the name of the ports exactly as you did it in source code • Specify the type of the port either a simple data type, a file type, or Complex data type (see documentation for more details)

Create a description of your component: use CLAM workflow component configuration tool Step 2.5: Create a description • Action: • Click on save to created the description file “component-name.xml”

Create a description of your component: use CLAM workflow component configuration tool Step 2.6: Create a description • Action: • You will see in the filed file path the full path to the create Component-name

Create a description of fileReader: use CLAM workflow component configuration tool • The FileReader has • One output: port-name = “port_out” and Type=“file” • One parameter parameter-name = “filename” and Type=“String” Step 2.7: Example File Reader • Action: • Perform the tasks described in step 2.1 to step 2.6 to create the fileReader.xml

Step 3.1: Define the component dependencies • Dependencies are libraries & auxiliary software that are required by your component but are notavailable on standard Operating systems • You have to list the name of these libraries & auxiliary software in a file called “dependencies” • Every libraries & auxiliary software listed in the dependencies files have to be packaged as a “pseudo-module”

Step 3.2: Create a pseudo-module # ############ NOTE #################### # In the profile.sh export all the environment # variables needed to execute your package # # bash environment # # export VARIABLE_NAME=VARIABLE_VALUE # # Add your VARIABLES bellow the line ###################################### • A pseudo module is just a simple software package for which you need to create a profile.sh in the root directory • Using your favorite text editor created a file call profile.sh

Step 4.1: YOUR JAVA Module • From a developer’s point of view your components are applications that use a special library provided by the WS-VLAM framework: the “vlport” library

Step 4.2: JAVA modules (pre-requisite) • Before you build the jwrapper make sure that the SWIG toolkit and Python are installed in your system. • The SWIG version should be higher then 1.3.19. • To compile the java wrapper • the environment variable VLAM_INSTALL must be set. • Just type ‘‘make dist‘‘. • You will ﬁnd the library in dist/lib subdirectory. • The wrapper consists of an ”adapter” shared library with the name ” jwrapper.jar” and ”jvlport.jar”

Step 4.3: Extend your JAVA code • Initialize • import nl.wtcw.vlamg.jwrapper. ∗; • Create a reference to the input port • portIRef= VLIPort.getInstance("portIn"); • Create a reference to the output port • VLOPortportORef =VLOPort.getInstance("portOut"); • Note: should provide the same names for the ports as speciﬁed at the initialization stag

Step 4.4: API for reading from input port • void readInt() • reads an integer number from an input port; • void readFloat() • reads a ﬂoat number from an input port; • void readDouble() • reads a double precision number from an input port; • void readString() • reads a string from an input por

Step 4.4: API for reading from input port • void hardSync () • prepares an output buﬀer and waits until it is sent to network. • void readBytes (byte[ data)] • reads a byte array (raw data) from an input port. Array must be prepared before; • void readShort () • reads a short number from an input port; • void readLong () • reads a long integer number from an input port;

Step 4.5: API for writing to output port • void writeBytes (byte[ data)] • writes a byte array to an output port • void writeShort (short number) • writes a short to an output port • void writeLong (long number) • writes a long integer to an output port • void writeInt (int number) • writes an integer to an output port

Step 4.5: API for writing to output port • void writeDouble (double number) • writes a double precision number to an output port; • void writeString (String str) • writes a string to an output port; • void flush () • prepares an output buffer to be sent over network (flush internal library buffer)

Step 4.6: API for reading/writing to/from parameter • getParameter (String param−id) • Reads the value of “param−id” • setParameter (String param−id, String para−value) • Writes the value of “para−value” to “param−id”

Step 4.7: JAVA component • // YOU MUST IMPORT THE JWRAPPER PACKAGE • import nl.wtcw.vlamg.jwrapper. ∗; • VLOPort outputPortRef = null; • VLIPort inputPortRef = null; • … • // IT IS ASSUMED THAT REFERENCES TO THE PORTS ARE ALREADY TAKEN • try { • double rndn = 12345.12345; • // WRITING TO THE WS −VLAM • Output Port outputPortRef.writeDouble(rndn); ... • // READ FROM THE WS −VLAM INPUT PORT • rndn = inputPortRef.readDouble(); • ... • } • catch(VLAPPException e){ • // HANDLE EXCEPTION ... • } • ... • XXXXXXXX:

Step 4.8: JAVA code of the File Reader • import nl.wtcw.vlamg.jvlport.VLAppFactory; • import nl.wtcw.vlamg.jvlport.Vlport; • import nl.wtcw.vlamg.jvlport.OutputStream; • import java.io.*; • /** • * Created by IntelliJ IDEA. • * User: dvasunin • * Date: Jul 29, 2005 • * Time: 3:49:34 PM • * To change this template use File | Settings | File Templates. • */ • public class JFileReader { • static public void main(String args[]){ • String line = null; • String[] outputPorts; • try { • outputPorts = new String[] {"port_out"}; • Vlport vlport = • (vlport.getOutputPort("port_out"))); • VLAppFactory.activate(null, outputPorts); • PrintWriter pw = • new PrintWriter( • new OutputStreamBufferedReader • reader = new BufferedReader( • new FileReader("/etc/passwd")); • while((line = reader.readLine()) != null){ • System.out.println(line); • pw.println(line); • } • pw.flush(); • } catch (IOException e) { • e.printStackTrace(); • } finally{ • VLAppFactory.deactivate(); • System.exit(0); • } • } • } • The Example:

Step 4.9: build your JAVA component • build.xml for FileReader (½) • <project name="WSVLAM-java-modules" default="dist" basedir="."> • <description> • Project : VLE-wsvlam • Component : JFileReader source • Info : • Ant file to call 'make'. • Builds wsvlam java modules. • This component can be used as standalone. • In that case the Makefile should be called directly. • </description> • <path id="compile.classpath"/> • <property name="src.home" value="${basedir}/src"/> • <target name="init" description="making build directory"> • <mkdir dir="build/java"/> • </target>

Step 4.9: build your JAVA component • build.xml for FileReader (2/2) • <target name="build" depends="init" description="compile the source "> •  • <javac srcdir="src" destdir="build/java"> • <classpath refid="compile.classpath"/> • </javac> • </target> •  • <target name="dist" depends="build" description="generate the distribution"> • <copy todir="dist/JFileReader/bin"> • <fileset dir="${basedir}/build/java" excludes="**/*.java"/> • </copy> • <copy todir="dist/JFileReader"> • <fileset dir="${basedir}" includes="JFileReader.xml,main.sh"/> • </copy> • </target> • </project>

Step 4.10: package your FileReader (1) Compile the fileReader $ cd fileReader $ ant # this will created java classes in # fitleReader/bin (2) Create a tar file of your component: $ tar cf fileReader.tar fileReader (3) Compress the tar file you have created $ gzip fileReader.tar • Create a directory with the fileReader name as the component • Copy the files you have created: • build.xml(step 4.9) • fileReader.xml(step 2.7) • Copy any other files or directory related to your component: • src/fileReader.java(step 4.8)

Step 5: Deploy the fileReader • # On a machine where both GT4 & WS-VLAM • # runtime system are installed • # Create in your HOME directory called “.wsvlam” • (1) save fileReader.tar.gz to the default location • $ mkdir .wsvlam • $ cd .wsvlam/ • $ mkdir modules • $ cd modules • #untar the component tar file you have created • # in step 4 • (2) untar fileReader.tar.gz • $ tar –zxvf fileReader.gz • Test mode: you can test your component • Perform the steps described in the blue box • Shared mode: you can share your module with other users: • by moving the tar file you have created in step 4 to a component repository

Step 6: Complete examples in JAVA components • Complete examples in JAVA can be found here: • http://www.science.uva.nl/~gvlam/wsvlam/Documentation/wsvlam-module-examples.tar.gz • Download wsvlam-module-examples.tar.gzuntar the file and save the examples in a directory: $HOME/.wsvlam/modules (see step 5)

Step ?: Template for component in JAVA • <?xml version="1.0" encoding="ISO-8859-1" ?> • <config> • <ports> •  • </ports> • <java PropertyName="PropertyValue"> •  •  • <main> • your-java-main-class • </main> • <options> •  • <option> -server </option> • </options> •  • <classpath> /path/to/something.jar </classpath> • </java> • </config> • Is this needed • ???????? • Dmitry do we still need this??????

In the component list window you will see the created FileREader components as well as other components Step 6.1: Compose a workflow fileReader • Action: • Click on the fileReader in the list and drag and drop in the composition panel

An instance of fileReader will be instantiated Step 6.2: Compose a workflow using LA • Action: • Click on the fileReader in the list and drag and drop in the composition panel

Instantiate the fileWriter component: • This component will move the file produced by fileReader to any remote location the user specifies Step 6.3: Compose a workflow using LA • Action: • Click on the WriterReader in the list and drag and drop in the composition panel

Create a communication channel: • This component will move the file produced by FileReader to any remote location the user specifies Step 6.4: Compose a workflow using LA • Action: • Create a communication channel by dragging between an output port (read square) and input port (blue square)

In the step you should check if all the default values of the parameters of the workflow components composing your workflow are set correctly Step 7.1: customize the composed workflow • Action: • Click on the fileReader in composition panel • In the property window select the parameter tab • Type in the URI to the file you want stage in FileReader support many protocols: GSIFTP, http, file, ftp

In the step you should check if all the default values of the parameters of the workflow components composing your workflow are set correctly Step 7.1: customize the composed workflow • Action: • Click on the fileWriter in composition panel • In the property window select the parameter tab • Type in the URI to the file you want stage in FileReader support many protocols: GSIFTP, http, file, ftp

Input file: fileReader parameter • fs2.das2.science.uva.nl: Type in the following URI for parameter in each fileReader • gsiftp://fs2.das.3science.uva.nl/home2/<user-account>/<filename-you-want-to-cp> • output file: fileWriter parameter • fs0.das3.cs.vu.nl: Type in the following URI for parameter in each fileReader • gsiftp://fs0.das3.cs.vu.nl/home0/<user-account>/<filename-you-want-to-created> Step 7.1: customize the composed workflow

Now you are ready to execute the omnimatch workflow, which automatically stage in the data file needed for the omnimatch application, and stage out the result files Step 8: execute the composed workflow • Action: • Click on the run button

Some References • Science Z Zhao, A Belloum, M Bubak Editorial: Special section on workflow systems and applications in e-Future Generation Computer Systems 25 (5), 525-527 • A. Wibisono, D. Vasyunin, V. Korkhov, AS.Z. Belloum WS-VLAM: a GT4 based workflow management system , The 2nd International Workshop on Scientific Workflows, In conjunction with ICCS 2007, Beijing, China, 2007 • A.S.Z. Belloum, V. Korkhov, S. Koulouzis, M. A Inda, and M. BubakCollaborative e-Science experiments: from scientific workflow to knowledge sharing JULY/AUGUST, IEEE Internet Computing, 2011 • IlkayAltintas, Manish Kumar Anand, Daniel Crawl, Shawn Bowers, Adam Belloum, Paolo Missier, Bertram Ludascher, Carole A. Goble, Peter M.A. Sloot, Understanding Collaborative Studies Through Interoperable Workflow Provenance, IPAW2010, Troy, NY, USA

Some References • A. Belloum, Z. Zhao, and M. Bubak Workflow systems and applications , Future Generation Comp. Syst. 25 (5): 525-527 (2009) • Z. Zhao, A.S.Z. Belloum, et al., Distributed execution of aggregated multi domain workflows using an agent framework The 1st IEEE International Workshop on Scientific Workflows, Salt Lake City, U.SA, 2007 • Zhiming Zhao, Adam Belloum, Cees De Laat, Pieter Adriaans, Bob Hertzberger Using Jade agent framework to prototype an e-Science workflow bus Authors Cluster Computing and the Grid, 2007. CCGRID 2007

WS-VLAM Tutorial: Part III (part3): Learn how to WS-VLAM JAVA API