iRODS Tutorial Rules and Microservices. iRODS Rules and Microservices. Introduction to Rules and Microservices Simple Rules and Database Queries Complex Rules and Scheduling. iRODS Microservices Documentation. Available from Amazon
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation
Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement.Téléchargez la présentation en cliquant sur ce lien.Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.
iRODS Rules and Microservices • Introduction to Rules and Microservices • Simple Rules and Database Queries • Complex Rules and Scheduling
iRODS Microservices Documentation Available from Amazon The integrated Rule-Oriented Data System (iRODS) Micro-service Workbook http://www.amazon.com/dp/1466469129 Contains example rules for running each microservice. On-line documentation https://www.irods.org/doxygen/
Rules • See https://www.irods.org/index.php/Changes_and_Improvements_to_the_Rule_Language_and_the_Rule_Engine • Rules implement data policy by running (workflows of) microservices • Retention, distribution, arrangement • Authenticity, provenance, description • Integrity, replication, synchronization • Deletion, trash cans, versioning • Archiving, staging, caching • Authentication, authorization, redaction • Access, approval, IRB, audit trails, report generation • Assessment criteria, validation • Derived data product generation, format parsing Policy is the clear statement of how data will be managed over its life cycle.
Microservices • C code • The unit of work within iRODS • Run by rules • Composed into workflows by rules
Running Rules • Rules triggered by events/policy points are contained in the (distributed) rule base: • iRODS/server/config/reConfigs/core.re • conditional execution • first rule with satisfied condition is executed; others are skipped • Can be run with irule - manual execution • Delayed execution • iqstat • iqmod
irule – to run a rule manually • Example rules to tweak and run are in the software distribution at iRODS/clients/icommands/test/rules3.0 • irule -F rule_file_name.r • Some rules can only be run by admin users
An example – list all microservices • Download listMS.r from /snicZone/home/pracerods/rules in iRODS • listMS.r (lists all available microservices) ListAvailableMS { msiListEnabledMS(*KVPairs); writeKeyValPairs("stdout", *KVPairs, ": "); } INPUT null OUTPUT ruleExecOut • Run it with irule: irule –F listMS.r
Format of a Rule Rule_name{ microservice1(…,*A,…,*B); microservice2(*A,…); } INPUT *A=”first_input", *B=”second_input" OUTPUT ruleExecOut OR Rule_name(*arg) { on(exp) { microservice1(…,*arg,*C); microservice2(*C,…); } } INPUT null OUTPUT ruleExecOut *A and *B are workflow variables. ruleExecOut accesses the internal ruleExecInfo (rei) structure managed by iRODS. • A rule can take arguments. • A rule can be executed conditionally. • Use “null” if there are no input parameters.
Rules and Parameters • Literals • constants: strings or numbers • a variable name not beginning with a special character (#, $ or *) is taken as string input • can only be used as input parameters (not output) • Workflow variables • Session state variables • Persistent state variables
Workflow Variables (*variables) • For example, in the following workflow chain: myRule{ msiDataObjOpen(*file,*FD); msiDataObjRead(*FD,10000,*BUF); writeLine(“stdout”,*BUF); … } INPUT *file=“/compZone/home/leesa/hello” OUTPUT ruleExecOut (‘stdout’ is a structure managed by iRODS.) • *fileis an input parameter • *FD is output from msiDataObjOpen and input to msiDataObjRead. • *file, *FD, and *BUF are workflow variables
Session Variables ($variables) • contain temporary information maintained during a server session • contain information about client-server connection, data objects, user information, resource information, etc. • contain information that can be sent back to the client. Example: stdout, stderr • persistent across rule executions in the same session, so can be used to pass information between rule executions • pre-defined by iRODS, stored as a complex C-structure (the rei structure). See https://www.irods.org/index.php/RuleExecInfo_Structure_%26_$-variables
Session Variables ($variables) • $variables map to specific locations in the REI structure - mapping contained in server/config/reConfigs/core.dvm • Example: $objPath||rei->doi->objPath $objPath||rei->doinp->objPath $dataType||rei->doi->dataType $userNameClient||rei->uoic->userName $collName||rei->coi->collName $collParentName||rei->coi->collParentName • https://www.irods.org/index.php/Session_State_Variables (Mappings are not necessarily unique.)
Persistent State Variables (#variables) • See iRODS Primer • “iquest” uses these field names: iquest attrs • https://www.irods.org/index.php/Persistent_State_Information_Variables • iRODS/lib/core/include/rodsGenQuery.h defines the attributes available via the General Query interface. • Names begin with ‘COL_’ (column) for easy identification in the source code; mapping to variables without COL_ contained in iRODS/lib/core/include/rodsGenQueryNames.h
Rule Condition • Boolean expression • Examples • Run if msiService succeeds: rule1 { on (msiService >= 0) { ... } } • Run if resource is demoResc8: rule2{ on ($rescName == demoResc8) {…} } • Run if the pathname begins with /x/y/z: Rule3{ on ($objPath like /x/y/z/*) {…} } • Same rule can give different actions depending on which condition is met • Many operators ==, !=, >, <, >=, <= %%, !! (and, or) expr like reg-expr , expr not like reg-expr , expr ::= string
Policy Enforcement Points Rule Triggers – see core.re • These are locations within iRODS framework where an event or state (of the environment) prompts a rule to execute • An action (eg, an icommand) may involve multiple policy enforcements points • Policy enforcement points • Pre-action policy (eg, selection of storage location) • Execution/action policy (eg, file deletion) • Post-action policy (eg, create secondary data products) • Actions (the triggered rules) and policy enforcement points are contained in iRODS/server/config/reConfigs/core.re
core.re • Take a look at iRODS/server/config/reConfigs/core.re • Contains rules corresponding to a number of events • Create, delete, modify users • Open, create, put, copy, replicate, delete files • Create, remove collections • Create/modify metadata • Contains some rules to set general policy • ACL policy (strict or not) • Public user policy • Default resource policy • An empty rule corresponds to a trigger point for which no action has been defined
Microservice Modules • A module must be compiled with the code, though pluggable modules (coming) will no longer require recompiling the whole code • Consult the Microservice book: The integrated Rule-Oriented Data System (iRODS) Micro-service Workbook to see which module a microservice is contained in • Enable that module: Enabled… yes in info.txt • Example • > irule -F rulemsiCopyAVUMetadata.r ERROR: rcExecMyRule error. status = -1102000 NO_MICROSERVICE_FOUND_ERR Level 0: DEBUG: execMicroService3: no micro service found line 12, col 2 msiFlagDataObjwithAVU(*Source,*Flag,*Status); • msiFlagDataObjwithAVU is contained in module ERA • enable module ERA • ./irodsctl istop • ./irodssetup
Creating New MicroservicesModules Define Function Siganture Register mService Mapping Create Internal Function Describe mService Any function can be converted into a microservice, but it’s important to implement recovery microservices Important!! Implement recovery mService This procedure will change with coming releases.
Example Rules • See iRODS/clients/icommands/test/rules3.0 in the iRODS distribution • See /compZone/home/leesa/rules in compZone data grid • See /compZone/home/rods/rules in compZone data grid • See /snicZone/home/pracerods/rules in snicZone • See http://www.renci.org/~leesa/rules/
Listing the Rule Base showCore.r showCoreRules { # Listing of the core.re file # # Input parameters: # none msiAdmShowCoreRE(); } INPUT null OUTPUT ruleExecOut An admin user can execute the rule to show the rule base: • irule –vF showCore.r
rulemsiExecCmd.rRun an executable script myTestRule { #Input parameters are: # Command to be executed - located in directory irods/server/bin/cmd # Optional command argument string # Optional host address for command execution # Optional hint for remote data object path, command is executed on host # where the file is stored # Optional flag. If > 0, use the resolved physical data object path as first argument #Output parameter is: # Structure holding status, stdout, and stderr from command execution #Output: # Command result is: “Hello world written from irods” # msiExecCmd(*Cmd,*Arg,"null","null","null",*Result); msiGetStdoutInExecCmdOut(*Result,*Out); writeLine("stdout","Command result is"); writeLine("stdout","*Out"); } INPUT *Cmd=”hello", *Arg="written" OUTPUT ruleExecOut “hello” is an executable script in iRODS/server/bin/cmd.
Example Policy ImplementationUsing “asPostProcForPut” to implement policy, depending on resource Data coming in to a target iRODS resource triggers a script that takes some desired action, triggers message to admin (unix) user acPostProcForPut{ on($rescName like ”uncResc") { writeLine("serverLog","USER, OBJPATH, and FILEPATH: $userNameClient, $objPath and $filePath"); msiExecCmd("resource-trigger.sh","$rescName $objPath $userNameClient","null","null","null",*Out); msiSendMail("leesa@renci.org","resource $rescName","User $userNameClient just ingested file $objPath into $rescName."); } } • acPostProcForPut is contained in • iRODS/server/config/reConfigs/core.re • resource-trigger.sh is contained in /server/bin/cmd • and must be executable!!
Example script resource-trigger.sh • > more resource-trigger.sh #!/bin/sh # echo "execCmdRule: "$execCmdRule rescName=$1 objPath=$2 userNameClient=$3 echo "User $userNameClient just ingested file $objPath into $rescName" echo "User $userNameClient just ingested file $objPath into $rescName" > /tmp/resource.out
Example Policy ImplementationUsing asPostProcForPut to implement policy: inputs to a specific resource Data coming in to a target iRODS collection triggers a script that takes some desired action (sending data to a remote ftp site) acPostProcForPut{ on($objPath like ”/snicZone/home/prace-class/*") { writeLine("serverLog", "$userNameClient sending $objPath to NCDC."); msiSplitPath($filePath,*fileDir,*fileName); msiExecCmd(“outgoing.sh","*fileDir *fileName","null","null","null",*Out); msiSendMail("leesa@renci.org", "send to NCDC","User $userNameClient sent $objPath to NCDC."); } } • acPostProcForPut is contained in • iRODS/server/config/reConfigs/core.re • acPostProcForPut is the same rule in both examples! Just using different conditions.
Example script outgoing.sh • > more outgoing.sh #!/bin/sh HOST=ftp.****.****.*** USER=anonymous PASSWD=leesa@renci.org srcDir=$1 srcFile=$2 echo $srcDir echo $srcFile #echo "Place holder for outgoing script. Dir: $srcDir, File: $srcFile" echo "Place holder for outgoing script. Dir: $srcDir, File: $srcFile" > /tmp/test.out
iquest – for constrained querying of the iCAT • “iquest attrs” shows the attributes you can query on • Query iCAT of remote zoneA: iquest –z zoneA … Example: User rods#compZone, logged into compZone, gives the command > iquest -z snicZone "SELECT COLL_NAME, DATA_NAME WHERE COLL_NAME like '/snicZone/home/rods%'" Zone is snicZone COLL_NAME = /snicZone/home/rods#compZone DATA_NAME = hello ------------------------------------------------------------ The response comes from the snicZone iCAT, not the user’s home compZone iCAT.
More on iquest Set SQL logging to see actual SQL queries generated using iquest • Edit scripts/perl/irodsctl.pl - uncoment the line $spLogSql = "1”; • ./irodsctl irestart • Logged into iRODS/server/log files
Specific QueriesMore on iquest • iadmin asq – add SQL query Example: show all data files with metadata, along with the metadata > iadmin asq 'select object_id, data_id, data_name, meta_attr_name, meta_attr_value from r_objt_metamap, r_meta_main, r_data_main where r_meta_main.meta_id = r_objt_metamap.meta_id and object_id = data_id;' file-metadata Now any user can make this query using iquest: > iquest –-sql file-metadata Data grid administrators: beware what you enable with specific queries; the iCAT contains passwords, ACLs, etc. Control carefully the access you give. Alias to the query
GenQuery • The primary query interface in IRODS • msiMakeGenQuery constructs a SQL string that’s passed to the iCAT by a subsequent call to msiExecGenQuery • Sets up a data structure, so use msiGetValByKey to get values out of the call to msiExecGenQuery • Many iRODS commands (eg, ils) use genQuery behind the scenes • Generates SQL to perform the query subject to access control • Simplified SQL-like interface – user doesn’t need to know the iCAT schema to make queries • Queries on same attributes as iquest
GenQuery Example test0.r – list all data objects with the name “hello” mygenQuery{ # Input: # string containing attribute list to select on # string containing condition # Output: # GenQInp: an internal data structure # msiMakeGenQuery("COLL_NAME", *Condition, *GenQInp); msiExecGenQuery(*GenQInp,*GenQOut); foreach(*GenQOut){ msiGetValByKey(*GenQOut,"COLL_NAME",*outcoll); writeLine("stdout", "*outcoll/hello"); } } INPUT *Condition="DATA_NAME = 'hello'" OUTPUT ruleExecOut
GenQuery Example irule –F test0.r /compZone/home/leesa/hello /compZone/home/outgoing/hello /compZone/home/rods/hello /compZone/trash/home/rods/hints/linux-hints/ischia2-hints/hello This rule was run as user rods (admin user) on compZone. GenQuery will not allow all users to make all queries. ACL permissions are respected with GenQuery. Examples: see /compZone/home/rods/rules/genQuery or /snicZone/home/pracerods/rules/genQuery Try tweaking and running some examples.
Database Resources • DataBase Resource (DBR): an external database (or similar tabular information) that can be queried and updated via SQL statements (or other, for non-SQL) • Database object (DBO): an interface to the DBR - typically a query that returns results • DBOs typically contain SQL • Query results can be stored to an iRODS data object, a DBO Results file (DBOR). • idbo command – to run the DBO query • access controls imposed by iRODS ACLs • https://www.irods.org/index.php/Database_Resources and https://www.irods.org/index.php/Database_Resource_Administration
Setting Up a DBR • See https://www.irods.org/index.php/Database_Resource_Administration • Install a non-IES server (an additional resource) on a machine that has access to the remote DB. Example: install on iren.renci.org. • On iren.renci.org, edit server/config/dbr.config - add a line containing: DBR-name DBMS-username DBMS-password DB-type (postgres, oracle, mysql) Example: leesabase leesa leesapw postgresql • On iren.renci.org, put a .odbc.ini file into $HOME [IRODS_DBR_leesabase] Driver=/home/leesa/tutorial-testing/pgsql/lib/libodbcpsql.so.2.0.0 Debug=0 CommLog=0 Servername=informatics.renci.org Database=leesabase ReadOnly=no Ksqo=0 Port=5432 (Can copy – mostly – from $HOME/.odbc.ini on your data grid server.)
Setting Up a DBR • Edit iRODS/config/config.mk: • uncomment DBR=1 • define POSTGRES_HOME: POSTGRES_HOME=/home/leesa/tutorial-testing/pgsql • Make sure the LD_LIBRARY_PATH includes the postgres libraries: setenv LD_LIBRARY_PATH /home/leesa/tutorial-testing/pgsql/lib • Do a new "make" of the code in iRODS/and then ./irodsctl irestart
Setting Up a DBR • Go over the main data grid (ischia2.renci.org) and use iadmin to define the DBR in the ICAT. See 'iadmin h mkresc’. mkresc Name Type Class Host [Path] (make Resource) Example: mkresc leesabase database postgresql iren.renci.org /home/leesa/tutorial-testing/iRODS3.1/Vault • Give ownership of the DBR to the appropriate user(s) - for write/query permission to the DBR: 'ichmod -R own User-name DBR-name’ Example: ichmod -R own rods leesabase • See iRODS/server/icat/dbr, containing example DBOs. Put a query into a file and ingest into iRODS: iput lt.pg
Setting Up a DBR • Use isysmeta to declare this as a DBO type: isysmeta mod lt.pg datatype 'database object' • Use idbo to execute a query (contained in a dbo) on the DBR (leesabase in this case), running as a user that has ‘own’ permission on the DBR: idbo exec dbrname dboname Example:idbo exec leesabase lt.pg Note: when giving this command, you either need to be in the collection that contains the dbo file, or you need to give the full path name to it.
Using a DBR • idbo to execute a query contained in a DBO • Store results into a DBO Result file (DBOR) by using idbo as an interactive shell: >idbo idbo> output lt.out 'exec' results will be stored in /compZone/home/rods/dbr-examples/lt.out idbo>exec leesabase lt.pg Output written to /compZone/home/rods/dbr-examples/lt.out idbo>quit • Examples: in /compZone/home/rods/dbr-examples NB: For this tutorial the DBR hasn’t been set up on snicZone, so DBOs must be run on compZone.
Delayed Execution • Example myTestRule{ delay("<PLUSET>1m</PLUSET>"){ writeLine("stdout",”Writing message with a delay."); msiSendStdoutAsEmail(*Mailto, "Sending email"); } } INPUT *Mailto="leesa@renci.org" OUTPUT ruleExecOut • Queue management: • iqstat • iqdel • iqmod
Periodic Execution Example myTestRule { # Input parameters are: # Source collection path # Target collection path # Optional target resource # Optional synchronization mode: IRODS_TO_IRODS # Output parameter is: # Status of the operation # Output from running the example is: # Synchronized collection 1 with collection 2 # delay("<PLUSET>5m</PLUSET>EF>1h</EF>"){ msiCollRsync(*srcColl,*destColl,*Resource,"IRODS_TO_IRODS",*Status); writeLine("stdout","Synchronized collection *srcColl with collection *destColl"); } } INPUT *srcColl="/compZone/home/leesa/tutorials", *destColl="/compZone/home/leesa/tutorials2", *Resource="demoResc" OUTPUT ruleExecOut
More Interesting Rules Policy to control where data is stored upon ingestion (courtesy of iPlant) • Data ingested into TACC collections will be forced to resource “corral”: acSetRescSchemeForCreate { ON($objPath like "/tacc/Collections/*") {msiSetDefaultResc(corral,forced);} } (This forces physical location, with no effect whatsoever on logical location.) • All other data goes into iplant resource group, randomly distributed among resources there: acSetRescSchemeForCreate { msiSetDefaultResc("iplantRG","preferred"); msiSetRescSortScheme("random"); } (This means that a default resource doesn’t need to be specified by users.)
Connexion au compte
Connectez-vous pour publier votre contenu sur SlideServe.
