GUS Plugin System

1. GUS Plugin System Michael Saffitz Genomics Unified Schema Workshop July 6-8th, Philadelphia, Pennsylvania

2. Plugin Overview • Small Perl programs that load and manipulate data within GUS • Written using the GUS Plugin API and Perl Object Layer • Provide automatic support for: • Data Provenance • Object layer and database connectivity • Standardized documentation • Command line argument processing • Logging • Error Handling • “Supported” and “Community” Plugins provided with GUS

3. Supported Plugins • Have been tested in Oracle and Postgres and are confirmed to work • Portable • Useful beyond the site that developed them • Meet the GUS Plugin Standard

4. Community Plugins • Fail to meet one or more of the criteria above • Have not been tested • Provided as a general resource to the community

5. Plugin Life Cycle • Plugin Initialization • Documentation • Command Line Arguments • Data Loading • Reading, Parsing, Querying • Data Manipulation • Insert or Update? • Restart Logic • Data Submission

6. InsertArrayDesignControl.pm InsertAssayControl.pm InsertBlastSimilarities.pm InsertExternalDatabase.pm InsertExternalDatabaseRls.pm InsertGOEvidenceCode.pm InsertGeneOntology.pm InsertGeneOntologyAssoc.pm InsertRadAnalysis.pm InsertReviewStatus.pm InsertSecondaryStructure.pm InsertSequenceOntology.pm LoadArrayDesign.pm LoadArrayResults.pm LoadFastaSequences.pm LoadGusXml.pm LoadNRDB.pm LoadRow.pm LoadTaxon.pm GUS Supported Plugins

7. Plugin Shell package GUS::Supported::Plugin::LoadRow; @ISA = qw(GUS::PluginMgr::Plugin); use strict; use GUS::PluginMgr::Plugin; sub new { … } sub run { … }

8. Plugin Initialization sub new { my ($class) = @_; my $self = {}; bless($self, $class); $self->initialize({ requiredDbVersion => 3.5, cvsRevision => '$Revision: 2934 $', name => ref($self), argsDeclaration => $argsDeclaration, documentation => $documentation }); return $self; }

9. Declaring Arguments stringArg({name => 'externalDatabaseVersion', descr => 'sres.externaldatabaserelease.version for this instance of NRDB', constraintFunc => undef, reqd => 1, isList => 0 }), fileArg({name => 'gitax', descr => 'pathname for the gi_taxid_prot.dmp file', constraintFunc => undef, reqd => 1, isList => 0, mustExist => 1, format => 'Text' }),

10. Argument Types • String • Integer • Boolean • Table Name • Float • File • Enumeration • Controlled Vocab • Local, Database Term Pairs for “dinky” CVs

11. Declaring Documentation my $tablesDependedOn = [ ['GUS::Model::DoTS::NRDBEntry', 'pulls aa_sequence_id from here when id and extDbId match requested']]; my $documentation = { purposeBrief => $purposeBrief, purpose => $purpose, tablesAffected => $tablesAffected, tablesDependedOn => $tablesDependedOn, howToRestart => $howToRestart, failureCases => $failureCases, notes => $notes };

12. Plugin Initialization sub new { my ($class) = @_; my $self = {}; bless($self, $class); $self->initialize({ requiredDbVersion => 3.5, cvsRevision => '$Revision: 2934 $', name => ref($self), argsDeclaration => $argsDeclaration, documentation => $documentation }); return $self; }

13. Plugin Shell package GUS::Supported::Plugin::LoadRow; @ISA = qw(GUS::PluginMgr::Plugin); use strict; use GUS::PluginMgr::Plugin; sub new { … } sub run { … }

14. Run Method • “Entry point” for plugin • Concise overview/“table of contents” for plugin: sub run { my ($self) = @_; my $rows = 0; my $rawData = $self->readData();my @parsedData = $self->parseData($rawData); foreach $data (@parsedData) { $data->submit(); $rows++; } return “Inserted $rows ”; }

15. Accessing Data • Command line arguments: • $self->getArg(‘nrdbFile); • Through Objects: • my $preExtAASeq =GUS::Model::DoTS::ExternalAASequence->new ({'aa_sequence_id'=>$aa_seq_id});$preExtAASeq->retrieveFromDB(); • Direct Database Access: • my $dbh = $self->getQueryHandle();my $sth = $dbh->prepare(…);

16. Persisting Data • Saving & Updating: • $obj->submit(); • Will cascade and submit children • Delete: • $obj->markDeleted(1);$obj->submit();

17. Logging and Error Handling • For general logging, use logging functions • Printed to STDERR • $self->log(“message”) • For error handling: • Either die() immediately or • Write errors to a file (for recoverable errors) • Restart functionality • Check for object existence • Check, but ensure loaded from a valid proper invocation • Store data from previous run and use as a filter

18. Clearing the Cache • Historical: Perl previously had poor garbage collection support • Default capacity of 10000 objects • At the bottom of the outermost loop: • $self->undefPointerCache();

19. Data Provenance • Tracks plugin revisions-- Name, Checksum, Revision • Tracks parameters that a specific plugin is executed with Algorithm AlgorithmParamKeyType AlgorithmImplementation AlgorithmParamKey AlgorithmInvocation AlgorithmParam

20. Plugin Evolution • Changes abound: • Data file formats • Schema • Be flexible in writing plugins-- command line configuration • Be clear about what schema objects you use

21. Plugin Standard • See Developer’s Guide: • http://gusdb.org/documentation/3.5/developers/developersguide.html