PDCC: Portable Data Collection Centers

PDCC:Portable Data Collection Centers Version 3.0 Rick Benson and Chad Trabant IRIS DMC Data Management Workshop Sao Paulo, Brazil

Mission Statement • Provide a SEED management tool for network operators. • Allow easy updates to current SEED information. • Persistent storage of changes with a RDBMS • Ease of use and accessibility. • Cross-platform support. Data Management Workshop Sao Paulo, Brazil

Why is metadata important? They are the Rosetta Stone- making it possible to decode information well into the future • Increases accessibility: “tell me where sites are in a • region”, or “what are station names in an area”, or “what • instruments operate in a region”, etc. • Carefully created, they create both short and longterm • benefits by capturing “tribal knowledge” when it’s • still available, like compression, sample rates, • digital filtering, documenting an accurate history • of ground motion • Retention of context- who operates instruments, etc • Remember: People forget! Data Management Workshop Sao Paulo, Brazil

Metadata In(di)gestion • Functionally, we use “technical” metadata, in that it is used to describe how the seismic system behaves, and limited software description (like compression, etc) • metadata can be shared in dataless SEED, • which makes metadata “portable” amongst heterogeneous • systems • Whenever anything changes at a site, new metadata gets • exchanged Data Management Workshop Sao Paulo, Brazil

PDCC Features • Edit SEED data blockettes • Station metadata • Response coefficients • Station and channel comments • Reads and writes SEED format • Persistent storage Data Management Workshop Sao Paulo, Brazil

Database tables Data Management Workshop Sao Paulo, Brazil

PDCC Features • Limited support for waveform data records (miniSEED) • RESP formatted text files • SAC file export • MySQL database Data Management Workshop Sao Paulo, Brazil

PDCC features • Accessible GUI • Platform independent, written in Java • Extensible software; open source and modular • MySQL database is one storage option Data Management Workshop Sao Paulo, Brazil

Start Up • Users typically start their session by Importing a file. • SEED files are supported as well as RESP files. • MySQL database (created with PDCC) import is an option. Data Management Workshop Sao Paulo, Brazil

Importing a SEED file • When you import a SEED file, you are presented with a file selection tool. • Selecting a file and opening it triggers the import process. Data Management Workshop Sao Paulo, Brazil

Importing a SEED file • PDCC works best on small SEED files. Larger files take some time to load. • You can monitor the status of the import operation with the status bar at the bottom Data Management Workshop Sao Paulo, Brazil

Importing a SEED file • When the SEED file has been read in, PDCC renders the contents in the Tree Viewer on the left side. • You will see on OK prompt at the bottom when this process is completed. Data Management Workshop Sao Paulo, Brazil

Browsing the SEED file • Navigation through a SEED file’s contents is performed through the Tree Viewer • The loaded file appears as a new volume • Selecting a node in the tree will cause the contents to be displayed Data Management Workshop Sao Paulo, Brazil

Three Kinds of Nodes In the Tree Viewer, you’ll find: • Volume Nodes- highest level • Category Nodes- station, then channel… • Blockette Nodes (SEED data) Data Management Workshop Sao Paulo, Brazil

Volume Node Display • A Volume is a collection derived from a data source. • The data source may be a file on disk • It may be another source as well, like a database. Data Management Workshop Sao Paulo, Brazil

Volume Node Display • Volumes can be removed from PDCC by using the Close Volume action in the File menu • Remember: this is a reflection of the SEED format Data Management Workshop Sao Paulo, Brazil

Category Node Display • Relates information regarding a certain family of nodes • There are currently three kinds of category nodes: • Dictionary • Station • Data Data Management Workshop Sao Paulo, Brazil

Blockette Node Display • Primary data object is the Blockette • Blockettes are identified by a type number • Blockettes also have a name, which is what is clearly displayed at the top. Data Management Workshop Sao Paulo, Brazil

Blockette Node Display • At the top is a unique ID number • No two objects have the same ID number • The Fields of the Blockette are shown with their FieldName and corresponding value Data Management Workshop Sao Paulo, Brazil

Blockette Node Display • Most of the fields are Text Fields, and can be edited by typing in the box • Some of the fields are Expanding Fields and present a button to be clicked • Clicking the button results in a popupdisplay, showing the details of that reference Data Management Workshop Sao Paulo, Brazil

Field Validation • When a user enters a value into a data field, it is examined for proper formatting. • If the field entry is improperly formatted, the user is notified of the need for correction. Data Management Workshop Sao Paulo, Brazil

Restore Button • Changes to a Blockette are automatically saved to cache when the user exits that edit view • Hitting the Restore button before exiting the edit window restores all original entries Data Management Workshop Sao Paulo, Brazil

Popup Display • The popup display is used to show additional information that relates to a given field. • Sometimes this is dictionary reference information • Sometimes this is tabular information, such as response coefficients Data Management Workshop Sao Paulo, Brazil

Dictionary Popup • A popup view from a blockette field refers to an Abbreviation Dictionary entry • The popup tool allows you to change the abbreviation reference • You can also edit the dictionary contents Data Management Workshop Sao Paulo, Brazil

Dictionary Popup • In dictionary popups, the topmost field contains a list selector • You use this selector to change the abbreviation that your blockette references. Data Management Workshop Sao Paulo, Brazil

Tabular Display • Some information in a Blockette comes in tabular format, such as response coefficients • Each cell in a row can be edited in place • Rows can be inserted and deleted using the buttons on the left Data Management Workshop Sao Paulo, Brazil

Tabular Import • You can import raw numeric columns from text files directly into the Tabular Display • Inserting text data replaces any existing tabular data • Appending adds data to the end of the table Data Management Workshop Sao Paulo, Brazil

Copy and Paste • You can Copy and Paste Tree Nodes from the Edit menu. • Copying puts a separate copy in the PDCC clipboard. • Pasting a copy of the clipboard contents at the selected position in the tree. Data Management Workshop Sao Paulo, Brazil

Cut • The Cut function removes the node from its place in the navigation tree • You can Paste a Cut node, just as you would with Copy • If you Copy or Cut another node, the earlier copy is replaced in the clipboard Data Management Workshop Sao Paulo, Brazil

Delete • The Delete function deletes a node entirely • The user is asked to confirm the deletion • All children of the deleted node are removed from memory • Volume nodes and category nodes cannot be removed this way Data Management Workshop Sao Paulo, Brazil

Exporting a Volume • Once a volume is ready for writing to a file, the Export operation is selected. • The menu indicates the supported output formats. • Writing to the database is also an Export operation. Data Management Workshop Sao Paulo, Brazil

Writing to SEED • Three kinds of SEED volumes may be written: • Dataless SEED - station metadata • miniSEED - waveform data records • Full SEED - metadata, waveform data records, and volume headers • miniSEED and full SEED volumes require that waveforms are present in the volume you export. Data Management Workshop Sao Paulo, Brazil

Writing to SEED • Select the Volume node that you want to export to SEED. • Select File->Export->SEED file (or miniSEED or dataless SEED) • Enter the filename and directory for the SEED file in the selector tool. Data Management Workshop Sao Paulo, Brazil

Writing miniSEED • Only writes the data records to disk in miniSEED format, in station.channel.day file format. Data Management Workshop Sao Paulo, Brazil

Writing SAC • PDCC can also dump data records to SAC formatted files. • These files are placed in a user-configured directory. Data Management Workshop Sao Paulo, Brazil

Create a new volume • Many times, a user needs to create a new volume instead of editing an old one • In the File menu, select New->Export Volume • The user will be prompted for a volume name Data Management Workshop Sao Paulo, Brazil

Adding content There are two ways to add new content to a volume: • Create a New node of the appropriate type • Copy and paste from existing nodes, then modify them Data Management Workshop Sao Paulo, Brazil

Adding content • New, blank Blockettes can be generated from the File menu • Most menu selections create a single new Blockette • Some generate a set of blockettes representing a type of response stage Data Management Workshop Sao Paulo, Brazil

Response Files • Response files are a good source of instrument information • Response files are Imported, just like SEED files • PDCC can also write RESP files from a selected Volume Data Management Workshop Sao Paulo, Brazil

Database operations • A MySQL database is used for long-term persistent storage • The database is optional for PDCC • MySQL needs to be configured to support PDCC data tables • PDCC can build the necessary database tables from the GUI. This is where you initialize the database: Data Management Workshop Sao Paulo, Brazil

Database Volume • There is always just one database volume in the Tree View • The database volume is created through the Import function • The database volume keeps track of what it has loaded and what changes it needs to write to the database Data Management Workshop Sao Paulo, Brazil

Treat like a file volume • In most respects, you can treat the database volume like a file volume • Perform all the usual operations and edit functions • Cut and paste to and from database volume to other volumes Data Management Workshop Sao Paulo, Brazil

Exporting changes to DB • When the database volume has been edited, the system will not write these changes back to the database automatically • You must Export the database volume for the updates to occur Data Management Workshop Sao Paulo, Brazil

Exporting files to the DB • New file volumes can be added to the database • Select the file volume and activate the Export to database menu option • The file volume’s contents will be appended to the database as new entries Data Management Workshop Sao Paulo, Brazil

Exporting Data to the DB • PDCC supports the archiving of miniSEED data records to the database. • The miniSEED records are actually written to disk as station.channel.day files. • The data itself is catalogued in the PDCC database for later retrieval. • Implementation is rudimentary and still needs further development to be an effective data archival system. (For example, suturing records isn’t implemented yet) Data Management Workshop Sao Paulo, Brazil

Database Refresh • There is a Refresh option that functions like an Export, followed by an Import • You can use this to routinely keep your changes up to date with the database • It also helps to verify that the database is holding your data Data Management Workshop Sao Paulo, Brazil

Cached Workspace • PDCC persistently caches the data volume contents to disk. • When PDCC is restarted, all cached volumes will be reloaded to their last edited state.   Data Management Workshop Sao Paulo, Brazil

Dataless SEED Files from IRIS DMC • Getting an initial sample of dataless SEED information can make construction of new station information easier • They serve as building blocks for new metadata • Send a breq_fast request for a custom IRIS DMC dataless: mailto: dataless@iris.washington.edu Data Management Workshop Sao Paulo, Brazil

Dataless SEED Files from IRIS DMC • Dataless SEED files can be requested from the BUD Query Interface on the IRIS DMC web site http://www.iris.washington.edu/bud_stuff/dmc/index.htm • There are pre-built, always current, dataless SEED files available through FTP: ftp://ftp.iris.washington.edu/pub/RESPONSES/DATALESS_SEEDS Data Management Workshop Sao Paulo, Brazil

Response Files from IRIS DMC • Response (RESP) files are also available through FTP, in network directories ftp://ftp.iris.washington.edu/pub/RESPONSES • The Response file import feature is not yet bulletproof, so some editing of the RESP file may be necessary for import into PDCC Data Management Workshop Sao Paulo, Brazil

PDCC: Portable Data Collection Centers