1 / 21

Michael Pipp Program Manager Information Management & Technical Services

MWCC Spring Training Effective Water Quality Monitoring. Data Management. Michael Pipp Program Manager Information Management & Technical Services Water Quality Planning Bureau. May 17, 2012. Topics. Data Management Best Practices WQPB Data Flows MT-eWQX Tools & Support EDP Demo.

larryparker
Télécharger la présentation

Michael Pipp Program Manager Information Management & Technical Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MWCC Spring Training Effective Water Quality Monitoring Data Management Michael Pipp Program Manager Information Management & Technical Services Water Quality Planning Bureau May 17, 2012

  2. Topics • Data Management Best Practices • WQPB Data Flows • MT-eWQX Tools & Support • EDP Demo

  3. Citations Best Practices for Preparing Environmental Data Sets to Share and Archive. Les A. Hook, Suresh K. SanthanaVannan, Tammy W. Beaty, Robert B. Cook, and Bruce E. Wilson. September 2010. Oak Ridge National Laboratory Distributed Active Archive Center, Oak Ridge, Tennessee. Environmental Sciences Division, Oak Ridge National Laboratory 5 Essential Elements of a Successful Hydrological Monitoring Program.2012, Stu Hamilton. Published by: Aquatic Informatics, Inc. www.aquaticinformatics.com Network Folder and File Management Directive & Protocols.MT DEQ Water Quality Planning Bureau, WQPBDMSPOL-04, v1.2, 2009

  4. Best Practices: Systems Organization • Quality Management System (QMS) • QA, QC, & Data Management • QA: independent review procedures • QC: system of routine & consistent checks for integrity, completeness, & compliance w/ SOPs • DM: establishes data defensibility by providing evidence of QMS compliance

  5. Best Practices: Systems Organization • Process Definition • Event planning • Authorizations • Control points • Tracking Systems • Supplies & Equipment • Current year events and activities • Location in DM flow/life cycle • Commit actions to system (e.g., prog, e.g., pub)

  6. Best Practices: Data Management • Data File Contents • Data & parameter dictionary • Data Organization • “Record” Format • Content (grouping) • File limitations • MS Excel 2003 vs 2007/2010

  7. Best Practices: Data Management • File Structure & Formats • Only data in data files • ASCII text tab-delimited (Excel) • General BP for Header Rows • First row: descriptors linking data file to data set • Other rows: column contents, parameter names, parameter units • Column Headers: numbers, letters, hyphens, underscores (no spaces or special chars)

  8. Best Practices: Data Management • Text Files • Column delimiters (comma, tab, semicolon, pipe) • Missing values (NULL, -99.999) • File extension describes delimiter (*.csv &*.txt) • Dataset Documentation (core metadata) • Describe file names & delimiters • Expanded parameter descriptions • Missing values codes • Example data file record

  9. Best Practices: Data Management • File Names • Avoid special characters, .e.g. • \ / : * ? “ < > | [ ] & $ , . • Use underscores not spaces • Err on brevity • Excel limits: file names (31); total path (218)

  10. Best Practices: Data Management • File Name Examples: Example of a 200 character path & file name: [DEQCluster3\Mirrortest\DEQcluster_Unitshr]G:\WQP\3_MONITORING_&_ASSESSMENT\Temporary_Datashare\Zach\Not_Adam\TemperatureDataLoggers\TempMacrosOriginals\Temperature_Information\SOP_WQPBWQM_TEMPLOG.doc Example of a 263 character path & file name: G:\WQP\6_DataMgmt\EQuIS\EQuIS_Staging\2011_FS_MDEQ_WQ\2011_ProcessedSpecialProjects\External\MT305b_SecondaryData\Dearborn_EricChase_305bCallForData\Submission_2\Received_01272011\20110127+1440.MTWTRSHD_WQX.MTDEQ_WQX.zip

  11. Best Practices: Data Management • File Names (continued) • Identify file independent of storage location • Network location &/or contents • Include dates & format consistently • Include version number • Standard conventions (www.iso.org) • Version control: *_v1.csv or *_r1.csv, or *_20120517.csv • Date (ISO 8601): YYYY_MM_DD (or YYYYMMDD) • Time (ISO 8601):hh:mm:ss • Date and Time (ISO 8601): YYYY-MM-DDThh:mm:ss • e.g., 2003-04-01T13:01:02

  12. Best Practices: Data Management • General Quality Control Actions • Check file structure [EDP] • Check for data file completeness [EDP] • Check measured or derived values [EDP] • Map locations (lat/long) • NRIS Topofinder application • Search Tools --- lat/long entry and mapping • Perform statistical summaries • Excel summary stats: min\max\count

  13. Best Practices: Data Management • Quality Control for WQPB Submittals • EDD - Activities • Ensure Activity IDs match the SVF • Ensure Activity Start Date and Activity Start Time match the date/time the samples were collected • Ensure field duplicates and field blanks are clearly identified and the Activity Type is appropriate • i.e., QC-FR, QC-FB

  14. Best Practices: Data Management • Quality Control for WQPB Submittals • Lab Reports • Reporting detection limit met the project-required detection limit defined in SAP. • Laboratory blanks/duplicates/matrix spikes/lab control samples were all within the required control limits defined within the SAP. • Samples that exceed control limits, flag all associated data • EDD – Results • Refer to hand-out /book & 14 point check list!

  15. Best Practices: Data Management • Metadata for local management & publication • Assign Descriptive Data Set Titles (web discovery) • data, date range, location,&/or instruments • Limit to 85 characters (including spaces) • No special characters • Similar to underlying data files in data set • Provide Documentation (METADATA!) • User unfamiliar with project, sites, methods, and/or observations • metadata standards for dataset discovery • FGDC, NetCDF Attribute Convention for Dataset Discovery, Ecological Metadata Language (EML), ISO-19115

  16. WQPB Data Flows: Data Life Cycle • Data Life Cycle • Project Planning • Sample/Data Collection • Sample Submittal & Analysis • Results & Reports • Data Verification & Validation • Data Use & Decision • Long-term Storage / Archives

  17. WQPB Data Flows: Field Forms • Field Forms Processing Flow • Internal QC & DM • Laboratory Data EDD

  18. MT-eWQX Tools & Support • MT-eWQX Tools & Support • Website, guides, & forms • Field forms • EDD • Lab EDDs (chemistry & biology) • MTWTRSHD Result EDD • EDD formats • Current is only x86 (32-bit OS) • 2012 development for x64 (64-bit OS)

  19. MT-eWQX Tools & Support • EQuIS Data Processor (EDP) Demo

  20. WQPB DM Contacts IMTS Program Manager IMTS MT-eWQX Manager Jolene McQuillan Billings DEQ Office jmcquillan@mt.gov 406.247.4436 Michael Pipp Helena, Metcalf Building mpipp@mt.gov 406.444.7424

  21. QUESTIONS?

More Related