LEADS/EMS SYSTEM ADMINISTRATION

LEADS/EMS SYSTEM ADMINISTRATION

LEADS/EMS Typically Uses Two File Spaces (Disks) For Its Installation Many Of The Machine Directory Structures Are The Same – LEADS1, DSR, And WWW Have A Common Directory Structure The CFEP And NOAAPort Have Different Directory Structures The NOAAPort Machine Is Not Covered In This Section Of The Training Material DIRECTORY STRUCTURE

Refer to: “Appendix A – Directory Structures” DIRECTORY STRUCTURE (Cont.)

DATA FLOW

Refer to: “Appendix B – LEADS-EMS Data Flow” DATA FLOW (Cont.)

Equivalent To Unix Cron Jobs Scheduled Tasks Is The Microsoft Windows Utility Used To Schedule An Operation To Be Executed At Periodic Intervals All Tasks Run Under The “leadsadmin” account There Are Tasks Scheduled For Periodic Execution On The LEADS1 and CFEP Machines Most Scheduled Tasks Can Be Manually Invoked As Needed Certain Tasks May Need To Be Invoked As New Sites Are Brought On-line Data Reloads May Necessitate This As Well Scheduled Tasks Can Also Be Temporarily Disabled This Is Done With The Scheduled Tasks Utility SCHEDULED TASKS

CommandedTimeSync – Runs Every Minute Looking For A Web Commanded Time Sync Of A Single CAMS ModemPolling – Runs Every Six Hours Poll Data Loggers With Telephone/Modem Connections InternetPolling – Runs Every Fifteen Minutes Poll Data Loggers With Internet Connections SendZenoFiles – Runs Every Fifteen Minutes Send Data Sets Collected From All Polled Sites To The LEADS1 Machine For Ingest Processing TimeSyncs – Runs Twice A Week Send Time Synchronization Updates To All Data Loggers CFEP SCHEDULED TASKS

Build Starts – Runs Every Three Hours Scans The Hourly Database And Creates An Auxiliary File Containing Information About Each Parameter Monitored BuildPreviousYearSummaries – Runs Each Sunday Forces A Rebuild Of The Various Summary Files For The Previous Year (This Is Necessary In Case Someone Has Modified Any Data. Through Manual Validation For Example.) Create Loss – Runs Once A Day Scans The Hourly Database And Creates An Auxiliary File Noting The Number Of Flags Recorded For Each Monitor Create1Hour – Runs Every Hour Scans The Hourly Database And Creates An Auxiliary File Containing The Maximum Ozone Averages Recorded For Each Site For Each Day LEADS1 SCHEDULED TASKS

Create24Hour – Runs Every Hour Scans The Hourly Database And Creates Auxiliary Data Files Containing Daily Averages For Select Parameters Currently Configured To Create Daily Averages For SO2, PM-2.5, PM-10, And O3 Create30Minute – Runs Every Seven Hours Scans The Hourly Database And Creates Auxiliary Files Containing Running 30-Minute Averages Of SO2 And H2S Create8Hour – Runs Every Hour Scans The Hourly Database And Creates Auxiliary Files Containing Rolling 8-Hour Averages Of O3 And CO CreateAQIMaps – Runs Every Hour Constructs The AQI Map Image – Uses Many Of The Auxiliary Data Files LEADS1 SCHEDULED TASKS (Cont.)

CreateExceed – Runs Every Hour Scans The Hourly Database And Creates An Auxiliary File Containing Any Ozone Averages That Exceed 125 PPB CreateOzoneMaps – Runs Every Hour Creates The Maps Of Current Ozone Levels EPA AirNow Export – Runs Every Hour Scans The Hourly Database And Constructs An Input File For The EPA AIRNow System, Then FTPs This File To The AIRNow Site Load Calibration Reports – Runs Every Six Hours Update The Ancillary SQL Database With Calibration Run Results Harvested From The Data Stream LEADS1 SCHEDULED TASKS (Cont.)

Missing Data Summary – Runs Every Fifteen Minutes Scans The Hourly Database Looking For Any Sites That Have Not Had Data Harvested For The Last Two Hours Constructs An Auxiliary File Enumerating Any Sites With Problems Sync AutoRestart – Runs Every Five Minutes If A Unison Synchronization Problem Is Reported, Rename The DANGER.README File In Order To Allow Synchronization To Continue Sync Files from CFEP – Runs Every Fifteen Minutes Synchronize The Contents Of The “//CFEP/leads/logs/” Directory To “//LEADS1/leads/logs/CFEP/” And To “//DSR/leads/logs/CFEP/” These Files Are Automatically Created By The CFEP Processing Software And Contain System Information LEADS1 SCHEDULED TASKS (Cont.)

Sync Files to DSR – Runs Every Fifteen Minutes Synchronize The Contents Of The “//LEADS1/dsrweb/” Directory To “//DSR/dsrweb/” Synchronize The Contents Of The “//LEADS1/leads/data/pdcs/” Directory To “//DSR/leads/data/pdcs/” Syncrhonize The Contents Of The “//LEADS1/leads/data/config/” Directory To “//DSR/leads/data/config/” Sync Files to WWW – Runs Every Hour Synchronize The Contents Of The “//LEADS1/dsrweb/” Directory To “//DSR/dsrweb/” Synchronize The Contents Of The “//LEADS1/leads/data/pdcs/” Directory To “//DSR/leads/data/pdcs/” (Excluding “…/pdb/five-minute/”) Syncrhonize The Contents Of The “//LEADS1/leads/data/config/” Directory To “//DSR/leads/data/config/” LEADS1 SCHEDULED TASKS (Cont.)

Sync Remote Commands to CFEP – Runs Every Minute If A Web-Commanded Time Sync Has Been Requested, Synchronize The Command to The CFEP LEADS1 SCHEDULED TASKS (Cont.)

Important LEADS Log Files Include CFEP Error Logs – Use The CFEP Error Log Web Page To Peruse These Comms Log Files – Use The Comms Report Web Page To Peruse These r_script.debug – A Log File Of r_script Messages Maintained In The /leads/data/tmp Directory On The CFEP CAMSxx.backup – Not A Log File, But Backup Data Files Maintained In The /leads/data/backup Directory On The CFEP LOGS AND LOG ANALYSIS

This page intentionally left blank

Independently Of The Real-Time Operational Data Flows Of The LEADS/EMS System There Is A Set Of Data Exchange Activities Which We Group Under The Category Of "Synchronization“ When We Refer To Synchronization We Are Referring To The Maintaining Of Certain Data Sets In A 'Close Enough To The Same' State On Multiple Discrete Computers (I.E., Identical But With A Time Lag) There Are Three Main Sets Of Data Which Are Being Maintained In Such A 'Synchronized' Condition SYNCHRONIZATION

Results Of Communications With The Data Loggers In The Field Note That The “Results Of Communications” Is Quite Different Than The “Content Of Communications” This Is The Information About Success, And Failure, Of Just Doing The Talking – Not The Gathered Sensor Data Itself This Data Consists Of The Log Files Accumulated By The Communications Front End Processor (CFEP) System The //CFEP/leads/logs/ Directory Is Periodically Synchronized To Both The LEADS1 And The DSR Systems COMMS RESULTS

Processed Field Sensor/Monitor Data This Is The Stored Sensor Data After Real-time Calculations Made By The LEADS/EMS System As Well As Any Changes Made By Data Validators Or Other System Users With Authority To Make Such Changes This Data Set Is Further Segregated Into Two Main Components: Five-minute Data One-hour Data To Enable Access To The Extensive Set Of Web Enabled Reports The //LEADS1/leads/data/pdcs/ directory Hierarchy Is Synced To Both The Internal (DSR) And External (WWW) Web Servers With The Exception That The .../five-min/ Directory Hierarchy Is Excluded From The Sync To The WWW Machine In The Interest Of Reducing Network And CPU Overhead Since The Five-minute Data Reports Are Not Provided To The Public PROCESSED DATA

Static And Dynamic Web Pages And Supporting Data All Web Page Modifications Are Performed On The Leads1 System And Then Replicated (Sync'ed) To The DSR And WWW Systems There Are Three Files (INDEX.HTML, EXTERNAL.CFG, And INTERNAL.CFG) Which Are Excluded From This Synchronization Step These Files Configure The Behavior Of The System As Either An Internal Access Only DSR System Or As A Public Access WWW System WEB PAGES AND SUPPORT DATA

The Mechanism Used To Perform These Data Synchronization Operations Is The Unison Application Which Resides In /Leads/Bin, The Assorted Command Files ( *.Cmd ) Invoked Via Shortcuts From "Scheduled Tasks" And The Profile Definition Files (*.Prf) In "C:\documents And Settings\leadsadmin\.unison\“ Which Specify The Detailed Synchronization Configurations All Operations Are Performed Under Account "leadsadmin" And, Therefore, Target Directories Must Be Accessible Via A Shared/Mapable Directory With Modify/Read/Write Access Permissions Granted To "leadsadmin" Errors In The Synchronization Process Are Captured In Files In Directory "C:\documents And Settings\leadsadmin\" Named DANGER.README (Or Very Similar Names) With An Incrementing Number In The Event Of The Detection And Auto Error Recovery Of Multiple Failures Auto Error Recovery/Resume Operations Are Handled By A Scheduled Task Driven Script Which Renames DANGER.README This Allows Synchronizing To Resume But Does Not Clean Up Failed Sync Issues SYNCHRONIZATION MECHANISM

SYSMON • Provides A Method Of Determining Which LEADS Processes Are Running On The Local LEADS1 Machine • Divided Into Process Groups • Green Circle Means Process Is Running, Yellow Is In A Wait State, And Red Indicates The Process Is Not Currently Running • User Can Start Or Stop Process Groups, As Well As Perform A Reinitialization Of The System By Pressing The “Master Re-read Config” Button To Trigger The Master Process To Re-read The SYSTEM.CFG Configuration File

The Manual Validation Tool Is Always Run On The LEADS Primary Server To Provide Access For Users, A Remote Desktop Connection Is Used The RDC Provides A Mechanism Under Windows So That Users Can Run An Application Remotely And Have The Display Presented On Their Local Computer REMOTE DESKTOP CONNECTION

Refer to: “Appendix C – TS and RDC for ManVal” RDC (Cont.)

Data Collection Begins With The Real World If The Data Logger Is Misconfigured, Then Reality Is Misperceived And Misreported Likewise, If The LEADS/EMS System Is Misconfigured (Does Not Match The Data Logger Configuration), Then Reality Is Misperceived And Misreported The Site Initiation Form (SIF) Is Used To Record The Configuration Of A Monitoring Site SITE INITIATION FORM

The SIF Attempts To Capture All The Relevant Information Required To Define A Site, Collect Data, And Report Data To The Various End Users The Template Has Defaults For The Most Common Channel Assignments, Method Codes, And Units Codes As Well As Standard Data Flag Settings The Contents Of The SIF Are Transferred To The Various LEADS/EMS Configuration Files SITE INITIATION FORM (Cont.)

Please Refer To: “Appendix D – SIF TEMPLATE” SIF TEMPLATE

Mandatory Fields Are Denoted With A “” Without The Information From These Fields, The Site Cannot Be Configured Within LEADS/EMS Site Name A Descriptive Name For The Site; Primarily Seen On The Various Web Pages AQS Number The 9-Digit EPA-Style Site Identifier The First Two Digits Are The State Code The Next Three Digits Are The County Code The Last Four Digits Are The Site Identifier; These Are Used To Differentiate Between The Sites Located In The Same County The AQS Number Is A Primary Identifier For Data Storage SIF FIELDS

Project Manager The Name Of The Project Manager Overseeing This Site SIF Applicability Date The Date The Site Is Expected To Start Collecting Data With The Configuration Described On The Form Street Address The Physical Street Address Of The Monitoring Site Data Logger ID One To Four Digits; Allowable Range Is 1 To 9998 For Convenience Used As A Shorthand Reference To The AQS Number Many Data Logger ID’s Can Map To One AQS Number SIF FIELDS (Cont.)

Site Operator The Name Of The Person Responsible For Day-To-Day Site Maintenance Site Phone Number If There Is A Phone At The Site, Enter The Number Here If All Sites A Network Connected, This Is Optional Since All The Sites Use The Internet For Data Collection; If There Are Any Sites Added That Are Serviced By Standard Phone Lines And Modems, This Field Becomes Mandatory As It Is The Number Dialed For Data Collection City The Name Of City Where The Monitoring Site Is Located Zip Code The Zip Code For Where The Monitoring Site Is Located SIF FIELDS (Cont.)

Time Zone Time Zone The Monitoring Site Is Located In All Data Is Collected And Stored In UTC (Greenwich Mean Time); Many EPA Submissions And Web-Based Reports Use Local Standard Time; This Is The Key To Translating Between UTC and Local Standard Time Data Validator The Name Of The Person Responsible For Data Validation At The Site IP Address (Port For Serial Adapter) Mandatory For Internet Connections This Is The Internet Address Used For Data Collection SIF FIELDS (Cont.)

County Name (Code) The Code Should Match The Middle Portion Of The AQS Number Area Description (Code) This Is A Discriminator Used For Mapping Ozone And Reporting The Air Quality Index Due To The General Geographical Layout Of Monitoring Networks And The Close Spacing Of Sites In Large Metropolitan Areas, It Is Necessary To Have Some Way Of Discriminating Which Map Background To Use Most Networks Can Be (Arbitrarily) Divided Into Logical Areas SIF FIELDS (Cont.)

Region (Code) The EPA Region The Monitoring Site Is Located In Responsible Entity This The Entity That Operates And/Or Validates The Monitoring Site This Becomes Very Important When Outside Entities Are Brought Onto The System Location Method The Method Used To Determine The Site Latitude And Longitude Should Include The Datum And Any Other Critical Information SIF FIELDS (Cont.)

Contact Person The Name Of The Contact Person Representing The Responsible Entity Lat(itude) (Deg:Min:Sec) The Latitude Of The Monitoring Site In Degrees:Minutes:Seconds The EPA Rules For Monitoring Site Locations Correspond To An Accuracy Of About One Arc-Second North Is Considered Positive; All Sites In The Continental US Have Positive Latitudes Office Phone Number The Office Phone Number Of The Contact Person That Represents The Responsible Entity SIF FIELDS (Cont.)

Lon(gitude) (Deg:Min:Sec) The Longitude Of The Monitoring Site In Degrees:Minutes:Seconds The EPA Rules For Monitoring Site Locations Correspond To An Accuracy Of About One Arc-Second West Is Considered Negative; All Sites In The Continental US Have Negative Longitudes Cell Number The Cell Phone Number Of The Contact Person That Represents The Responsible Entity Elevation (Meters) The Elevation Of The Monitoring Site In Meters This Is Used To Determine The Standard Barometric Pressure Of The Site Which Is In Turn Used For Automated Met Data Processing SIF FIELDS (Cont.)

Site Photos (Date) The Date The Site Photos Were Taken The EPA Desires That Site Photos Be Updated Periodically (Especially If Instruments Are Added Or Removed); This Is A Convenient Reminder Of When The Last Photos Were Taken Comments Any General Comments About The Site The Remainder Of The Form Describes The Data Captured By The Data Logger These Are Crucial To Proper Processing And Display Of The Data Mistakes Or Errors In This Part Will Require Extraordinary Measures To Correct – Especially Once A Large Quantity Of Data Is Collected SIF FIELDS (Cont.)

Data Triplet This Is The Output Channel Configured On The Data Logger Each Parameter Transmitted By The Data Logger Must Have A Unique Output Channel It Is Strongly Recommended That Standard Output Channels Be Used Across The Data Loggers – This Greatly Simplifies Field Support Parameter Name A Mnemonic To Help With What Is Configured On The Data Triplet The Parameter Code Fully Defines What The Parameter Is, But Is Much Harder To Remember SIF FIELDS (Cont.)

Parameter Code The Five-Digit LEADS Parameter Code Generally The LEADS Parameter Code Is Identical To The EPA Parameter Code Every Parameter MUST Have A Parameter Code Defined Sometimes A Pseudo-Parameter Code Must Be Used – For Parameters That Are Not Registered With The EPA Data Is Stored In The Various LEADS/EMS Databases And Auxiliary Files Based On Parameter Code – This Makes The Parameter Code A Primary Data Identifier The Following EPA Web Site Can Be Used To Find Parameter Code Information (As Well As Other Information) http://www.epa.gov/ttn/airs/airsaqs/manuals/codedescs.htm SIF FIELDS (Cont.)

EPA AQS Method Code The Three-Digit EPA Method Code That Describes How The Instrument Makes Its Measurements This Information Is Mandatory For Data Submittal To The EPA Each Instrument Manufacturer That Has Received An EPA Equivalency Code For A Particular Instrument Will Place A Sticker On The Instrument Specifying The Reference And Equivalence Code Of That Particular Instrument The Last Three Digits Of The Equivalency Code Are The Method Code Unfortunately, Method Codes Are Not Unique – You Will Find The Same Three-Digit Method Code Applied To Different Instruments Measuring Radically Different Parameters The Following EPA Web Site Can Be Used To Search For Equivalency Codes http://www.epa.gov/ttn/amtic/criteria.html SIF FIELDS (Cont.)

EPA AQS POC The Parameter Occurrence Code The POC Is A Convenient Method For Tracking Multiple Measurements Of A Single Parameter At One Site Each Additional Instrument That Measures A Specific Pollutant At A Monitoring Site Is Assigned A New POC For Example, The First Ozone Monitor At A Site Will Have A POC Of 1; If A Second Ozone Monitor Is Installed, It Will Have A POC Of 2 Sometimes, An Instrument May Have Been Present At A Site In The Distant Past; Often, A New POC Will Be Assigned In This Case The POC Should NOT Be Used To Track Instrument Swaps The POC Is The Only Way To Differentiate Between The Same Data Measured By Multiple Instruments At A Site The POC Is A Primary Identifier For Data Storage In The LEADS/EMS Databases And Auxiliary Data Files SIF FIELDS (Cont.)

Monitor Manufacturer In Combination With A Specific Monitor Model, The Equivalency Code (If One Has Been Assigned By The EPA) Can Be Determined Monitor Model In Combination With A Monitor Manufacturer, The Equivalency Code (If One Has Been Assigned By The EPA) Can Be Determined Monitor Output Range And Units The Physical Output Range Of Each Instrument Installed At A Monitoring Site This Information Is Used To Program The Conversion Of Raw Measurements To Engineering Units Some Conversions Are Programmed Into The Data Logger, Others Are Done Centrally SIF FIELDS (Cont.)

Corresponding Measurement Range And Units What The Raw Measurements Are Expected To Be Converted Into These Descriptions Will Greatly Influence Calibration Limits (For Calibrated Parameters) The Expected Instrument Range Can Also Be Used To Establish Upper And/Or Lower Limit Checks Measured Units Code The Three-Digit LEADS Units Code That Describes The Units The Particular Parameter Is Measured In Generally The LEADS Units Code Is Identical To The EPA Units Code This Information Is Mandatory For Data Submittal To The EPA The LEADS/EMS Database Uses Implicit Units – Each Parameter Is Expected To Always Be Measured In The Same Units At Every Site There Is Leeway In Setting Up What Units You Measure In, But They Have To Remain Constant Across Sites (And Time) SIF FIELDS (Cont.)

The Remainder Of The Fields Are Data Flags That Influence Processing And Display Of The Data AQI A Flag Indicating Whether Or Not The Parameter Should Be Included In The Air Quality Index Calculation Some Sites Do Not Meet Siting Criteria And Should Be Excluded (i.e., Sites Located Near A Strong Point Source Which Causes Measurements To Always Be Higher Than Surrounding Ambient Conditions) Many Of The Web Pages And Background Processes Check This Flag Only Certain Parameters Go Into The Air Quality Index Ozone, Carbon Monoxide, Sulfur Dioxide, PM-10, And PM-2.5 Nitrogen Dioxide Could, But It Is Normally Not Included The Background Processing And Web Pages Default To NOT Include A Parameter In The AQI Calculations (And Some Image Processing) SIF FIELDS (Cont.)

AQS A Flag Indicating Whether Or Not The Parameter Is Expected To Be Reported To The EPA AQS Database This Should Influence The Data Validators To Emphasize Validation Of Data That Is Going To Be Reported To The EPA Over Data That Is Not The Web Pages And Other Processes Default To A Value Of “No – Do Not Expect To Send This To The EPA AQS Database” Currently, This Is Used Only As A Data Descriptor (No Actual Processing Is Controlled Based On This Flag); Future Processing May Use This Flag As A Control SIF FIELDS (Cont.)

AIRNow Controls Whether Or Not A Parameter Is Automatically Exported To The EPA AIRNow Web Site Only Certain Parameters Are Exported To AIRNow Ozone PM-10 PM-2.5 The Following Met Parameters Can Be Exported In Support Of Homeland Defense Resultant Wind Speed Resultant Wind Direction Outdoor Temperature (10 Meter) Relative Humidity Solar Radiation Barometric Pressure Precipitation The Background Processing And Web Pages Default To NOT Export A Parameter To AIRNow SIF FIELDS (Cont.)

Public A Flag Indicating Whether Or Not The Parameter Is Available To The General Public Certain Parameters Such As Internal Temperatures, Test Probes, Etc. Should Never Be Exposed To The General Public The Web Pages And Background Processing Default To Block Any Parameter From Public Consumption Regulatory Many Of The Web Pages Have The Capability To Add Embellishments Describing Whether Or Not A Particular Instrument Should Be Considered Regulatory This Flag Controls This The Web Pages Default To Showing Any Instrument As Non-Regulatory SIF FIELDS (Cont.)

Experimental A Flag Indicating Whether Or Not An Instrument Is Considered Experimental Under Certain Conditions, It May Be Desired To Install An Instrument At A Site That Is Experimental In Nature This Flag Will Trigger Embellishments On The Web Pages Indicating That A Particular Instrument Is Experimental Background Processes Will Exclude Any Instrument That Is Flagged As Experimental From Certain Averages And Indices The Web Pages And Background Processing Default To Assume That An Instrument Is NOT Considered Experimental SIF FIELDS (Cont.)

LEADS/EMS SYSTEM ADMINISTRATION