QA101

1. QA101 Institute for Tribal Environmental Professionals

2. Overview of course: • Why QA? • Quality systems • Quality assurance activities • Quality control activities • Objectives for your measurements (QMPs) • QC calculations: precision, bias, and accuracy • QAPPs, QMPs, and other monkeys

3. Why QA? • The purpose of QA is to help you get the data you need to answer the questions you have as cheaply and quickly as possible • QA done right does save you time and money • QA makes you: • Plan ahead • Ensure everyone agrees to the • goals and equipment used, • what their jobs are, • the schedule, and • who reports what to whom when

4. Terminology • QA = quality assurance • QC = quality control • Quality System includes both QA activities and QC activities, which often overlap • A Quality System = a good management system, which will produce: • follow up on problems with permanent solutions, • complete documentation, • everyone knowing what their responsibilities are, • making sure all data and reports are reviewed before going out to the public or EPA, • Data that answers the question you are asking about your environment

5. Quality Assurance improving your program Quality Control training documentation Within the quality system:

6. Within the Quality System: • Quality Assurance, which are more management type activities, overseeing: • Quality Control, which are actual checks of instruments, checking battery voltage, etc. • QC is one part of QA • In a two-person program, both people will be doing QA and QC, just as part of doing a good job

7. Graded Approach for Air Monitoring Important Decisions • Category 1 projects are to compare against the NAAQS standard, and require all 24 elements • Category 2-3 projects would include looking at air pollution transport, air toxics, exploratory studies, or evaluating whether further measurements should be made, and require even fewer of the 24 elements of a QAPP • Category 4 projects are education and outreach and may make a few qualitative measurements and only require 6 elements of a QAPP (see example on course website) Less Important Decisions

8. EPA follows: American National Standard ANSI/ASQC E4-1994 • National consensus standard authorized by American National Standards Institute (ANSI) • Developed by American Society for Quality (ASQ) • Consistent with ISO standards (there are ISO standards for everything, tires, toilets, • EPA Guidance for Quality Assurance Project Plans: EPA QA/G-5 based on E4 (just so you know)

9. What this means for us: • Anyone expecting their data to be respected must have some form of quality system (remember this is just good management plus documentation, in a QAPP) • Anyone receiving grant funds from a federal agency must have a quality system • Even 1-person programs can have data that is legally and scientifically defensible by following these guidelines

10. John W. Smith Director Your Tribe Laboratory, if weighing filters or analyzing samples Alexandria Washington Tribal Environmental Program Manager EPA as appropriate Regina Lamb Tribal Air Quality Technician/QA Coordinator Michelle Winston Tribal Air Quality Specialist Example org chart:

11. Example of QA/QC in daily life: • Bank balances: the goal of our personal quality system is to not bounce checks • QA are the activities of depositing checks, checking bank balances every week, planning ahead about what bills to hold off on paying • QC is routinely balancing our checkbooks and calculating whether checks written or online payments are going to bounce

12. Quality Assurance starts with Project Organization

13. Who are the Decision-makers? • Tribal Council • Tribal Environmental Office director • EPA Region

14. Identify... • Governmental Entities, Contractors, and Key Individuals. • Roles and Responsibilities. • How often will these be done? • How will each person do their job? • To whom will they report?

15. Possible Contractors • Weighing or Analysis Laboratory • Audits of equipment (performance audits, in which they compare their hand-held standard device against your device and make sure they agree)

16. Program Manager:Roles & Responsibilities • Oversees monitoring project • Prepares or reviews quarterly & annual reports for submittal to EPA • Ensures staff is hired and trained

17. QA Coordinator (if there is only one person this is all YOU) • Operates and checks equipment (checking batteries, flow rate) • Prepares QA reports (this may be half a page, saying all is well, flow rates are stable and within the ranges in the owner’s manual) • Does data verification, validation and assessment (we will explain these terms more later, but they are basically checking your equipment, looking carefully at what you are doing and making sure the data are answering the question you have about your environment)

18. QA Coordinator: • Independence: EPA guidance (and common sense) dictate that there should always be a second person who reviews the data and is not involved in routine data-gathering. • If only 1 person, independence might be done by working with another dept. (water?) and exchanging an afternoon every two weeks to review each others’ data

19. Terms: • May sound complicated, but are basic common sense: • Knowing who is to do what, • When and how they are supposed to do it • Where they write down what they did • How they know if what they did shows that the equipment is operating as it is supposed to • Who is supposed to fix things and where they write down what they did

20. Quality Objectives Why are we doing this?

21. Project Objectives • Why are we making these measurements? • Is our air over a standard? • Obtain baseline data • To determine the need for additional monitoring • Health risk evaluation: Report to community, EPA, health officials

22. Measurement Quality Objectives • Precision (wiggle) • Bias (jump) • Accuracy (total error, or wiggle and jump together, “evened out” over time)

23. Precision Precision =“wiggle” (variability within many measurements of the same thing) • You are trying to estimate the variability within the population of “all” your measurements of the same thing (concentration) • Two ways to estimate precision for a single instrument • If you have enough equipment, side-by-side, can be two or more devices measuring the same concentration • If you have only one continuous instrument, you must estimate precision by how much the measurement fluctuates over time when it is measuring the same concentration?

24. Bias Bias=“jump” (consistent difference between your result and the “truth” or a standard) Bias = how far from “truth” (some handheld standard) you are, in terms of percentage Bias = your result – “truth” “truth” Expressed as a percentage, so multiply by 100. 0.03 would be 3% 0.11 would be 11%

25. QAPPs, QMPs, and other monkeys: Terminology and Requirements

26. Quality Management Tools Planning: QA Project Plans Doing: QA Project Plans Standard Operating Procedures QA Annual Report and Work Plans Checking: Management Assessments Technical Assessments Data Quality Assessment Data Validation and Verification

27. Quality Management Plans Usually can combine with QAPP; this is up to the grantor (may be US EPA Regional office) EPA Requirements for Quality Management Plans (QA/R-2)

28. QA Project Plans (QAPPs) Purpose: To document type and quality of data for environmental decisions; a blueprint for the requirements for collecting and accessing data Responsibility: Organization performing activity Documentation: Turbo-QAPP EPA Requirements for Quality Assurance Project Plans (QA/R-5) Guidance for Quality Assurance Project Plans (QA/G-5)

29. QA Project Plans (QAPPs) • explains how environmental data collection activities are planned, implemented, documented, and assessed (checked) • QA Project Plans are required when environmental data operations occur for: • Contracts, work assignments, delivery orders • Grants, cooperative agreements • Interagency agreements (when negotiated) • Tribal-EPA agreements providing funding • Responses to statutory or regulatory requirements and to consent agreements

30. Common Elements in All Systematic Planning Approaches • Questions to be answered: • Who is making the decision? • Why are data being collected? • What data are needed to make the decision? • Why does the decision maker need that type and quality of data? • How does the decision maker plan to use the data to make a defensible decision? • What are the "measures of success" for the project? • Get only the type, quantity, and quality of data necessary

31. Standard Operating Procedures (SOPs) Purpose: To document routine technical and administrative activities to ensure consistency in the quality of the product Responsibility: Appropriate technical personnel working with QA Manager Documentation: Guidance for the Preparation of Standard Operating Procedures (QA/G-6)

32. Standard Operating Procedures (SOPs) • Written documents that clearly describe what you will do • Detail who will do what when things go wrong • If different people do it, but follow the same SOP, it should result in the same answer • You can print out once, and then revise in pen and photocopy if different people do the work • Can be short, even ONE PAGE or less

33. Quality Control Plan, do, check, fix, write it down

34. QC: An Ongoing System • Measuring • Comparing with an objective, such as +-10% • Graphing it • Fixing it when needed Everything must be documented and, when significant, taken into account in the reports

35. Evaluate Where Things Can Go Wrong—and How To Check • Preparing for the field • Sampling in the field • Gathering numbers • Entering the data into computer spreadsheet • Reporting the data

36. Two Sources of Precision Error • Field • Lab Cows can be problems….

37. Estimating Precision in the Field • Estimate the random “wiggle” error • If you have more than one of the same type of instrument, place side-by-side (measuring the same sample of air within a room or small area) • If you only have one instrument make repeated measurements (same sample of air, quickly in time so the air does not “change”) • Verify that results are within limits (do they agree at all? Or are they perfectly the same?)

38. Blanks • Measure anything that affects the result outside of what you are measuring • May make the result greater (contamination) • Or decrease the result

39. Types of Blanks • For real-time measurements zero checks display the value with no air • Manual methods using filters must use field blanks—accompany “real” samples • Labs must measure their own blanks to assess whether there is any contamination in the lab • If it is possible that samples get damaged or contaminated during shipping use shipping blanks (trip blanks)

40. Field Blanks • Handled exactly as field samples • Some field blanks go everywhere field samples go • With each operator, site, procedure

41. An integral part of QA: Data Management “if you didn’t write it down, can you prove it happened?”

42. Six Elements of Data Management (1) Data processing (WHAT) (2) Data end use (WHY) (3) Data access (WHO) (4) Data distribution (WHERE) (5) Data storage and retrieval (HOW) (6) Data disposal (WHEN)

43. Types of Field Data Sheets • Site Data Sheets • document the site information • in site files and in computer • Sampler run data sheets • go into field • input info into database • Verification data sheets • any checks that you make of flow rate, temp, pressure, battery checks, zero checks

44. Before leaving office for the field: • Review the number of each item you will need, and bring backups (use a checklist) • Check field data sheet from previous visit to site • Ensure that there are enough materials for routine, field blanks, and collocated samples • Ensure there are enough field transport containers, ice substitutes, max/min thermometers, preprinted mailing labels, if mailing immediately

45. Write in pen and update the documents: • Continuously update the checklist in pen • make photocopy • put in “to-do” pile to • add the information to the database • change the form in the computer if appropriate • At the site draw a map on the field data sheet • Take digital photos-these are GREAT data management tools

46. Computer Backups • Every two weeks or as data are gathered—add “back up today” to calendar • Alternate between two sets of backup USB drives, use CDs for permanent archive data storage • Store archive disks in a relatively fireproof location (another building, garage) • Plan for the time and money it takes to save copies of files

47. Assessments = Audits, both internal and external Checking up on yourself

48. All Assessments: • Basically compare • What is actually being done in the field and the office • Against what is stated in the QAPP and SOPs • Can be done by you or someone outside your office

49. Describe in your QAPP • Number, frequency, and types of assessments • People or organizations doing the assessments • Schedule • Criteria for assessments (we will compare what we do against the SOP or QAPP) • Reporting and responsibility for follow-up (we will revise the SOP in pen for now, and retype it with changes within a week)

50. Responsibility for Follow-Up and Verification of Corrective Action CLOSE THE LOOP (fix problems and take action to make sure they do not happen again) And make sure it is documented