Create Presentation
Download Presentation

Download Presentation

Verification of Performance Specifications

Download Presentation
## Verification of Performance Specifications

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Verification of Performance Specifications**An Advanced View of Method Validation Version 5.0, August 2012**Objectives**• Identify test classifications • Define what each validation experiment details for testing methods • Discuss what is recommended to perform each of the validation experiments for testing methods • Recognize how to evaluate data obtained from each of the validation experiments**Pre-Assessment Question #1**A rapid Human Immunodeficiency Virus (HIV) test would likely be classified as a: • High complexity, modified assay • Moderate complexity, unmodified assay • Food and Drug Administration (FDA)-approved, modified assay • Waived, FDA-approved, unmodified assay**Pre-Assessment Question #2**The precision of a test method gives information related to the method’s: • Systematic error • Comparison of results to a reference method • Reproducibility • Likelihood of being affected by hemolysis, lipemia and icterus • Both A and B**Pre-Assessment Question #3**When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals? • 20 • 18 • 16 • 15**Pre-Assessment Question #4**Which linear regression equation component gives information regarding constant bias? • y • x • m (slope) • b (intercept)**Selecting a Method**• Evaluate diagnostic tests • Characteristics of testing methods • References: Technical literature and manufacturer’s information • Select method of analysis • Validate method performance • Implement method • Perform tests with appropriate Quality Control (QC) and External Quality Assurance (EQA)**Method Validation**What is method validation? Why must we validate? When should we validate? What should we validate?**Method Validation (cont’d)**• Why is validation important? • Division of Acquired Immunodeficiency Syndrome (DAIDS) requirement • How important is it that the results produced by the testing method are reliable? • Shouldn’t the laboratory know the level of performance of an adopted test method?**Tests to Validate**Waived Non-waived • Unmodified FDA-approved • Modified and/or Non-FDA-approved**FDA Approval Resources**• Vendor • Publications • http://www.fda.gov/MedicalDevices/ProductsandMedicalProcedures/InVitroDiagnostics/LabTest/ucm126079.htm**Skill Check**What would you consider to be the complexity, per Clinical Laboratory Improvement Amendments (CLIA), of the glucose assay in the workbook? • Waived • Moderate • High**Skill Check**What would you consider to be the complexity of a rapid urine pregnancy assay? • Waived • Moderate • High**Skill Check**What would you consider to be the complexity of performing a manual white cell differential using a stained whole blood smear? • Waived • Moderate • High**Method Validation**• Before you begin: • Be sure you are familiar with the test method before starting • Know what to expect from the method (package insert, discussions with technical assistance, and field service representatives) • Do not include results outside of stated reportable ranges • Predict your findings; establish limits/evaluation criteria**Terms for Discussion**Central Tendency Dispersion**Terms for Discussion (cont’d)**Values Run**Error in Test Methods**• Some error is expected • Examples • Error must be managed • Understanding • Defining specifications of allowable error • Measurement**Total Error of Testing System**Total Allowable Error • CLIA Guidelines per analyte • Other Guidelines Systematic Error Random Error Total Error**Error Assessment**Random Error (RE) Systematic Error (SE) Total Error (TE) In one direction, cause results to be high or low In either direction, unpredictable Combined effect**Total Error Considerations**• Low End Performance Standards • Recommendations derived from upper portion of reportable range are more difficult to achieve at lower concentrations • Maximum Total Error Allowed • Considered to be 30% by David Rhoads, except for amplification methods**Systematic and Random Errors**• Systematic Error • Slope/Proportional error • Intercept/Constant error • Bias • Random Error • Mean • Standard deviation (SD) • Coefficient of variation (CV)**Tools for Use**Data-Crunching Tools Statistical calculators, graph paper Spreadsheets with calculations Validation Software (Westgard, Analyze-It, EP Evaluator)**How We Will Work Through This Module**• One quantitative test taken through the validation process • One qualitative method taken through the validation process**Reportable Range**Precision Accuracy Elements of Validation Reference Intervals Sensitivity Specificity**Precision**• Definition: Reproducibility • Gives information related to random error Introduction • 20 samples of same material (typically two levels; e.g., Glucose at 50 and 300 mg/dL) • Standard solutions • Control materials • Pools (short term only) What is needed • Repeat testing over short and long term (one day and 20 days, respectively) How we perform the testing**Precision: How We Evaluate the Data**• Mean • Standard deviation (SD) • Coefficient of Variation (CV) Calculate the following: • Short term: 0.25 of allowable total error • Long term: 0.33 of allowable total error What amount of random error is allowable, based on CLIA criteria?**Allowable Total Error Database**Link for: • Clinical Laboratory Improvement Amendments (CLIA) • College of American Pathologists (CAP) • Royal College of Pathologists of Australasia (RCPA) • Others http://www.dgrhoads.com/db2004/ae2004.php**Precision: Levey-Jennings (LJ) Charts**Values Run**Precision: How We Evaluate the Data**• Mean • SD • CV: More commonly used, allows for easier comparison How do we compare to manufacturer’s data?**Precision Example**90 mg/dL Mean of Level 1 Glucose CLIA Total Allowable Error 6 mg/dL or ± 10% Total Allowable Error Level 1 Glucose 0.1 x 90 = 9 mg/dL Random error allowed: 0.25 x total allowable 0.33 x total allowable Long-term precision Short-term precision 0.25 x 9 mg/dL 0.33 x 9 mg/dL 2.25 mg/dL 2.97 mg/dL**Activity**• Work with Levey-Jennings graph and data • Work with mean and standard deviation to calculate a coefficient of variation, as well as a mean and a coefficient of variation to calculate a standard deviation • Determine if precision data is acceptable**Accuracy**• Definition: How close to the true value • Comparison of methods • Gives information related to systematic error • Potential conflicts on interpretation of results (reference values) Introduction • 40 different specimens • Cover reportable range of method • Quality versus quantity What is needed • Duplicate measurements of each specimen on each method • Minimum of five days, prefer over 20 (since replicate testing is same) How we perform the testing**Accuracy: How We Evaluate the Data**Graph the Data: Difference plot Real time Comparison plot Calculate y = mx + b Test method on Y-axis b represents constant error m represents proportional error Reference (comparative) method on X-axis Shows analytical range of data, linearity of response over range and relationship between methods**Visual Inspection for Accuracy**(x1, y1) Test Method (x2, y2) Slope = (y2- y1) / (x2- x1) Intercept Reference Method**Accuracy: How We Evaluate the Data**• Slope: Usually not significantly different from 1 • Intercept: Not significantly different from 0 • Significant difference with Medical Decision Points**Calculate Appropriate Statistics**Slope • Measure of proportional bias • m = (y1-y2)/(x1-x2) or “rise/run” • Slope greater than 1 means the Y (Test) values are generally higher than the X (Comparative) values • Slope of 1.11 means the Y (Test) values are on average 11% higher than the X (Comparative) values**Calculate Appropriate Statistics (cont'd)**Intercept of the Line • Measure of constant bias between two methods • Y (Test) value at the point where the line crosses the Y axis • If Y intercept is 12, then all Y (Test) values are at least 12 units higher than the X (Comparative) values**Accuracy**What type of bias do you see?**Accuracy (cont’d)**Constant Bias Proportional Bias**Skill Check**Can a linear regression formula offer predictive value in relation to method comparisons? • Yes • No**Activity**• Create graph based on sample set • Determine slope from best-fit line • Determine Y-intercept from best-fit line • Explain the relationship between comparative and test results**Reportable Range / Linearity**• Definition: Lowest and highest test results that are reliable • Especially important with two point calibrations • Analytical Measurement Range (AMR) and derived Clinical Reportable Range (CRR) Introduction • Series of samples of known concentrations (e.g., standard solutions, EQA linearity sets) • Series of known dilutions of highly elevated specimen or spiked specimens; EQA specimens • At least four levels (five preferred) What is needed How we perform the testing • CLSI recommends four measurements of each specimen; three are sufficient**Reportable Range:How We Evaluate the Data**• Measured values on Y-axis versus • Known or assigned values on X-axis Plot mean values of: Visually inspect, draw best-fit line, estimate reportable range Compare with expected values (typically provided by manufacturer)**AMR vs. CRR**Analytical Measurement Range (AMR) Linearity Clinically Reportable Range (CRR) Allows for dilution or other preparatory steps beyond routine**Skill Check**If you do not have enough specimen to perform a dilution, upon which reportable range component must you rely? • AMR • CRR • Neither A or B • Both A and B**Linearity Materials**Utilizing the marketing materials from the two chemistry linearity kits in your handouts: • Determine which kit would be more appropriate for use with the chemistry assay you chose earlier • Explain your reasoning