Presentation Transcript

1. Statistical Background for CBER Proposals for Acceptable Process Control Plans

   John Scott, Ph.D. Division of Biostatistics FDA / CBER / OBE October 19, 2012
2. Outline Statistical Quality Control overview Sampling for quality control Single-stage sampling plans Binomial plans Hypergeometric plans Double-stage sampling plans Binomial plans Hypergeometric plans Further possibilities
3. Statistical Quality Control Overview
4. What is quality control? Quality = “fitness for purpose” Many aspects to assessing quality, e.g. Performance, reliability, durability, serviceability, aesthetics, features, perceived quality, conformance to standards Much of the quality control literature centers on reduction of variability We’re focused on assessing conformance Non-conformance is the end result of uncontrolled variability
5. Methods of statistical quality control Statistical process control methods Describe and explain variability in an ongoing process Methods focus on graphical displays – simple plots, Pareto charts, control charts Experimental design methods Systematically assess sensitivity of outputs to changes in input Acceptance sampling methods Inspect and test final product for conformance
6. Stem and leaf plot Box plot Pareto chart Some SPC graphs
7. Control charts Control charts map samples of a product characteristic over time to identify when a process might be out of control Features: A center line corresponding to the average “in control” value Upper and lower control limits Out of control values or unusual runs may be signals for investigation or action
8. Control chart example
9. Sampling for QC Population
10. Conformance to standards The plateletpheresis and leukoreductionGuidances each recommend: Standards for QC testing of individual units A maximum acceptable proportion of non-conforming units A minimum confidence level Only process failures are counted as non-conforming Non-process failures (failures due to uncontrollable parameters) are replaced in QC testing
11. Leukoreduction standards
12. Plateletpheresis standards
13. Maximum process failure rates 5% for: Residual WBC content Content recovery / retention pH 25% for: Platelet yield
14. Minimum confidence levels In each case, the Guidances recommend establishing a process failure rate below the maximum “with 95% confidence” That means: if the true process failure rate for a given month is at the maximum allowable level, there’s: A 95% chance that QC testing will detect a problem, or A 5% chance that the problem will go erroneously undetected 95% confidence in a 95% success rate is called “95%/95% acceptance” 95% confidence in a 75% success rate is “95%/75% acceptance”
15. How to get to 95% confidence There’s an “easy” way to have 100% confidence: test every unit. E.g.: Suppose you produce 200 units of RBC, LR in a given month You test residual WBC content on every unit All but 8 units meet WBC < 5.0 x 106 The failure rate is 8/200 = 4% with 100% confidence When testing every unit is impractical (i.e. usually), you can use statistical sampling to get to 95% confidence
16. The population is what you want to draw a conclusion about E.g. this month’s RBC units Take a random sample, measure failure rate in the sample, use probability models to make inference about population rate Yields an estimate of population rate Estimate has some uncertainty because sample is incomplete The logic of statistical sampling
17. Acceptance sampling Acceptance sampling consists of taking a sample from a lot and making a decision to accept or reject the lot based on the sample Not the same as SPC for blood establishments A month’s worth of units is not a “lot” A month’s worth of units will not be discarded on failed QC Common methods of acceptance sampling do correspond to FDA recommendations
18. Acceptance sampling translations In order to use acceptance sampling terminology, we need a couple translations “Accept” = Pass QC testing for the month No further action required “Reject” = Fail QC testing for the month Launch a failure investigation
19. Single-stage sampling plans
20. Single-stage sampling plans In a given month, N units will be produced In a single-stage sampling plan: A sample of n units is tested If more than c process failures, reject; otherwise accept E.g. n = 60, c = 0 means: We test a sample of 60 units If no process failures, accept If at least 1 process failure, reject, launch a failure investigation n and c need to be chosen to provide recommended confidence
21. Binomial single-stage sampling The Guidances recommend that n and c be chosen such that the probability of accepting is at most 5% when the true failure rate is at the maximum allowed We need a probability model to calculate the probability of accepting For large N we usually use the binomial distribution Called “Type B sampling” in SPC literature The binomial distribution assumes a random sample is taken with replacement from an infinite population
22. *The binomial distribution Assume: The true failure rate is p The sample size is n The acceptance number is c Then, under binomial sampling, the probability of accepting is given by: For example, p = .05, n = 60, c = 0 gives a probability of accepting of
23. Operating characteristic curves For any sampling plan, we can ask: what’s the probability of accepting at any given true failure rate? Recall that the Guidances recommend, e.g., 5% probability of accepting at a true residual WBC failure rate of 5% We also care about the probability of accepting at a “good” failure rate An operating characteristic (OC) curve plots the probability of accepting against the true failure rate
24. OC curve examples
25. Practical binomial single-sampling Very few binomial single-stage sampling plans are useful in practice Once you choose c, the smallest allowable n is the best choice For 95%/95%: c = 0, n = 59* c = 1, n = 93 c = 2, n = 124 *note that the Guidances mention c = 0, n = 60; establishments often prefer the round number, but c = 0, n = 59 is also acceptable
26. Binomial sampling key points Should be used when N is large Also appropriate for process validation n and c should be chosen to meet 95/95 or 95/75: Larger values of c and n mean lower false positive rates
27. Hypergeometric single-stage sampling The binomial distribution assumes an infinite population You don’t produce an infinite number of units, but this works well enough for large N If the population is small, we can use the hypergeometric distribution instead Called “Type A sampling” in SPC literature The hypergeometric distribution assumes a random sample is taken without replacement from a finite population of size N
28. *The hypergeometric distribution Assume: The population size is N The population number of successes is m (m ≥ (1-p)N) The sample size is n The acceptance number is c Under hypergeometric sampling, the probability of accepting is given by:
29. Hypergeometric OC curve examples
30. Binomial vs. hypergeometric plans Hypergeometric plans have logistical difficulties: You need to know N (or put upper bound on N) QC process may change from month to month or component to component Binomial plans require larger samples As N gets large, difference between binomial and hypergeometric approaches narrows Almost no difference when N is at least 10 times bigger than n
31. Hypergeometric key points Useful when N is small Sampling plan depends on knowing N You may need to plan around an upper bound on N Can’t be used for process validation For each N, you can choose n and c to meet 95/95 or 95/75. E.g. for N = 100:
32. Single-stage sampling issues The sampling plan needs to be prespecified Not acceptable to plan on c=1, n=93, then get to 59 with no process failures and accept without further testing You need to sample in such a way that you’ll have enough tests to meet QC acceptance rule With a c=1, n=93 sampling plan, if you have 1 process failure in 80 tests in a month, 95%/95% hasn’t been met For small N, hypergeometric plans may involve testing almost 100% of units You may need to test consecutively from start of month Potentially problematic if something changes mid-month
33. Double-stage sampling plans
34. Double-stage sampling plans In a double-stage sampling plan: A sample of n1 units is tested If c or fewer process failures are observed, accept If c+2 or more process failures are observed, reject If c+1 process failures are observed: A second sample of n2 units is tested If no more process failures are observed, accept If one or more process failures are observed, reject c, n1 & n2 are chosen to meet recommended criteria (e.g. 95%/95%, 95%/75%)
35. Binomial double-stage plans For large N, we calculate probability of acceptance using the binomial distribution Some acceptable double-stage sampling plans:
36. Double vs. single-stage sampling
37. There are more choices with double-stage plans All of these meet 95%/95%: Flexibility in double-stage plans
38. Hypergeometric double-sampling As with single-stage sampling, binomial double-stage sampling is pretty efficient if N is large For small N, hypergeometric double-stage sampling may allow smaller samples You need to know N (or be able to put an upper bound on it) The calculations are a little complicated; Appendix A of the leukoreduction guidance provides a table
39. Hypergeometric double-stage sampling examples
40. Double-stage sampling issues As with single-stage sampling: Pre-specification required Need to ensure enough units are tested Note that many hypergeometric plans have a second stage of 100% (“ALL”) or almost 100% You can’t get to 100% if you skipped any For N=50, c=0, n1=31, n2=18, you may be done with testing two weeks into the month Will you catch issues late in the month? Supplemental testing may be advisable
41. Double-stage sampling key points Adopting a double-stage sampling plan: Gives you a chance to “rescue” a process in the event of a failed test Reduces risk of false positives Binomial plans are appropriate for large N Hypergeometric plans yield smaller sample sizes for small N Can’t be used for process validation Plans should be pre-specified
42. Other Possibilities
43. Multiple-stage sampling Double-stage sampling can be generalized: The second stage can be designed to allow for 1 or more process failures Third, fourth, or … stages can be added None of these techniques are likely to lead to practical sample sizes in the blood establishment setting
44. Chain sampling In chain sampling, a sample of size n would be taken each month If 0 process failures, accept If 2 process failures, reject If 1 process failure: Accept if there have been 0 process failures in the past i months Otherwise reject Hasn’t been proposed, to my knowledge
45. Scan statistics Scan statistics can be used to continuously monitor an ongoing process to look for event clusters How it works: N tests over a year Look at every possible sequence of m consecutive tests (a “window”) If there are more than k failures in a window, the process is out of control N, m and k chosen to achieve probability of declaring: In control at a “good” failure rate, p1 (e.g. 1%) Out of control at a “bad” failure rate, p2(e.g. 5%)
46. Scan statistic pros and cons A natural way to look at blood product QC Months are arbitrary (although c.f. CFR) Avoids issues related to non-random sampling Mathematically & logistically complex No free lunch! Added flexibility of the scan statistic comes at a price If you want 95%/95% for any scan window, large N (e.g. N > 1000) required, or false positives high Compare to 60/month binomial N = 720
47. Thanks!