Statistical Sampling Overview and Principles

1. Statistical Sampling Overview and Principles Alvin Binns 205-220-4522 Abinns@csallc.com

2. Scenario • Provider X is identified for billing excessive ambulance services. A decision was made to pull all his/her ambulance services for a specified two years period. Results: - 3,000 claims, 7,000 lines and $1.8M in payments.

3. Reasons for Sampling • Time • Cost • Available resources • Available staff

4. What is Sampling? • Sampling - is the selection of observations to acquire some knowledge of a statistical universe (population). • From the characteristics of samples, we can infer the characteristics of universes, if the sample is representative of the universe.

5. In order for statistics to be good estimates of parameters, they must, on average, return the value of the universe parameter When the expected value of a statistic equals a universe parameter, we call the statistic an unbiased estimator of that universe parameter How Do I Get a Representative Sample?

6. How do you ensure that your statistic is an unbiased estimator? How Do I Get a Representative Sample? RANDOMIZATION!!!

7. A sample that is randomly selected from a universe yields sample statistics that are unbiased estimates of the universe parameters Many software packages, such as SAS and RAT-STATS have a valid random number generator Randomization

8. Another idea behind random sampling is that each sampling unit has a known probability of being selected Probability Sampling

9. Sampling Terms • Universe : An event or things of interest that the researcher wishes to investigate. Eg. All Medicare beneficiaries that received a left heart catheterization from Dr. John Doe between January 1, 2007 and June 30, 2008 paid up to September 30, 2008.

10. Samples are usually drawn by taking a subset of sampling units from the total universe Sampling units are non-overlapping collection of elements from the universe that cover the entire universe (eg claims, beneficiaries) Sampling Terms

11. Estimation • We can infer the values of the universe from the sample by the use of estimation • Ideally, we would like gather information from the sample and then estimate that value for the entire universe • These estimates calculated from the sample data are called statistics

12. Statistics ENTITY Sample Estimates Census CHARACTERISTIC Statistic Estimates Parameter

13. In an simple random sample where we had sampled 100 units out of 1000, suppose we had a $5,000 total overpayment from the sample The Mean Total Overpayment would then be: Estimation

14. Why Should I Care? • HCFA Ruling 86-1 allows the use of statistical sampling for the purpose of estimating a provider’s overpayment to the Medicare trust fund • Thus, we can use sampling to estimate overpayments to providers and avoid having to review the entire universe!

15. CMS Sampling Guidelines • CMS guidelines for Statistical Sampling for Overpayment Estimation • Program Integrity Manual Section 3.10 • Some of the issues addressed are: • Methodologies • Sample Size • Estimation techniques

16. This replaces and clarifies (for older cases) the old HCFA Sampling Guidelines Appendix (CR 1363) “This program memorandum (PM) provides clarified guidance and direction for Medicare carriers to use when conducting statistical sampling for overpayment estimation. The attached replaces the prior Sampling Guidelines Appendix for reviews conducted after issuance of this PM. For reviews conducted prior to this issuance, the attached are a clarification to aid interpretation of the earlier instructions, particularly where specific numbers are suggested” Sampling Guidelines

17. Sampling Methodologies • Simple Random Sampling • Cluster Sampling • Stratified Sampling • Other Methodologies

18. Simple Random Sampling • This is the most straightforward method of sampling • X number of sampling units are randomly selected from Y total sampling units in the Universe • Each sampling unit has an equal probability of being selected

19. Cluster Sampling • A cluster sample is a probability sample in which each sampling unit is a collection, or cluster, of elements • A good example is the random selection of beneficiaries, then selecting all relevant claims from each beneficiary

20. Stratified Random Sampling • A stratified random sample is one obtained by separating the universe elements into non-overlapping groups, called strata, and then selecting a simple random sample from each stratum • An example of this would be samples involving multiple procedure codes, selecting simple random samples from each code • Stratified random sampling generally has less sampling variability that other sampling designs

21. Stratified Random Sampling – Proportional Allocation Example Universe = 1000 Units 99211 99212 99213 99214 99215 50 150 300 450 50 Sample = 100 Units 99211 99212 99213 99214 99215 5 15 30 45 5

22. PIM 3.10 – Sample Sizes • PIM 3.10 states about sample sizes: • “It is neither possible nor desirable to specify a minimum sample size that applies to all situations” • “…real-world economic constraints must be taken into account. As stated earlier, sampling is used when it is not administratively feasible to review every sampling unit in the target universe. In practice, sample sizes may be determined by available resources. That does not mean, however, that the resulting estimate of overpayment is not valid as long as proper procedures for the execution of probability sampling have been followed. A challenge to the validity of the sample that is sometimes made is that the particular sample size is too small to yield meaningful results. Such a challenge is without merit as it fails to take into account all of the other factors that are involved in the sample design”

23. PIM 3.10 – Sample Sizes • CSA procedure: • If we can, we like to pull at least 10% of the universe, however, this is not a rule that is set in stone • We must, at a minimum, pull at least 30 sampling units to satisfy distribution requirements through the central limit theorem

24. PIM 3.10 – Overpayment • PIM 3.10 also states: • “In most situations the lower limit of a one-sided 90 percent confidence interval should be used as the amount of overpayment to be demanded for recovery from the physician or supplier. The details of the calculation of this lower limit involve subtracting some multiple of the estimated standard error from the point estimate, thus yielding a lower figure.”

25. PIM 3.10 – Overpayment • It further states that: • “This procedure, which, through confidence interval estimation, incorporates the uncertainty inherent in the sample design, is a conservative method that works to the financial advantage of the physician or supplier. That is, it yields a demand amount for recovery that is very likely less than the true amount of overpayment, and it allows a reasonable recovery without requiring the tight precision that might be needed to support a demand for the point estimate. However, you are not precluded from demanding the point estimate where high precision has been achieved.”

26. PIM 3.10 – Overpayment • What we really do then is calculate the Mean Total Overpayment and subtract a multiple of the standard error from it to achieve the lower level of the confidence interval

27. PIM 3.10 – Overpayment • Below is the formula for the total variance for cluster sampling

28. PIM 3.10 – Overpayment • Look at how the overpayments work: OP w/ Small Variance (Large n) OP w/ Large Variance (Small n) 90% Upper Limit 90% Upper Limit $$ Overpayment Mean Total Overpayment Mean Total Overpayment 90% Lower Limit 90% Lower Limit

29. Analysis Variable : CLUSTAMT N Mean Std Dev Sum Minimum Maximum 20 17,462.81 30,551.07 34,9256.24 0.00 81,603.20 10 17,462.81 31,388.24 17,4628.12 0.00 81,603.20 5 18,323.58 35,639.19 91,617.92 0.00 81,603.20 Sample Size Comparison

30. Sample Size Univ. Size Mean Total Overpayment Std. Error 90% 1-sided Lower Bound 20 44 $768,363.73 221,995.08 $483,766.04 10 44 $768,363.73 383,912.92 $276,187.37 5 44 $806,237.70 660,329.49 $91,617.92 Estimation Of Total Amount Of Refund & It's Lower 1-sided 90% C.I. Difference if sample Size 5-10 beneficiary: $184,569.45 Difference in sample Size 10-20 beneficiary: $207,578.67

31. Large Sample Sizes Use when the expected overpayment is large Use in high profile cases Resource intensive Increase precision even more using stratified sampling plans Small Sample Sizes Use when the expected overpayment is small Use in routine, low $ cases Not as resource intensive Does not work as well for stratified sampling Bottom Line

32. Sub-samples • It is often beneficial to evaluate a sub-sample before moving to a full statistical sample. (sample size of about 30) • Get a good idea of the point estimate (Mean Total Overpayment). • Sampling for Consent Settlements.

33. Summary for Sampling • Define the Universe • Determine the sampling methodology • Create the sampling Frame • Determine sample size • Create your sample After Sampling review is completed • Perform overpayment Projection

34. Questions? Thank You!