A Bayesian, Meta Cost-Benefit Model

1. A Bayesian, Meta Cost-Benefit Model John Roman, Ph.D. P. Mitchell Downey District of Columbia Crime Policy Institute Partnership for Greater Washington Research The Urban Institute The Brookings Institution The Brookings Institution June 30, 2010

2. Presentation Overview • Background • Existing Models • DCPI • Drug Courts • Meta-Analysis • Proposed Model • Analytic Strategy • Results • Next Steps

3. General Approach Estimate: • Costs of drug court in DC; • Benefits of reduced crime (general); • Expected benefits for a DC drug court population; • Effect of drug court on criminal behavior; • Translate effects into benefits, difference the costs.

4. Presentation Overview Background Existing Models DCPI Drug Courts Meta-Analysis Proposed Model Analytic Strategy Results Next Steps

5. The Origins of DCPI Funded by the Justice Grants Administration (JGA) in the Washington, D.C. Mayor’s office using Recovery Act funds. Key goal was to create an entity similar to the Washington State Institute of Public Policy (WSIPP). WSIPP is a non-partisan, non-profit that provides evidence to support the Legislature; Created in the late 1980s, WSIPP (under the leadership of Steve Aos and Roxanne Lieb) conducted research funded by the legislature to inform decision-making.

6. The Origins of DCPI In 1998, WSIPP researchers create a meta, cost-benefit model (often called the Aos model). Performs meta analysis across a range of policies and programs; Links to Washington state specific costs of operations; Monetizes outcomes; Prioritizes policy choices and makes recommendations to the legislature. In 2008, recommended funding several cost-effective initiatives. Savings from those programs allowed plans to build two new prisons to be shelved. Passed by the Legislature.

7. How Can Existing Models be Improved? The WSIPP model embodies a quantum leap is evidence-based policymaking. However, there are two (intractable) problems with the model: First, general problem in meta analysis that if the underlying studies are not identified, no aggregate causal relationship can be established; Second, the model does not account for uncertainty in several steps of the estimation process, nor does it account for uncertainty in combining multiple estimates;

8. Identification Issues in the Existing Model Consider the question of whether DC should implement an Adult Drug Court in Washington. In order to believe that there is a causal relationship between drug court participation and successful outcomes (Y) must believe: YT > YC and, that assignment is exogenous, such that in the absence of drug court YT = YC If assignment is not exogenous and YT ≠ YC then no causal attribution is possible. Problem gets worse with aggregation. We have not solved this identification problem.

9. Integrating Uncertainty into the Model If model is not identified, and causal attribution is not possible, must include all sources of uncertainty into the model. Two types of uncertainty: Estimation. Each step in the process should yield both a point estimate and a distribution. Aggregation. Final results should integrate uncertainty from each estimate. WSIPP model produces a single point estimate with no confidence interval. Cannot determine probabilistically whether expected outcomes will occur. Problem akin to comparison of fingerprints and DNA

10. Integrating Uncertainty into the Model, con’t Why isn’t uncertainty included in the WSIPP model? The problem was intractable using classical techniques at the time the model was conceived due to limitations in computing power. Also, difficult to revisit estimates in a policy environment. Solution: Bayesian inference allows distributions rather than ‘moments’ to be aggregated, maximizing how much uncertainty is included in the model.

11. Donald Rumsfeld on the Importance of Incorporating Uncertainty "Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.“

12. Other Benefits of a Bayesian, Meta, Cost-Benefit Analysis The model is designed to be replicable; Does not requires abundant reputational capital with stakeholders, because; It allows for stakeholder input; Can be adjusted in real-time to test different assumptions; All assumptions are transparent. Reduces the number of stages of estimation.

13. Presentation Overview Background Existing Models DCPI Drug Courts Meta-Analysis Proposed Model Analytic Strategy Results Next Steps

14. What is the District of Columbia Crime Policy Institute (DCPI)? A nonpartisan, public policy research organization focused on crime and justice policy in Washington, DC. DCPI’s mission is to support improvements in the administration of justice and public safety policies, through evidence-based research.  Collaboration of UI and Brookings researchers to inform long-term strategic and short-term operations research on juvenile and criminal justice mattres. Goal is timely, practitioner-oriented research to develop and implement evidence-based crime policy in DC.

15. What will DCPI Do? Year 1 DCPI activities include: Develop a cost-benefit model to identify cost-effective DC based interventions; Develop a research library (DCCrimePolicy.org); Conduct independent research: An Evaluation of the Mayor’s Focused Improvement Area Initiative; A Study of Promising Practices at the Metropolitan Police Department; A Study to Understand the Impact of Pre-Trial Detention on Defendants and its Implications for Evidence-Based Practice

16. Presentation Overview Background Existing Models DCPI Drug Courts Meta-Analysis Proposed Model Analytic Strategy Results Next Steps

17. What are Drug Courts? Drug courts use court-based treatment as an alternative to incarceration. Begun in Miami in 1989, drug courts are specialized dockets that process drug-involved offenders. Client behavior closely monitored by a judge. Key premises: Drug involved offenders would commit fewer crimes if they desisted from drug use. Relapse is part of recovery. Treatment participation can be encouraged (coerced?).

18. Do Drug Courts Work? Yes? Can’t randomly assign courts to have a drug court or not. So next best studies find client outcomes are moderately better using; RCTs in a single site; Meta-analyses of many studies with varying designs; Simulation studies using population data. Generally, recidivism is reduced 10-15 percent. Potential for large net social benefits.

19. Presentation Overview Background Existing Models DCPI Drug Courts Meta-Analysis Proposed Model Analytic Strategy Results Next Steps

20. What is Meta Analysis? Policymakers seem most convinced by meta-analysis as it uses evidence from many settings. Steps in meta analysis are to: Calculate an effect size (standardized mean effect size comparing T and C); Sum across studies (weighting by the inverse of the variance of each effect size); Account for heterogeneity; Apply ad hoc weights.

22. What is Meta, Cost Analysis? Translate effect size into net benefits to account for: Savings to law enforcement - reducing new crime; Savings to courts – not prosecuting new offenders Savings to corrections - not locking those people up; Savings to those who are not victimized; Add in costs of new policies and programs.

23. Presentation Overview Background Existing Models DCPI Drug Courts Meta-Analysis Proposed Model Analytic Strategy Results Next Steps

24. What is Meta, Cost Analysis (Again)? What are the criticisms? Garbage in, garbage out (identification); No uncertainty in these estimates. Can’t solve identification issues at this time; Can introduce additional uncertainty. How? Use Bayesian inference rather than classical (inferential, repeated sample) statistics. Analogous to measuring using the metric system. Represents the same fact, but standardized Base 10 can be more efficient than Base Whatever.

25. Why go Bayesian (Statistically) Accounts for uncertainty in the presence of common problems: Multi-stage model. Output of one stage is next stage’s input. Bayesian inference takes a weighted average of all output (weighted by the probability) versus just using the mode (frequentist). Non-symmetric distributions. Default is ‘mode’ in frequentist inference. Presence of Missing Data. Again, uses distribution rather than MI or listwise deletion.

26. Why Should DCPI Go Bayesian? Policy Buy-In. Presents data more intuitively (e.g. weather forecaster does not predict an expected mean precipitation of 0.5 inches with a standard deviation of .25). Allow Policymaker input. (e.g. can provide information about prior beliefs that can improve model performance (effect of policy changes on various problems, changes in crime patterns in DC, demographic changes, etc.). Flexibility. Can adapt to all manner of common problems in inference (missing data, etc.); Can’t combine multiple estimates with uncertainty without going Bayes. Replaces sensitivity analyses.

27. Presentation Overview Background Existing Models DCPI Drug Courts Meta-Analysis Proposed Model Analytic Strategy Results Next Steps

28. Analytic Strategy: General Estimate: • Costs of drug court in DC; • Benefits of reduced crime (general); • Expected benefits for a DC drug court population; • Effect of drug court on criminal behavior; • Translate effects into benefits, difference the costs.

29. Analytic Strategy: Costs • Taken from a prior study of a DC drug court (Harrell, et al. 1998) • Inflation adjusted to get present day per participant costs of drug court operations • Estimates $11,500 per participant (for entire process) • Limitations: • Based on FY 1995 costs – could have changed • Assumes same size drug court (returns to scale) • Advantages: • DC specific

30. Analytic Strategy: Valuing Benefits of Reduced Crime The cost of crime is the product of the price of crime (P) and the quantity of crime (Q), so: Costi= Pi*Qi Where i’s are the categories of crime costs. • Law enforcement costs of investigating crime; • Court costs of processing crime; • Corrections costs of supervision; • Victims costs of being victimized. Main source of uncertainty: price of crime to victims. Extant estimates produce point estimates, Roman (2009) includes distribution of prices.

31. Analytic Strategy: Estimates of the Price of Crime to Victims

32. Analytic Strategy: Calculating Benefits The goal of this analysis is to create an estimate of the distribution of net harms averted. Calculate reduced crime (impact) • From large scale, multi-site evaluation of drug courts find: • the rate of re-arrest among comparison group (not participating in drug court); • the distribution of crimes committed by those comparisons who were re-arrested; • the proportion of arrests that led to incarceration.

33. Analytic Strategy: Benefits (cont.) In the next step, we will estimate how many crimes are prevented by drug court. • For each arrest prevented, estimate what the crime would have been by sampling from the distribution of crimes committed by controls; • Estimate the price of that crime by sampling from Roman (2009); • Find the probability that would have led to incarceration (calculate costs of incarceration); • These last two prices are the benefits of preventing one crime.

34. Analytic Strategy: Impact Estimates • Meta-analysis • Data: • From Shaffer (2006) • Coded typical meta-data (effect size, research design, length of follow-up, sample size, etc.) • Called drug courts which had been evaluated and interviewed about policies and practices • Our approach: • Bayesian linear regression of estimated effect size (correlation between re-arrest and drug court participation) • Condition effect size on court and study characteristics

35. Analytic Strategy: Impact Estimates • Variable selection (court traits): • Preliminary regressions of effect size on theoretically meaningful characteristics (treatment type, program design, response to graduation/failure, eligibility, etc.) • Selected those which were significant: violent offenders eligible and motivation required for enrollment • Variable selection (study traits): • Length of follow-up; • Methodology (random trial, matching, etc.); • Similarity between treatment and control groups (race, age, gender, prior criminal history); • Composition of control group (dropouts, ineligible, etc.).

36. Analytic Strategy: Final Comments • Missing data • Sample size • Pearson’s Correlation Coefficient (unconditional/zero-order correlation) • Follow-up time

37. Presentation Overview Background Existing Models DCPI Drug Courts Meta-Analysis Proposed Model Analytic Strategy Results Next Steps

38. Results: Effect Sizes, Tests Eligibility Rules Does not Require Motivation Requires Motivation Allows Violent Offenders Good Effects Bad Effects

39. Results: Effect Sizes, Probability of a Large Effect Does not Require Motivation Requires Motivation Allows Violent Offenders Good Effects Bad Effects

40. Results: Effect Size Over Time Allows Violent Offenders Does not Require Motivation Requires Motivation

41. Densities of Change in Rearrest Reduction in Rearrest | Increase in Rearrest

42. Results: Averted Arrests

43. Why Effect Size is Misleading

44. Results: Averted Arrests

45. Results: Distribution of Net Benefits

46. Costs Per Crime

47. Presentation Overview Background Existing Models DCPI Drug Courts Meta-Analysis Proposed Model Analytic Strategy Results Next Steps

48. The Last Word Finalize the estimates of costs of drug court in DC; Code other alternatives to incarceration strategies; Code other interventions; Rank policy priorities; Link to fiscal policy.

49. The Last Word "I would not say that the future is necessarily less predictable than the past. I think the past was not predictable when it started." - Donald Rumsfeld “All models are wrong, some are useful." - George E.P. Box and Norman Draper

50. Full Analytic Strategy • Distribution of possible Effect Sizes • Distribution of possible recidivism rates • Difference between control recid. rate and estimate recid. rate is effect of drug court • Distribution of crimes prevented • Distribution of prices of those crimes (and associated incarceration) • Benefits of drug court are P*Q • Subtract costs of participation to get net benefits of drug court