Process, Outcome and Cost Evaluation: Seeing the Whole Elephant

1. Process, Outcome and Cost Evaluation: Seeing the Whole Elephant American Evaluation Association November 3, 2006 Shannon M. Carey, Ph.D.

2. Presenters:Shannon Carey, Ph.D.Kimberly Pukstas, Ph.D. - MarylandGwen Marchand, M.S. – Michigan Katharina Wiest, Ph.D. - Indiana

3. NPC received grants from federal sources (NIJ and BJA) and from state sources (E.g., California AOC • To perform process, outcome and cost evaluations of 5 drug courts in OR and a statewide drug court cost evaluation in CA • Since then we’ve added drug courts in Maryland, Michigan, Indiana, New York, Nevada, Minnesota and Guam

4. What is a Drug Court? • The purpose of drug courts is to guide offenders identified as drug-addicted into treatment that will reduce drug dependence and improve the quality of life for offenders and their families.

5. What is a Drug Court? • Participants are closely supervised by a judge who is supported by a team of agency representatives that operate outside of their traditional adversarial roles including addiction treatment providers, prosecuting attorneys, public defenders, law enforcement officers, and parole and probation officers who work together to provide needed services

6. What are the main goals of Drug Court? • Reduce recidivism • Reduce substance use • Improve family/community/individual functioning

7. Drug Courts 10 Key Components (NDCI-1997) 16 Strategies (NDCI/BJA 2003) (See Handout)

8. Guam (N=100) Portland, OR (n= 600) L.A. (El Monte) (n=127) Orange (Santa Ana) (n=289) San Joaquin (n=202) Stanislaus (n=399) Average Age 38 34 32 32 36 33 Gender 72% Male 74% Male 75% Male 71% Male 61% Male 66% Male Race/ Ethnicity 76% Chamorro 24% F, W, K 74% White 9% Hisp 68% Hisp 28% White 45% Hisp 43% White 43% White 24% Hisp 31% AfAm 80% White 16% Hisp Drug of Choice 90% Meth/(“Ice”) Alc, MJ 29% Meth 21% Coc 49% Coc 33% Meth 38% Meth 26% Her 26% Coc 29% Coc 25% Meth 76% Meth 11% MJ Who Participates in Drug Court? Most common average: 2 arrests in the two years prior

9. Needed to create a methodology that: • Includes rigorous research design • Is reproducible (providing the same type of • information for cross-site comparison) • Is flexible across multiple sites (method is • responsive to different court characteristics) • Produces information useful to program • staff, policy makers and the research field Cross-Site Evaluation

10. To see the whole elephant: • (to produce useful information) • Process Evaluation • Outcome/Impact Evaluation • Cost Assessment (Cost-Benefit Analysis) Cross-Site Evaluation

11. Purpose: • To examine program policies and procedures • Determine if: • the program was implemented as intended • the program is serving its target population • To explain/interpret outcome and cost results Process Evaluation

12. Benefit: • Useful Information about program functioning • Contribute to program improvement • Increasing effectiveness for participants • Better Outcomes, Better Cost-Benefits Process Evaluation

13. How do we do it? • Interviews and focus groups about drug court process, policies, and staff and client successes and challenges (learn from the program!) (Drug Court Typology Interview Guide – www.npcresearch.com) • Document review (program policy manuals, budgets and grant proposals) • Databases, Paper files - Program participant characteristics: demographics, referral dates, exit status Process Evaluation

14. Purpose: Determine whether the program has improved participant outcomes as intended Outcome Evaluation: Within Program (services received, grad rate, completion in intended time-frame, factors that lead to graduation) Impact Evaluation: Outside/After Program (recidivism, subsequent treatment, social services, health care) Outcome/Impact Evaluation

15. Benefit: • Information about program effectiveness • Intended outcomes/impacts achieved • 2. What program characteristics (process) led to successful outcomes Outcome/Impact Evaluation

16. How do we do it? • Collect administrative data from databases (preferred) and paper files • Ask questions – Search for any available databases that track individuals (keep asking/ask in different ways) • Participant interviews over time? (Not usually) Outcome/Impact Evaluation

17. Type of Session Low Intensity Medium Intensity High Intensity Expert Opinion Stated Policy Administrative data Intensive Tracking Group 69 103 47.5 45 Individual 25 25 26 24 Acupuncture 51 51 44/31 29 Court Sessions 11 11 14 19 Outcome/Impact EvaluationNIJ – STOP DC Study

18. Data Needed: • Identifiers • Demographics • Drug Court entry and exit dates • Date of arrest and court case number • Date of referral to drug court program • Drug Court status on exit • If terminated, reason for termination • Dates of entry into each phase • Criminal justice status on exit • Dates of UAs (and other drug tests) • Dates of positive UAs (and other drug tests) • Dates of drug court sessions • Drugs of Choice (Primary and secondary) • Attitude toward treatment Outcome/Impact Evaluation

19. Data Needed (Cont.): • Dates of services received • General treatment issues • Rewards and Sanctions (Dates, types and duration) • Non-compliant behavior • Aftercare services (Dates and types) • (For Juvenile) School attendance status at entry and exit Impact Data: • Subsequent treatment episodes • Dates of re-arrest after entering the drug court program* • Probation start and end dates • Jail/Detention entry and exit dates • Prison start and end dates • Social and health services information • (For juveniles) School related data such as completion status Outcome/Impact Evaluation

20. Why Do Cost Analysis? Policymakers face tremendous challenges in providing cost-effective public services. Limited financial resources require difficult decisions to be made about resource allocation. Cost-benefit analysis is an effective tool to help with these decisions.

21. Policy Questions Cost-Benefit Analysis Can Help Answer 1. Which policies and programs are cost beneficial to the taxpayers? 2. How cost effective are alternative programs? 3. What are the hidden costs in existing programs?

22. Policy Questions Cost-Benefit Analysis Can Help Answer 4. Which expenditures provide taxpayers with the best return on their money? 5. How much does everyday “business as usual” actually cost in time and resources?

23. Cost Research Strategies • Costs and Benefits (Opportunity Resources) • Cost to taxpayer approach (Public Funds) • Transactional Cost Analysis

24. NPC Cost Methods Step 1: Determine the flow/process Step 2: Identify the transactions Step 3: Identify the agencies involved Step 4: Determine the resources used Step 5: Identify costs associated Step 6: Calculate cost results

25. Step 1: Determine the flow/process • (Process Evaluation) • DC program and “business-as-usual” • Interviews • Observation • Document review NPC Cost Methods

26. Step 2: Identify the transactions • Examine the process description from Step 1 • Examples of transactions: • Drug court hearings • Treatment sessions • Drug Tests • Re-arrests • Jail Time NPC Cost Methods

27. Step 3: Identify the agencies involved Interviews and Observations NPC Cost Methods

28. Step 4: Determine the resources used • (Outcome Evaluation) • Interviews, Observations, Admin Data, Files • Do this for each transaction • Example: court hearings • Time spent in court hearing • Time spent preparing for court hearing • Number of court hearings for each participant NPC Cost Methods

29. Step 5: Identify costs associated • Interviews and Budget Reviews • Direct Costs • Support Costs (% of direct costs) • Institutional Overhead Costs (% direct costs) (“Fully-Loaded Cost”) NPC Cost Methods

30. Step 6: Calculate cost results • Investment Cost • Net Investment • Outcome Costs • Net Outcome Costs • Total Difference (Savings – or not) NPC Cost Methods

31. Investment Costs Investment Costs - Costs for the case that led (or could have led) to participation in drug court Net Investment – Cost for case that led to drug court for drug court participants subtracted by the cost for same kind of case for comparison group members.

32. Net Investment by Transaction Portland, Oregon

33. Agency Range Net Investment by Agency California Average Net Investment Per Participant Superior Court ($464) ($79) – ($898) District Attorney ($235) $103 – ($523) Public Defender ($279) ($76) – ($448) Probation $697 $2,143 – ($632) Treatment Agencies $1918 $706 - $3,808 Law Enforcement ($44) $1,060 – ($1,033) Corrections $0 $0

34. Net Investment California

35. Outcome Costs Costs that occurred after drug court entry that were not associated with the program or the “eligible” case. Net Outcome Benefits – Cost of drug court participants subtracted from the cost of comparison group members.

36. California Net Outcome Benefits

37. Portland, OR Net Outcome Costs By Agency(per participant over 2 years)

38. Promising Practices • A single (or overseeing) treatment provider • High drug court team attendance at staffings • Court sessions start 1 every 2-3 weeks (start) • Treatment 2-3 times per week (start) • Drug tests 3 times per week (start) • Judges voluntary with no fixed term (or at least two years) • Minimum 6 months clean before graduation

40. Agency Net Outcome Benefits by Agency California Average Net Outcome Benefit Per Participant Superior Court ($46) $342 – ($227) District Attorney ($12) $148 – ($106) Range Public Defender ($19) $171 – ($103) Probation ($53) $474 – ($650) Treatment Agencies $637 $336 – ($59) Law Enforcement ($1,525) $620 – ($3,619) Corrections ($3,292) ($541) – ($5,377)

41. Baltimore, Maryland ($5000)

42. Juvenile Drug Court (OR) Savings over 2 years (per participant) All Drug Court Youth = $1000 per participant Graduates = $10,958 per graduate Net Outcome Benefits

43. (Subtract investment costs from outcome costs) • Portland, OR $1.2 million • Baltimore City, MD $758,000 • El Monte, CA $150,777 • Monterey (- $300,000) • Laguna Niguel, CA $107,652 • Santa Ana, CA $75,000 • San Joaquin, CA $1.2 million • Stanislaus, CA $800,000 Overall Net Benefits Per Year