Llewellyn J. Cornelius, Ph.D.- University of Maryland, Ruth Young Center (RYC)

1. Court Related Research in Maryland: Lessons Learned from Research Conducted with the Administrative Offices of the Courts Llewellyn J. Cornelius, Ph.D.- University of Maryland, Ruth Young Center (RYC) David Crumpton, Ph.D. Administrative Office of the Courts Jeanne Bilanin, PhD. Institute for Governmental Service and Research (IGSR)- University of Maryland We would like to acknowledge both our collaborators in the Maryland Judiciary Research Consortium, our Colleagues here at the Ruth Young Center and our stakeholders across the state as such a complex endeavor would not have been possible without the concerted involvement of the many people who are part of this collaboration.

2. Overview • It is safe to say that this is a different type of presentation as the focus of this presentation is more about the collaboration of a large number of organizations and partners in the process of conducting evaluation research, than it is about presenting findings from one study.

3. Overview • The following areas will be covered in this presentation: • A short history of the collaboration process • An overview of the roles of the collaborators • An example of a cluster of projects that have been supported under this collaboration • Discussion of the challenges of balancing scientific methods with ever changing needs in the policy arena.

4. Goals of the presentation • Expose the audience to a multi-campus, multi-discipline collaboration model • Highlight the challenges of balancing scientific methods with ongoing policy and programmatic issues. • Highlight the complexities of working with a large range of stakeholders.

5. History • The University of Maryland has been involved in a multitude of academic state partnerships including work with the Department of Human Resources and Department of Health and Mental Hygiene

6. History • Much of this work has been influenced by the vision of Judge Robert M. Bell, Chief Judge of the Maryland Court of Appeals who issued an challenge to the Maryland Judiciary in 1996 to increase its focus on public outreach. This had led to the development of wide range of programs and the development of inter-disciplinary teams that uses multiple paradigms to assess the delivery of equitable services across the state.

7. History • This vision had set the stage of programs that pre-date the involvement of the Ruth Young Center in Court based evaluations. • The RYC collaboration with the Administrative Office of the Courts began in 2005 with an evaluation of the Foster Care Court Improvement Program that was conducted by Bruce DeForge. • By the summer of 2010, this Collaboration had expanded to an evaluation of 13 programs by the RYC and some 40+ programs by the Maryland Judiciary Research Consortium

8. History • The Maryland Judiciary Research Consortium is comprised of the Ruth Young Center here at the School of Social Work, the University of Maryland Law School, the Institute for Governmental Service and Research at the University of Maryland College Park, the University of Baltimore, Salisbury University, Morgan State University, Coppin State University and Bowie State University.

9. The Ruth Young Center collaboration team included: Kieva Bankins, MSW Charlotte Bright, PhD Jacqueline Booth, MA Leigh Casey, BA Trenette Clark, Ph.D Clara Daining, Ph.D. Rebecca Sander, PhD. Elizabeth Greeno, PhD. Corey Shdaimah, Ph.D. Marie Bailey Kloch, MSW Darnell Morris-Compton, MSW Naeem Shaikh, MSW Ninoosh Sadeghi, BA Llewellyn Cornelius, PhD. Bruce DeForge, Ph.D. History

10. History • This presentation focuses on a subset of those evaluations: • Evaluations of the continuum of Truancy Programs across the State (the Truancy Reduction Pilot Program (TRPP) in the Lower Eastern Shore), the Truancy Court Program in Baltimore City and the BSMART (Baltimore Students: Mediation About Reducing Truancy) program in Baltimore City.

11. MJRC Program evaluators Multi-disciplinary- social work, law, public health, sociology, psychology, business administration Multi-method research designs- Qualitative,, Quantitative Mixed methods, Community involved and Community Based Participatory Research. Examine key program concepts, purposes and goals to determine what can be evaluated within a given scope of work. Focus on what can and cannot be generalized within the limits of the social science paradigm AOC Program Administration Connect evaluations to locate, state and federal policy issues Use of stakeholders from the judiciary to advise the RYC regarding the design and implementation of the project evaluations Provide funding from the State of Maryland General Fund or through Federal grants. Collaborator Roles

12. Case Example: The Continuum of Truancy Research • The premise: The AOC expressed an interest in learning more about the processes that were involved in the administration of the continuum of Truancy Intervention/Prevention Programs across the state. • They were also interested in learning more about what factors contributed to the success of these interventions.

13. Case Example: The Continuum of Truancy Research • This led to development of a three year program evaluation (that is ending in December 2010) of the three types of truancy interventions. • This is a multi-institutional evaluation involving partners across the MJRC.

14. Case Example: The Continuum of Truancy Research • In particular they were interested in learning about three such programs: • A one session school based mediation program that focuses on school tardiness called BSMART Baltimore (Students: Mediation About Reducing Truancy) • A ten week early truancy intervention program called the Truancy Court Program (TCP) that is targeted to youth with 5-20 absences, and • A ninety day Truancy “Episode” based intervention called the Truancy Reduction Pilot Program (TRPP).

15. Case Example: The Continuum of Truancy Research • BSMART • BSMART is operated by the University of Maryland School of Law’s Center for Dispute Resolution (“C-DRUM”). Established in 2006, BSMART targets students who exhibit early patterns of five or more unexcused absences or instances of tardiness. BSMART program mediators work with students, parents and schools to improve communication and develop strategies in a unified effort to address factors that may be contributing to student truancy. Mediation is strictly voluntary and occurs only when the parent/guardian and teacher agree to participate. In conjunction with BCPSS, C-DRUM identified three schools as initial sites for the program.

16. Case Example: The Continuum of Truancy Research • TCP • TCP is operated by the University of Baltimore School of Law’s Center for Families, Children and the Courts (“CFCC”) at selected schools. • TCP involves weekly in-school sessions with a volunteer judge or master, a team of school representatives, a CFCC staff person, a law student, the student, and his/her family. • The model is based on an early intervention approach and targets students referred by the Baltimore City Public School System who are “soft” truants.

17. Case Example: The Continuum of Truancy Research • TCP • TCP targets students who have from 5 to 20 unexcused absences – in the belief that this group still has academic, social, and emotional connections to the school. • TCP participation lasts for ten weeks and is strictly voluntary on the part of the student and his/her family. Every week, the judge or master reviews each TCP participant’s file, speaks privately with the child and his/her parent/caregiver about the student’s school attendance, and inquires about difficulties encountered during the week. • In addition to the team that meets weekly with the student and his/her parent/caregiver, CFCC has established a TCP Mentor Program and a TCP Volunteer Program.

18. Case Example: The Continuum of Truancy Research • TCP • The TCP mentor program, coordinated by a full-time consultant, provides mentors and role models for both students and family members. The mentor coordinator works with the students during the TCP session and conducts phone calls each week with parents/guardians. • TCP currently operates in eight Baltimore City public schools. Eleven Circuit and District Court judges and masters have volunteered to participate in TCP in Baltimore City. TCP is also expanding to other jurisdictions in Maryland.

19. Case Example: The Continuum of Truancy Research • TRPP (in the 1st Judicial Circuit) • The TRPP was authorized and funded under the provisions of Chapter 551 of the Acts of 2004 and House Bill 1443. TRPP is a court-based truancy reduction intervention in the juvenile courts. As a ‘problem solving court”, the TRPP includes a special document of truancy cases that are heard by the same master/judge each month. At these monthly hearings, factors contributing to truant behavior are identified through comprehensive family assessments and it is as a result of these assessments that students and their families are referred to community resources to address these factors. Students in this program are monitored by the court for a minimum of 90 days. Students who demonstrated improved school attendance during these 90 days are “graduated” from the program.

20. Case Example: The Continuum of Truancy Research • In general, the evaluations of these program focused on: • Collecting data regarding school attendance, school absences, school behavior, academic performance and juvenile justice experience- pre and post intervention. • Interviewing program administrators in the school system, the courts and the department of social services regarding the program operated and the factors they believed contributed to program success. • Interviewing participants and their families regarding their satisfaction with the services that they received, barriers to participation in the programs and perceptions regarding what worked or did not work in these programs.

21. Case Example: The Continuum of Truancy Research • In general, the evaluations of these programs focused on: • Collecting data which described the availability of school and legal services on the community level to determine geographic barriers to obtaining services. • Interviews with senior program administrators to compare the intended goals of the programs vs. the actual program activities.

22. Case Example: The Continuum of Truancy Research • In general, evaluation methods included: • Conducting focus groups and key informant interviews to capture the views of stakeholders and senior administrators- using grounded theory to capture the patterns of themes that emerged from these interviews. • The administration of surveys to program participants to record their perceptions regarding program activities. • The abstraction/retrieval of school attendance and outcome data from the local school system, along with program participation data from the intervention site. • The abstraction, where relevant, of program data from the Department of Juvenile Services for youth that were adjudicated to DJS.

23. Case Example: The Continuum of Truancy Research • Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit • The process evaluation (Daining, et al, 2008) focused on describing the extent to which the program has operated as intentioned. The evaluation focused on: • Describing the program including its goals, organization and responsibilities of program personnel. • Describing the characteristics of the youth who participated in the program. • Describing the services that were delivered in the program • Presenting the perceptions of the stakeholders regarding what did or did not work well in the program; and • Describing the role of the courts in the TRPP operation.

24. Case Example: The Continuum of Truancy Research • Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit • Four data collection strategies were used for this process evaluation: • Administrative data relating to participant demographics, participation rates and court proceedings, • Stakeholder interviews with court personnel, and personnel from collaborating community agencies • Observations of truancy court hearing in each county, and archival data review.

25. Case Example: The Continuum of Truancy Research • Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit • Using a semi-structured interview guide, a total of 41 stakeholder interviews were conducted of judges and masters in the First Judicial Circuit, court administrators, family services departments personnel, offices of the clerk personnel, boards of education personnel, community service providers, DJS case management specialist and representatives from local management boards and health departments.

26. Case Example: The Continuum of Truancy Research • Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit • Administrative process data was abstracted from a centralized Ms Access database, where data was routinely collected over the course of the program regarding the demographics of the program participants, the number of days between truancy petition file and adjudication hearings, the length of time the youth participated in TRPP, the types of services recommended by the court (type of disposition at the TRPP hearing) and the reason for closing the case.

27. Case Example: The Continuum of Truancy Research • Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit • Court observations: a total of 72 truancy hearings were heard across the four counties (Somerset, Dorchester, Worcester and Wicomico) including initial TRPP hearings, ongoing case hearing and final review hearings.

28. Case Example: The Continuum of Truancy Research • Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit • Archival data review: archival data review included program reports, flow charts and TRPP procedural forms.

29. Case Example: The Continuum of Truancy Research • Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit • Key Findings: • 61 percent of the students who complete TRPP were deemed successful as indicated by improved school attendance in compliance with court recommendations. • Successful students were younger at the time of the petition filing on average than unsuccessful students. • There was no statistically significant difference in the length of TRPP involvement between successful students and unsuccessful students. • Grade point averages of middle and high school students completer increased from the time of petition to case closure.

30. Lessons learned • The lenses of evaluation influences what is collected and the analyses that are conducted- • Saying the primary outcome of a Truancy reduction program is improvement in school attendance- leads to the construction of one type of evaluation- saying that it also leads to improvements in behavior with peers, or positive self esteem leads to the construction of an alternative evaluation design.

31. Lessons learned • Early involvement of stakeholders and program partners is critical in minimizing measurement error and observation bias in program evaluations. • One of the challenges we faced was differences in operating paradigms- the paradigm of science in the design and implementation of studies and the paradigm of the active policy and program development. A change in a program design in the midst of the study may increase its program relevance but limit its generalization.

32. Lessons learned • Early involvement of stakeholders and program partners is critical in minimizing measurement error and observation bias in program evaluations. • A second challenge we worked through was ensuring that the team was clear regarding what was being evaluated- as terminology across disciplines has different meanings- for example- mandated may have very specific definition from a legal perspective as it is tied to a particular ruling or law, whereas mandated from a social science perspective may reflect more general guidance and principles.

33. Lessons learned • Available Resources Govern what can be evaluated. • All evaluations are governed by balancing scientific precision (for example, in terms of striving for designs that can measure cause and effect, designing sampling strategies that lead to the generation of statistically significant findings) with the resources at hand.