Fast Track Study Sample 445 intervention and 446 control subjects

1. ABSTRACT:Much empirical research has questioned the accuracy of clinical judgment and demonstrated that it is inferior to rule-based or actuarial procedures (e.g., Dawes, Faust, & Meehl, 1989; Goldberg, 1970). Nonetheless, many mental health workers consider the development of sound clinical judgment to be the cornerstone of their professional training and one of their most valuable assets in working with clients. Given that view, they, understandably, are reluctant to follow rule-based procedures that do not allow for individual discretion in the decision-making process. And, this position, too, is credible. There are several good reasons to doubt the ecological validity of the research on clinical judgment (see Rock, Bransford, Maisto, & Morey, 1987). Recent reviews have implicitly recognized the limitations of clinical judgment and provided guidelines for its appropriate use within the context of manualized treatments (Wilson, 1996). Along those lines, the present study empirically examined the impact of integrating clinical judgment into rule-based and standardized procedures in a prevention project for high-risk children and their families. This study seeks to determine the impact of staff members' use of clinical judgment in Fast Track. Based on recent advances in statistical theory (Murphy, van der Laan, Robins, & Conduct Problems Prevention Research Group, in press), it will rely on regression analyses to statistically model the multiple levels of clinical judgment in making home visit recommendations. It will then use a series of weighted analyses to control for each level when estimating intervention and control group differences on a variety of outcome measures. Because Fast Track specified that staff members use rule-based procedures in making home visit recommendations but allowed some staff member discretion, this study can provide a unique assessment of the impact of clinical judgment in an ecologically valid context. The Impact of Clinical Judgment in a Conduct Problems Prevention Project for High-Risk Children and their Families Jerry J. Maples, M.S., The Pennsylvania State University, Robert L. Nix, Ph.D., Vanderbilt University Susan A. Murphy, Ph.D., University of Michigan, Karen L. Bierman, The Pennsylvania State University and Conduct Problems Prevention Research Group Society for Prevention Research Annual Meetings, June 2001 Home visiting process measure and treatment recommendation guidelines Starting in second grade, an adaptive treatment regime for home visiting based on severity at the end of the previous semester was implemented. The severity score is the total score of six four-point items from the home visiting process measure. Lower scores indicate a higher severity and thus tre need for more home visits. Part of the protocol for the home visiting recommendation was that in exceptional cases, the staff were allowed to deviate from the rule in their recommendation. Over the second and third grades, the staff made 1379 recommendations for the number of home visits a family should receive. The table below shows the relationship between the actual home visiting recommendations and the recommendations based on the home visiting process measure guidelines. The staff’s recommendations and the home visiting process measure guidelines were in agreement 76.6% of the time. • Fast Track Study • Sample • 445 intervention and 446 control subjects • all children in top 10% of class on measures of aggression • children and their families recruited from schools in low income neighborhoods in Durham, Nashville, Central PA and Seattle • about one half of the families were minority and about two thirds of the children were boys • all children were in kindergarten at the beginning of the study • Procedure • multi-component treatment: reading tutoring, classroom-based, small-group and individualized social skills training, parent groups for caregivers, and home visits • in the first year, all intervention children and their families received the same treatment • in the second and third year, all children and their families received a reduced number of group sessions and the reading tutoring, home visits and individualized social skills training were assigned based on need and re-evaluated at the end of each semester. • Measures • Summer Interview: comprehensive assessment battery of parents and child functioning based on observational measures, interviewer ratings and written questionnaires • 1. Observed mother’s warmth and positive attention displayed towards child • 2. Observed mother’s harsh behavior towards child • 3. Mother’s symptoms of depression with the Center for Epidemiological Studies Depression scale • 4. Extent of mother’s involvement in her child’s education • 5. Mother’s endorsement of physical punishment as a means of addressing child behavior problems in six vignettes • 6. Mother’s use of consistent and appropriate limit setting • 7. Warmth a mother displays toward her child in the parent-child interaction task • 8. Tally of the number of stressors a family has experienced in the past year • Home Visiting Process Measure: staff ratings to assess six areas of family functioning • 1. Quality of parent-child relationship 4. Parent functioning • 2. Ability to use praise and positive attention 5. Family stability • appropriately 6. Issues in home-school relations • 3. Ability to use limit setting appropriately • Outcomes • School Behavior: teacher's report of 10 items assessing oppositional and aggression behavior at school at the end of third grade • Home Behavior: parent's report of 30 items assessing a variety of behavior at home on three separate days during the summer after third grade. We are using the oppositional and aggressive behavior subscale. What would have been the mean difference in home and school behavior measures between the control and intervention children had the recommendation rules for the home visiting portion been different? Scenario 1: What if home visiting recommendations were based on ratings by staff members and on judgments of staff members regarding their belief on what is best for the families? This scenario matches what was actually done in Fast Track. This is an ‘as implemented’ model. The regime for treatment recommendations allowed staff members to use first-hand knowledge and idiosyncratic information about a child and his or her family in making recommendations for extra home visits. For this analysis, we are accepting how the staff used their clinical judgment when making the home visiting recommendations. Scenario 2: What if home visiting recommendations were based on objective data only? In this scenario, the home visiting recommendations would be based on the objective summer interview measures. Because we are not positing a specific recommendation rule based on the objective measures, we will emphasize the outcomes that are consistent with staff treatment decisions that reflect the indirect use of the summer interview list as seen in the Fast Track. This is not a general evaluation on the use of summer interview measures to determine treatment, but only on the use of the summer interview measures to the extent they correlate with the home visiting recommendations in this data set. We are attempting to eliminate any clinical judgment that the staff may have added, intentional or not, when making the home visiting recommendations. Scenario 3: What if home visiting recommendations were based on ratings by staff members, but the staff members were required to follow the rating guidelines exactly? In this scenario, the home visiting recommendations would have been based on the home visiting process measure guidelines. We are allowing the staff members to use their clinical judgment when completing the severity questions in the process measure, rather than relying on comparable data collected in the standardized summer interviews, but we are not allowing staff to override project guidelines.

2. Models for Dynamic Treatment Regimes • How can we compare what would have been the treatment effect of the Fast Track program if the home visiting recommendations were done differently? We can use a statistical model based on the potential outcomes framework (Murphy et al., 2001). • Goal: We want to reweight the intervention children to reflect hypothetical experiments in which home visiting recommendations reflected the three different scenarios. • 1. Each semester t, we create a weight for subject j, Wj,t , from • We will discuss what is required to form the denominator and numerator of Wj,t for each of • the three scenarios below. • 2. For each semester, we need a model which best predicts the actual treatment recommendations as they occurred in Fast Track. In modeling the observational distribution of the data, it is important to identify the covariates that were most strongly correlated with the home visit recommendations regardless of their actual content. • In Fast Track, site, cohort, home visiting process measure and recommendation for home visits in the previous semester were the strongest predictors of recommendations for the current semester. The predicted probabilities from these logistic regression models were used to form the denominator of the weight Wj,t for all three scenarios. • 3. For each semester, the rule for treatment assignment may be deterministic or probabilistic and is allowed to depend only on prior severity measures. The treatment assignment rules and the severity measures on which they depend may change over time. • For the regime in Scenario 1, which uses the recommendations that actually happened in the Fast Track study, any covariate may be a ‘severity measure’. The predictive probabilities from logistic regression models for the recommendation level a child actually received will form the numerator of the weight Wj,t. Since this is the same model as the observational distribution for treatment assignment, all of the weights for the intervention children will be constant 1. • For the regime in Scenario 2, the severity measures were the eight summer interview measures. Logistic regressions were used to determine the relation between the actual recommendations and the objective measures as seen in Fast Track. Each family’s probability to be recommended for a low, medium or high number of home visit sessions was based on the family score of the eight objective summer interview measures. This predicted probability was used to form the numerator for Wj,t • The regime in Scenario 3 is a deterministic treatment assignment rule which depends on the guidelines from the home visit process measure. Once the home visit process measure score has been tabulated, the table indicates exactly how many home visiting sessions should be recommended. Thus, they would receive the indicated recommendation with probability = 1 and the other recommendations would have probability=0 for being assigned. If the recommendation matched the process measure guidelines, then the numerator of the weight Wj,t will be 1, otherwise the numerator will be 0. • 4. The final weight for each child, Wj, is the product of the child's weight over all four semesters. These weights are used in the outcome analyses. • 5. We used modified Weighted Least Squares to estimate the mean difference between the intervention and control children on home behavior and school behavior when the home visit recommendations reflected rules from the hypothetical experiment (Scenarios 1-3). Pretreatment levels of the outcome measures were controlled for. Results Scenario 1: What if home visiting recommendations were based on ratings by staff members and on judgments of staff members regarding their belief on what is best for the families? As implemented in Fast Track, the intervention program had a significant impact in lowering school behavior problems in high risk children. However, the impact for lowering home behavior problems in children was only marginally significant. The intervention children were .16 standard deviations lower in the school behavior measure than the control children. Scenario 2: What if home visiting recommendations were based on objective data only? Basing home visit recommendations in a manner that is consistent with the correlations between the summer interview measures and the home visiting recommendations in Fast Track would not have yielded a significant difference between control and intervention children for home or school behavior outcomes. Scenario 3: What if home visiting recommendations were based on ratings by staff members, but the staff members were required to follow the rating guidelines exactly? Basing the home visit recommendations on the guidelines set forth in the process measure would have resulted in a significant difference between control and intervention children. The intervention children would have been .32 standard deviations lower in the home behavior measure than the control children. • Conclusions • Staff members often appeared to make home visit recommendations based on their knowledge of a child’s overall functioning. • The home visit process measure was designed to assess a family’s need for home visits, which were intended to reduce children’s problem behaviors at home. Allowing staff members to override the guidelines set forth in the process measure reduced the effectiveness of the intervention on behavior problems at home. It would have been better to have required staff members to follow the guidelines exactly. • The home visit process measure was not designed to assess children’s functioning at school. Without a strong theory about how home visits might affect children’s functioning at school, allowing staff members to override the guidelines set forth in the home visit process measure would have increased the effectiveness of the intervention on behavior problems at school. For this outcome domain, it was better to have allowed staff members to to use discretion in making home visit recommendations. • This study suggests that clinical judgment is most beneficial before strong theory has been developed. The Conduct Problems Prevention Research Group (CPPRG) is composed of in alphabetical order: Karen L. Bierman (Pennsylvania State University) ,John D. Cie (Duke University) Kenneth A. Dodge (Duke University), Mart T. Greenberg (Pennsylvania State University), John E. Lochman (University of Alabama), Robert J. McMahon (University of Washington) , Ellen Pinderhughes (Vanderbilt University) This research was funded by : The Center for Prevention Methodology through grant P50-DA10075 from The National Institute for Drug Abuse