Are We Making Progress Yet? Linda A. Reddy, Ph.D. Rutgers University

1. Are We Making Progress Yet? Linda A. Reddy, Ph.D.Rutgers University

2. A rogue wave smashed the bridge of the ship, destroying all the instruments except the speedometer. The captain announced to the badly shaken crew, "We have no idea where we are and we don't have a clue where we are going, but we do know we are making record time."

3. Progress Monitoring is All About Change and Change is Hard

4. What is Progress Monitoring? • “A scientifically based practice that is used to assess students’ academic performance and evaluate the effectiveness of instruction” (National Center on Students Progress Monitoring) • A “seamless and flexible” approach ….. that can be used across students of different age and skill levels, different settings and curricula, and across school years (Wallace et al., 2007) • “A set of assessment procedures for determining the extent to which students are benefiting from classroom instruction and for monitoring effectiveness of curriculum” (The National Research Center on Learning Disabilities)

5. Historical Context of Progress Monitoring • Traditional progress monitoring has been used to identify one’s relative position within a group rather than evaluating individual progress across time (Deno, 1997) • The early practice was a means of establishing eligibility for certain classifications/programs (e.g., special education, gifted) • Originally, progress monitoring was based on “mastery measurement” • Criterion-referenced tests used to assess the mastery of specific curriculum objectives • Following mastery of specific objectives (skills), the teacher would provide instruction and assess the skill taught • This method of progress monitoring was reflected in common teaching practices (e.g., Wisconsin Instructional Design System, Precision Teaching)

6. Historical Context of Progress Monitoring ■ Questions arose - mastery measurement's assumption that a series of short-term objectives directly leads to broad-based competence (false sense of progress) • Resulted alternative ways of conceptualizing progress monitoring • Curriculum-Based Measurement (Deno, 1985) • Requires students to simultaneously integrate the various skills required for competent yearend performance on weekly test • As students learn the necessary components of the curriculum, their CBM score gradually increases • Slope can be used to quantify rate of learning and gauge students’ responsiveness to an instructional program or OTL • When “inadequate” responsiveness is revealed, program can be revised • NCLB (2001) • IDEA (2004) – LD assessment - removal of discrepancy criteria • Advent or “rebirth” of RtI

7. What is Student Social Behavior? • Broad global construct • Heterogeneous • Complex in nature and context • Lack of agreement

8. How “Student Social Behavior” is Defined? ■ School and interpersonal connectedness (attachments) (Bond et al., 2007) • Cooperation, assertion, self-control (Rutherford, DuPaul, & Jitendra, 2008) • Self and social awareness, responsible decision-making, self-management and relationship-management skills (Zins et al., 2004) • Physical and intellectual actions (e.g., hygiene, nutrition, physical activity, avoiding harmful substances, decision-making skills, creative thinking), social/emotional actions for managing oneself responsibly (e.g., self-control, time management), getting along with others (e.g., empathy, altruism, respect, conflict resolution), self-honesty, integrity, self-appraisal and continuous self-improvement (e.g., goal setting, problem solving, courage to try new things, persistence) (Snyder et al., 2010) • Cooperation (i.e., puts work materials or school property away), Assertion (introduces him / herself to new people without being told), and Self-control (control temper in conflict situations with adults). Level of aggression towards peers, helpfulness with other children, interactions with peers, involvement or exclusion by peers, level of anxious-fearfulness, and hyperactivity-distractibility (Rimm-Kaufmann & Chiu, 2007) • Awareness of self and others, positive attitudes and values, responsible decision making, social interaction skills (e.g. active listening, expressive communication, cooperation, negotiation, refusal, help seeking) (Payton et al., 2000)

9. Student Social Behavior and Academic Outcomes • A comprehensive K-5 school-based program targeting student behavior and character was linked to academic achievement, school attendance, and positive disciplinary outcomes (Snyder et al., 2010) • ADHD 1-4 grade students - Teacher ratings of reading gains following academic interventions related to improvements in social skills on the SSRS • Peer-mediated math interventions produced improvements in math fluency and self-control (Rutherford, DuPaul, & Jitendra, 2008) • Social problem-solving skills are related to cognitive problem-solving skills (Ben-Avie, & Ensign, 2003)

10. Student Social Behaviors and Mental Health Outcomes • Low school connectedness and high interpersonal conflict in early secondary school are linked to increased mental health problems and substance use in later schooling. • The presence of both school and social connectedness was associated with the lowered risk of depressive symptoms in students (Bond et al., 2007) • Good Behavior Game (GBG) universal prevention intervention in first grade classrooms predicted lower levels of antisocial behavior by middle school (Kellam & Anthony, 1998) • GBG intervention provided in 1-2 graders (n=1,196) significantly reduced the risk (50%) drug abuse/dependence, smoking, and antisocial behavior for adolescent and young adult (Kellam et. al., 2008)

11. Three Presentations: Some Shared Ingredients • Comprehensive and rigorous programs of research • Emphasis on student social behavior in schools • Informant is the teacher (general education or special education) • Reliance on teacher accuracy and cooperation • All tools are brief, user friendly and easy to administer • Present integrative assessment-intervention models • Assessment is dynamic, fluid, and continuous • Framed in a three tiered RtI model • Emphasis test scores (raw or scale score) used as a method for determining efficacy of interventions

12. Enhancing Individual Education Plans for Children with Attention-Deficit/Hyperactivity Disorder Using a Daily Report Card • Disorder-specific tool • ADHD representing a highly comorbid population • Assessment is fluid for daily use • Students receive immediate teacher feedback on their efforts • Can be used to develop positive student-teacher and student-parents interactions/relationships • Fosters parent and teacher communication and partnerships • Scores linked to home-based privileges • Tailored for individual student needs – linked to IEP and 504 Plan goals – critical for buy in • Assesses targeted behaviors within specific academic lessons (e.g., art, math, language arts, science etc.) • Includes benchmarks of specific social behavior within context • DRC raw scores discriminate between treatment conditions

13. Development and Validation of Progress Monitoring Tools for Social Behavior: Lessons from Project VIABLE ■ DBR - undergone significant development and investigation, until now there has been limited attention to the psychometric properties ■ Highly reliable and valid alternative to SDO, interviews, and behavior rating scales ■ Emphasis on specific time, place (context) and intended rater for assessing the behavior ■ Excellent tool for IEP Goals and 504 Plans for K-12 grade students ■ Terrific tool for minimizing effects of rater inference and retrospective judgments about student social behavior ■ Considerations for developing other DBR measures: • Rater bias; rater skill (error) • Feasibility • Consideration of base rates within context • Wording (molar/molecular, negative/positive) • Visual cues • Anchors • General outcome or individualized targets

14. Scale gradients – first study to examine a single item scale compared three scale gradients (categories – 6, 10, 14) Chafouleas et al. (2009) Educational and Psychological Measurement Illustrates that DBR are highly useful for measuring the effects of intervention on individual and classroom behavior problems DBR-SIS: 2 part web-based training program – frame of reference and rater error (accuracy) DBR-BASIS: web-based program for data entry, analysis, and graphing

15. Developing Change Sensitive Brief Behavior Rating Scales as Progress-Monitoring Tools for Social Behavior ■ Significant research on social skills (SSRS) serves as the spring board for the development of 4 brief behavior ratings scales for social skills, conduct, anxiety, and ADHD ■ Construct specificity - detect progress in specific area (skills) of social emotional functioning ■ Emphasis on change-sensitivity of constructs and items ■ Proposes that change sensitive item are identified by four data analytic methods [Odds ratio, Effect sizes, Paired and 2-sample T-tests, Interaction effect from a mixed-factorial ANOVA] ■ Emphasizes the assessment of both strength and problem behaviors “Dual lens” assessment approach – particularly important in the area of ED

16. Where to go from here? • Opportunities for measure development are endless! • Adopt a broad iterative test development and validation approach: • Unified model of validation (Messick, Cronbach) and… • Rigorously examine and validate test utility • Utility of test score(s) for informing decision making for intervention plans that improve students’ lives (meaningfully)

17. Test Utility Framework for Future Test Development and Validation “In developing a test, one must first decide on the proposed interpretations and uses - and the assumptions embedded in the proposed interpretations and uses. The chosen interpretations and uses then provide guidance on how to proceed in developing the test” (Kane, 2008; pp. 77).

18. Before Test Development and Validation • What is the specific construct(s) we intended to measure? • What is the intended purpose of the test? • Include a strong consumer participatory model for test development • What is the "end game” goal? • Test score utility (inference): • How can the test score(s) generalize to the function and process of the targeted social behavior in the classroom? • How can the test score(s) be used by the consumers to inform decision making for intervention plans: • individual student • classrooms • schools • school district level • We must go beyond student level assessment and focus on system level assessment and change

19. Where to go from here? • Adopt a Strong Program of External Validation • Acceptability and usefulness of consumers • Carefully examine consumers’ intended and unintended test score inferences • What are the extraneous factors that impact test score utility (inferences) for the consumer? • Do test scores change with targeted practical school interventions and how do test scores change? • Incremental validity over business as usual • What are the pragmatic factor (barriers) that may impede development and validation (e.g., time, cost, teacher sensitivity or bias towards constructs and items)?

20. Example Barriers for Future Measurement Development • Barriers: • Consumers’ perceptions of time, work, and cost • Consumers’ perceived utility • Progress monitoring instruments may not be sufficiently sensitive to monitor progress among high school students because of small growth rates in this age range • Progress monitoring tools must be validated for different age groups (e.g., secondary students) • Absence of school staff buy-in • School personnel misunderstanding and ineffective use of data

21. Example Facilitators for Future Measurement Development • Facilitators: • Committed administrative leadership and support • District-wide support • Ongoing professional development • Cohesive team membership • Team efficiency (e.g., clear operating standards) • Stakeholder agreement and shared vision regarding the change • Autonomous teachers • School psychologists serving as leaders, change agents, and consultants • Financial resources committed to the cause • Organizational restructuring – system change

22. Where to go from here?: How about teacher behavior? • What teachers do in the classroom influences students’ learning and behavior. • GBG intervention with 570 2-3 graders - reduced use of negative teacher remarks predicted improvements in on-task behavior and talking out behavior. • Improved student behavior mediated the impact of the intervention on the development of hyperactive and oppositional behavior (Leflot et al., 2010). • Teachers’ social and emotional competence are key to “creating a classroom climate that is more conducive to learning and that promotes positive developmental outcomes among students” (Jennings & Greenberg, 2009). • Teachers who interact with their students in supportive ways promote positive student social behavior (Wentzel, et al., 2010). • Problematic relationships between Kindergarten teachers and their students with behavior problems predict academic and behavior problems through eighth grade (Hamre & Pianta, 2001). • Students with ED are significantly influenced by teacher-student interactions (Sutherland et al., 2008; Reddy et al., 2009).

23. Future Measurement Development Projects • Tools for preschool, elementary school, and middle and high school that inform supports and interventions • Tools that are dynamic and flexible for 6-12 grade - multiple teachers and the complexity of settings (classroom, hallway, study hall, lunch, after school activities such as clubs and sports) • Tools tailored for specific disorders and conditions - example Aspergers, Autism, and Anxiety/Depression • Brief tools for teacher and student interactions (specific aspects of classroom and school climate) • Tools for teacher classroom practices – general and special education settings • Tools assess aspects of teachers' social emotional competency (e.g., anger management, efficacy, perceived risk of violence) • Tools that can link meaningful goals for IEPs and 504 plans

24. Future Measurement Projects - Opportunity for School Reform • By 2015 over 1/3 of the nation’s veteran teachers and school administrators (baby boomers) will be leaving the field of education. • This change in school personnel offer the nation tremendous opportunity for comprehensive school reform and innovation in teacher preparation and training across the nation. • The time is right for new progress monitor tools for students and teachers.

25. Thank you!