Bottom-Up vs. Top-Down Approaches to Scaling Up PALS

1. Two Multi-Site Randomized Control Trials: Bottom-Up vs. Top-Down Approaches to Scaling Up PALSIs Response to Word-Problem Intervention among Students with MD Moderated by Concurrent RD?Doug Fuchs and Lynn FuchsVanderbilt University

2. Bottom-Up vs. Top-Down Approaches to Scaling Up PALS Doug Fuchs, Kristen McMaster, Laura Saenz, Devin Kearns, Lynn Fuchs, Loulee Yen, Don Compton, and Chris Lemons Vanderbilt University Chris Schatschneider Florida State University R305G04104 Institute of Education Sciences

3. Purpose of PALS • Supplements the general education core program • Implemented 3 times per week in reading; 2 times per week in math • Creates a “routine” for teachers to differentiate instruction by creating many simultaneous peer-mediated lessons rather than one teacher-directed lesson • PALS Reading: kindergarten, first grade, grades 2-6, high school • PALS math: kindergarten, first grade, grades 2-6

4. PALS Research • Based on Juniper Gardens Classwide Peer Tutoring • Over 15 years of experimental research • Title I and Non-Title I schools • Urban and suburban schools • High, average, and low achievers • Students with learning disabilities • “Validated Practice” status (USDE, WWC, BEE)

5. Grades 2-6 PALS Partner ReadingParagraph ShrinkingPrediction Relay

6. Partner Reading • Conducted for 11-12 minutes • Stronger reader reads aloud for 5 minutes • Weaker reader reads same text aloud for 5 minutes • Weaker reader retells story for 1-2 minutes • Readers read quickly, correctly, and with expression • Coaches listen, correct mistakes, and mark points (1 point for each correctly read sentence and 10 points for story retell)

7. Paragraph Shrinking • Conducted for 10 minutes • For 5 minutes: • Stronger reader reads new text aloud, summarizing paragraph by paragraph • Name the most important who or what (1 point) • Name the most important thing about the who or what (1 point) • Shrink it to 10 or fewer words (1 point) • For next 5 minutes: • Weaker reader reads new text aloud, summarizing paragraph by paragraph (as above) • Coach listens, corrects mistakes, and marks points

8. Prediction Relay • 5 minutes, stronger reader read new text • Makes prediction (1) • Reads half page (1) • Checks prediction (1) • States main idea (3) • Makes new prediction • Continues to read • 5 minutes, weaker reader continues on in new text, with the same activities • Coach listens, corrects mistakes, and marks points

9. Two Kinds of PALS Research Randomized Controlled Trials Study schools include Title I and no Title I. Classrooms randomly assigned within schools to PALS and control groups. HA, AA, LA (including LD) students targeted in each classroom. Fidelity of treatment implementation. Individually administered pre-/posttests by trained examiners.

10. Improvement in Reading Improvement Over 16Weeks

11. Two Kinds of PALS Research School-Improvement Projects Title I schools implement PALS school-wide. Our TA funded by Nashville’s Title I office No fidelity of treatment assessed. Group administered high-stakes tests.

12. Report Card Scores Based on Students’ Performance on the TCAP (CTB/McGraw-Hill) GOWER Subject ‘93 ‘94 MATH 76.0 (61) 107.8 (12) READING 74.0 (60) 112.5 (25) LanguageArts 61.7 (60) 91.8 (33) Science 74.4 (58) 95.6 (24) Social Studies 60.1 (61) 81.4 (53) Note 1: A score of 100 means that students of a school are progressing at a rate equivalent to that of the national rate. Note 2: The numbers in parentheses represent Gower’s standing in relation to the Metro Schools’ other 66 elementary schools.

13. Typical Teacher Support in PALS Research • Support was ongoing and on-site: RAs gave in-class assistance 1x or 2x per wk during training and implementation. • RA support is costly, unlikely in wider implementations, an obstacle to scaling up. • Absent such support, quality of implementations suffer. • How to separate an intervention from its support system? How to scale-up (“export” the intervention from A to B) without researchers’ nurturance? • There’s also a matter of transcending time….

14. Purpose of the Present Study • With Grades 2-6 PALS as a “prop,” and students’ reading achievement as the criterion, does a “bottom-up” approach beat a “top-down” approach to scaling up? • Do these approaches affect teachers’ sustainability of PALS?

15. METHOD

16. Participating Sites

17. Participating Teachers • Two cohorts of 3rd-, 4th-, and 5th-grade teachers : • Cohort 1 • Entered study in 2006-07 • Cohort 2 • Entered study in 2007-08 • Two years of study participation: • Year 1 • Assigned randomly to PALS or Control • Year 2 • PALS Teachers selected Top Down or Bottom Up PALS • Control Teachers remained in Control group

18. Teachers by Study Group in Year 2

19. Students by Study Group in Year 2

20. Study Conditions: Year 1 • Control – Teachers implemented core language arts curriculum • PALS – Teachers implemented with fidelity: • 3 times/week for 35-40 min (about 54 sessions) • Coaches and Readers: higher-performing readers paired with lower-performing readers • Four PALS Activities: • Partner Reading (10 min) • Retell (2 min) • Paragraph Shrinking (10 min) • Prediction Relay (10 min)

21. Study Conditions: Year 2 • Control – Teachers implemented core language arts curriculum • “Top Down” (TD) PALS • Teachers did PALS “by the book” • Fidelity of PALS implementation was emphasized • “Bottom Up” (BU) PALS • Teachers implemented core components of PALS • Customization was strongly encouraged and supported

22. BU PALS: Core Elements 48 sessions minimum 35 minutes per session minimum 10 minutes of Partner Reading 10 minutes of Paragraph Shrinking A motivational peer reinforcement system 22

23. BU PALS: Requirements Teachers asked to: Conduct core elements of PALS as designed Develop changes Match to curriculum, students’ needs, teaching style Create a type of PALS for the long term 23

24. Student Measures Academic Measures Student Characteristics • Test of Word Reading Efficiency (TOWRE) • Woodcock Reading Mastery Test-Revised (WRMT-R) • Word Identification Subtest • Wide Range Achievement Test (WRAT) • Letter and Word Identification Subtests • Comprehensive Reading Assessment Battery (CRAB; 2 passages) • Oral reading (1 min & 3 min) • Comprehension (10 open-ended questions) • CBM Maze Task (2 passages) • Correct maze choices made in 2.5 min • Iowa Test of Basic Skills (ITBS) • Reading Comprehension • Vocabulary • Demographics • SWAN • Teachers rated each student’s abilities to focus attention, control activity, and inhibit impulses • Teacher ratings • Teachers rated each student’s effort in reading and behavior in the classroom

25. Teacher Measures Classroom Measures Teacher Characteristics PALS Calendars PALS Fidelity Language Arts Observation Classroom Atmosphere Rating Scale (Wehby) Survey of Enacted Curriculum (SEC): English and Language Arts Demographics Berends teacher survey (assesses school climate, teacher professional development, teacher efficacy, etc.)

26. Procedures • Pretesting (September-October) • PALS Workshops (September-October) • Year 1: All teachers attend same workshop • Year 2: Separate TD and BU workshops • PALS Implementation (~18 weeks) • Teachers implemented 3 times per week for 35-40 min • Weekly classroom visits from project staff • Three “booster” sessions for TD and BU PALS teachers • Two fidelity observations • Language arts observations in PALS and Control classrooms • 45-60 min • Momentary time sampling of a variety of reading instructional components • Supplementary field notes • Posttesting (March-May)

27. Scaling-Up PALS for Grades 2-6 Results

28. Organization of Study

29. Analysis Procedures • Create latent pretest and posttest variables combining 5 reading measures into 1 • Create a latent change score • Produces an “error-free” change value • Run 2-level HLM analyses • Outcome: Latent change score • Variables: Treatment condition (TD, BU, Control); Site (TN, MN, TX); latent pretest score • Random effects: Level 2 teacher effects; ICC = .10 • Test comparability of groups on variables plausibly related to selection of TD or BU

30. Descriptive Statistics

31. Regression Analysis

32. Effects for Study Groups by LA, AA, and HA Students

33. Effects for Study Groups by LA, AA, and HA Students * * *

34. Is Response to Word-Problem Intervention among Students with MD Moderated by Concurrent RD?Lynn Fuchs, Sarah Powell, Pamela Seethaler, Paul Cirino, Jack Fletcher, Doug Fuchs, Carol Hamlett, and Rebecca ZumetaVanderbilt University and University of HoustonJournal of Educational Psychology, 2009Grant #P01046261National Institute of Child Health and Human Development

36. Study Purposes • Examine the efficacy of tutoring protocols for remediating • Math fact deficits • Word problem deficits • Assess whether treatment efficacy is different for • Students with MD alone versus • Students with MDRD • Determine whether effects are comparable as a function of site • Nashville, where the tutoring protocols were developed • Houston, a site distal to the developers

37. Participants • 924 students screened in 63 classrooms in 18 schools in Nashville and Houston (similar sample size at each site) • Inclusion criteria: • WRAT-A: < 26th percentile • 5-item word-problem measure: score < 2 • At least 1 (of 2) WASI subtest T score: > 36 • 162 students eligible for the study; 133 students remained at posttesting • Blocking on site (Nashville and Houston) and MD status (MD vs. MDRD), students randomly assigned to tutoring conditions: • Math Facts Tutoring (“Math Flash”) • Word Problem Tutoring (“Pirate Math”) • Control

38. Participants • Treatment groups comparable on all variables • MD vs. MDRD differences (across treatment groups) as expected MD MDRD Age 9 9 Female 40% 48% Sub. Lunch 68% 90% Spec. Ed. 8% 28% WASI IQ 92 85 WRAT-A 88 81 WRAT-R 105 78

39. Examined Efficacy of Two Tutoring Protocols Both Tutoring Protocols • Delivered individually • 48 sessions: 3 per week for 16 weeks • 20-30 minutes per session • Scripted lessons, which tutors studied (not read) • Motivational system to ensure on-task behavior and hard, accurate work • Each session audiotaped; tapes sampled and coded for fidelity, which was high for both tutoring conditions

40. Examined Efficacy of Two Tutoring Protocols • The exclusive focus of Math Flash was math facts • The primary focus of Pirate Math was word problems • but it also addressed foundational skills (math facts, procedural calculations, and algebra skills)

41. Pirate Math Tutoring 48 sessions: 3 per week for 16 weeks 20-30 minutes per session Scripted lessons, which tutors study (not read) Four units Foundational Skills for Word Problems Total Word Problems Difference Word Problems Change Word Problems

42. Pirate Math: Introductory Unit • Teach students: • Efficient counting strategies to answer math facts • 2-digit procedural calculations • How to solve for X in addition and subtraction equations (a+b=c; x-y=z) • How to check work

43. Introductory Unit: Counting Up

44. Introductory Unit: Finding X in All 3 Positions of Equation • If X is at the end of a number sentence, do what the problem tells you to do (e.g., 3 + 2 = X; 6 – 2 = X) • If X is not at the end, and it’s an “X minus” problem, add (e.g., X – 2 = 4). • If X is not at the end, and it’s not a X minus problem, subtract (e.g., X + 2 = 8; 5 – X = 2; 7 + X = 12).

45. Introductory Unit: Checking Work

46. Remaining Units:Word-Problem Lessons Following Unit 1, four activities per session. 1. Flash-card warm up 2. Conceptual/strategic lesson using schema-broadening instruction 3. Sorting practice on identifying problem types 4. Paper/pencil review

47. 1. Math Fact Flash Card Warm Up 11 - 6 • Math Fact flash cards comprise 200 addition and subtraction facts • Sums 0-18 • Subtrahends 0-18 • Tutor shows flash card to student: Know it or Count Up! • If student answers correctly, flash card placed in correct pile. • If student answers incorrectly, tutor asks student to “Count Up”; once correct, goes in correct pile. • Student graphs score on graph. 4 + 5

48. 2. LessonPirate Math RUN • Students use “RUN” strategy for every word problem. • Students learn to circle relevant information directly in the text or picture/graph/chart.

49. 2. LessonPirate Math Setting Up Work • Write the equation that goes with the problem type. • Figure out what’s missing. Write X in your equation in the appropriate slot. • Figure out what numbers are known. Write those numbers in the appropriate slots. • Write the math signs. • Find X. • Make sure your answer has a number and a label.

50. 2. LessonProblem Types with Transfer • Problem types at grade 2: Total, Difference, and Change • Transfer features: • Irrelevant information • Money • Double-digit calculations • Finding relevant information in graphs and pictures. • Combining problem types.