Evaluating Creative Teaching Practices and Collaborative Learning Environments

1. Evaluating Creative Teaching Practices and Collaborative Learning Environments Ann Mastergeorge and Ingrid Roberson UCLA Graduate School of Education & Information StudiesCenter for the Study of EvaluationNational Center for Research on Evaluation, Standards, and Student Testing Annual AERA Convention April 1, 2002 New Orleans, LA

2. Overview • The Construct of Engagement in School Reform • Methodological Issues in Measuring Engagement in School Reform • Discussion and Implications of Engagement for Classroom Practice

3. Introductionto Creative Learning Communities • Supports school reform in schools across the country serving at-risk student populations • Promotes change centered on creative teaching practices and collaborative learning environments • Emphasizes the enhancement in students’ joy of learning in engaged learning environments

4. The Construct of Engagementin School Reform • Procedural Engagement • Focuses on student on-task behavior in learning activities • Emphasizes student concentration and completion of learning activities • Substantive Engagement • Focuses on the emotional quality of student’s involvement in learning activities • Emphasizes student’s enjoyment of learning activities since it is self-motivating and self-reinforcing

5. Developing A Dynamic Model of Engagement • Substantive Engagement depends on both student and teacher engagement in classroom learning activities • Includes the social dimension of learning • Requires a high degree of reciprocity • Learning Environments and Teaching Practices construct opportunities for students to experience engagement • Communicates support and care • Allows for student choice and control • Models enthusiasm for learning

6. Creative Learning Community Learning Communities Teaching Communities Learning links Teaching links Shifts in attitude Shifts in attitude Classroom Collaboration Student Engagement Teacher Engagement

7. Methodological Issues in Measuring Engagement in School Reform • Multiple Methods of Evaluation • Utilizes quantitative and qualitative approaches • Implementation Process of School Reform • Impact of School Reform on Engagement • Employs a theory of action • Provides focal points for the evaluation • Suggests a sequence in which various effects may be expected

8. Leadership & Goals Anchor in curriculum, instruction & assessment Attend to cultural and structural change Facilitate change Connect to student learning Build a shared vision School Environment Mobilize change Attend to changing attitudes, beliefs, & values Provide materials Provide Financial Support Build a community Provide Technical Support Embed in school culture Professional Development &Accountability Align with project goals Provide the type and timely information for project adjustments Promote and provide time for on-going professional inquiry Provide opportunities for teacher collaboration Develop measures aligned with creative teaching practices Creative Teaching Practice Embed in classroom practice Shift teacher attitudes, beliefs, and expectations Align with assessments (standardized and performance) Align with standards Link to curriculum Identify innovative practices Embed in school culture Engaged Student Learning Promote the “joy of learning” through exploration, collaboration, and interpretation Improve student attitudes towards learning

9. Measuring the Implementation Process • Features Five Dimensions of School Reform Framework • Goals • Leadership • School Environment (Culture and Structures) • Professional Development • Accountability • Focuses on the implementation of creative teaching practices • Teacher Survey • Teacher Interview • Classroom Practice Observation Protocol

10. Teacher Survey

11. Teacher Interview

12. Measuring the Impact on Engagement • Features both Procedural and Substantive Engagement Theory • Focuses on Student Engagement • Teacher Survey • Teacher Interview • Student Survey • Classroom Practice Observation Protocol

13. Teacher Survey

14. Teacher Interview

15. Student Survey

16. Classroom Practice Observation Protocol • Sensitive to initial changes in teaching practices • Appropriate for innovative and creative teaching practices • Captures students’ attention to and enjoyment of learning activities

17. Dimensions of Classroom Practice • Challenge of Lesson Activity • Implementation of Lesson Activity • Procedural Engagement • Substantive Engagement • Quality of Instructional Discussion • Quality of Instructional Feedback

20. Challenge of Lesson Activity Exemplar Teacher: Let’s think about all of the different conditions. Student: My experiment needs more water. Teacher: I heard someone else say too cold… so cold or warm. Student: Ours is darker and theirs is lighter [amount of light] . Teacher: Can you think of anything else? Student: I think we should put less water and less worms.. with the worms they will just dry, will shrivel and do nothing and the water will just get everything damp. Teacher: What do you think is the most important thing to make the leaves decompose? Student: Dry, the leaves crumble up. It’s like they turn into dirt. Teacher: Maybe one thing we can do this week, half put water and the other half put no water and keep everything else the same.(CO.10.08.YR2)

23. Implementation of Lesson Activity Exemplar After reviewing both the proof reading process and scoring rubric (both clearly written on chart paper), teacher runs through following example. Dear Curator: I really love the ring of the bell. I also thought the water wheel was cool. Finally, the machines were interesting to see. Thank You, Ms. X Teacher: Alright, so this letter… Based on that, where would you put them if you were the teacher? Student: 2, though it does have an introduction. Student: 2, it doesn’t give them a sense and is supposed to be in paragraphs. Teacher: Anybody agree or disagree and why? Student: It looks like a letter but it doesn’t sound like a letter (CO.01.03.YR2)

24. Procedural Engagement 1 2 3 4 NA 10-29% of the 30-59% of the 60-79% of the 80-100%of the students appear students appear students appear students appear Not to be procedurally to be procedurally to be procedurally to be procedurally appli- engaged. engaged. engaged. engaged. cable to this Example: Example: Example: Example: activity Few students Some students Most students are Almost all students are conduct their light conduct their science on-task, conducting on-task, trying to get the bulb experiments experiment but about their experiment. light bulb to work, using while the rest of the half of the class They write their different metal wires. students wander wanders around the science notes, They note their around the room, room or sits at their detailing their steps observations in their chatting with other desks chatting with and results. Most science journal. Most students about other students about students complete the students complete both the unrelated topics or unrelated topics or experiment and most experiment and their play with each stares out into space complete or nearly writing-up of the science other. day dreaming, complete their experiment in their science notes. journal. Approximately what percent of procedurally engaged?

25. Procedural Engagement Exemplar All students are on-task, attentive to the teacher. Some students raise their hands to answer teacher questions. Teacher: Is this a sentence? Student: Yes. Teacher: So what does it have to have at the beginning? Student: A capital letter. Teacher: What does it have to have at the end? Student: A question mark. Teacher: What do I have to do to show these words came out of my mouth? [Three students raise their hand]...come up to the board X. Student goes to board and puts quotation marks around the sentence. (CO.02.01.YR2)

26. Substantive Engagement 1 2 3 4 NA 60-79% of the 80-100% of the 10-29% of the 30-59% of the students appear students appear students appear students appear Not to be substantively to be substantively to be substantively to be substantively appli- engaged. engaged. engaged. engaged. cable to this Example: Example: Example: Example: activity Most student are actively Almost all student are actively Most students are About half of the engaged. Some students engaged. Many students shout, either not doing the student are not talk excitedly with their "It works!, I did it! or Its alive!" experiment, playing actively engaged, neighbors saying, "Did when their light bulb works. Some or talking with other even if they are you see that?, That is students smile, their eyes students or if conducting the pretty neat; Did yours do widening, as their light bulb goes conducting the experiment, their that?" Other students, on. Other students still trying experiment, they are attention is focused smile to themselves as review their notes or ask another rushing through it, elsewhere, listening or their light bulbs lights up, student, “Hey, can you help me?” being careless, talking to other while others still trying Near the end, students can be paying more attention students, playing with scrunch up their faces in heard talking with each other to unrelated talk. their pencils, looking deep concentration. about which metal worked better outside the window. as a conducter. Approximately what percent of students are substantively engaged?

27. Substantive Engagement Exemplar The students are all listening. They are riveted, looking at the teacher and the book intently. The teacher engages them in a discussion with 90% of the students participating. Students’ hands shoot up in the air waving back and forth frantically with wiggling fingers, going "oh, oh”. Teacher: What woke him up? Student: The sun. Teacher: X, what wakes you up? Student: My mom. Student: Sometimes my dog comes in and licks me. Student: Sometimes my brother. Student: My mom’s boyfriend hits me with a pillow. Student: My dad gets me up. Teacher: So our caterpillar comes out of the egg and how did he feel? He was what? Students: HUNGRY!!! (CO.02.03.YR2)

30. Quality of Instructional Discussion Exemplar • At one table, there are three rather lively boys and as each of them works on drawing two parallel lines the following discussion ensues: • Student 2: “That’s parallel,” referring to the littlest one’s two parallel • lines. • Student 4: “Can I make it parallel?” • Student 2: “That is parallel. It doesn’t matter the size of the other • line, if they don’t intersect, then they’re parallel.” • Student 1: “If they don’t intersect, they’re parallel.” • The little one is still unconvinced by the two boys, looking at his board with a confused expression. [Student 2] continues to teach the other student the concept of parallel lines, emphasizing the word intersect and then upon seeing the continued state of confusion lets out a loud sigh and starts again. He explains that the length of the line segment or how close they are doesn’t matter as long as they don’t intersect. • (CO.05.03.YR2)

33. Quality of Instructional Feedback Exemplar • Teacher: “Did you measure first? If you’re going to do me, put my name in, and then you’ll have to do an estimate.” • Both students quickly go to their papers to write down their estimates on a small piece of paper. • Teacher: “Now this is in meters,” upon seeing one student write down 5. • The student nods knowingly but continues to write down the number 5. • Teacher: “Do you think I’m 5 meters high?” • He looks at her questioningly, not understanding. So she takes the yardstick and starts to raise it slowly. • Teacher: “One meter.” Raises it a meter. “Two meters.” Raises it again. “Three meters.” • Student: “Oh, two!” He vigorously erases the number five and writes down the number two. (CO.06.01.YR2)

34. Comparison of All and Matched Mean Ratings

35. Comparison of Year 1 and Year 2 mean ratings by domain across all case study schools

36. Comparison of Year 1 and Year 2 matched mean ratings by domain across all case study schools

37. Discussion and Implications for Classroom Practice • Documenting Shifts in Classroom Practice from Year 1 to Year 2 • Issues in Accountability and Sustainability of School Reform • Re-focusing, Re-thinking, and Re-tooling Measures of Classroom Practice

38. Documenting Shifts in Classroom Practice • Overall, small effect on Teacher Practice (Effect Size of .33) and Student Engagement (Effect Size of .201) • Between Schools, difference in degree of implementation of creative practices • Professional development linked to promoted teaching practices • Assessments to monitor progress

39. Issues in Accountability and Sustainability in School Reform • Combining and creating assessments that are accessible for progress monitoring • Improvements • Inquiry • Creating and sustaining supportive infrastructures to sustain school reform • Leadership • Community Culture

40. Re-focusing, Re-thinking, and Re-tooling Measures of Classroom Practice • Re-focus efforts to include realistic expectations for reform implementation • Re-think the product and outcome model of classroom measures: measures should reflect process in action • Re-tool measures to reflect conceptual and procedural realities of classroom practice

41. Paper will be posted on the UCLA CSE/CRESST website: www.cse.ucla.edu • Ann Mastergeorge • Phone 530-754-9629 • E-mail ammastergeorge@ucdavis.edu • Ingrid Roberson • Phone 310-794-4404 • E-mail roberson@cse.ucla.edu