American Educational Research Association New Orleans, LA, April 2002

Classroom Assignments as Indicators of Instructional QualityLindsay Clare Matsumura, Helen Garnier, Jenny PascalCRESST/UCLARosa ValdésLos Angeles Unified School District American Educational Research Association New Orleans, LA, April 2002

Presentation Overview • Reliability • Validity • Stability • Quality of assignments • Teachers’ perspectives on the data collection

Data Sources for the Pilot • 181 teachers (50 participated) • 3 language arts assignments (2 reading comprehension, 1 writing) • 4 samples of student work for each assignment (2 high, 2 medium) • Teacher response forms • Observations in 16 classrooms of teachers (who submitted assignments)

Dimensions of Quality • Cognitive challenge • Clarity of instructional goals • Clarity of grading criteria • Alignment of goals and task • Alignment of grading criteria and task • Overall quality

Reliability of Classroom Assignment Ratings • Elementary • Kappas significant at .01 to .001 levels (.39-.56) • Alphas ranged from .87 to .96 • 84% exact scale-point agreement between experts, range from 84% to 68% with novices • Secondary • Kappas significant at .01 to .001 levels (.32-.59) • Alphas ranged from .91 to .95 • 86% exact scale-point agreement between experts, range from 71% to 55% with novices

Relationship of the Quality of Observed Instruction and Teachers’ Assignments

Relationship of the Quality of Student Work and the Quality of Teachers’ Assignments

Feasibility: Number of Assignments and Raters • G-Coefficient = .46 for Elementary (most of the variation was within teacher) • G-Coefficient = .88 for Secondary (most of the variation was between teachers)

Estimated Variance Components and Percent of Variance Explained by Teacher, Assignment Type, and Rater (n = 26 Elementary Teachers)

Estimated Variance Components and Percent of Variance Explained by Teacher, Assignment Type, and Rater (n = 24 Secondary Teachers)

Quality of 4th Grade Writing Assignments

Sample 4th Grade Writing Assignment • Students wrote a paragraph on a topic of their choice (going to a concert, the zoo, etc.). Students were instructed to write a topic sentence with 4 supporting sentences. • Learning goals were “To write a good paragraph using good sentences, spelling, and punctuation” • Grading criteria were “Content, sentence structure, punctuation and spelling”

Quality of 7th Grade Writing Assignments

Sample 7th Grade Writing Assignment • Students wrote a 5-paragraph essay on their dreams for the future • Learning goals were “To teach students step by step how to write a 5-paragraph essay and to demonstrate the creativity and fun involved in essay writing” • Grading criteria were “Content is well explained, writers focus on what needs to be talked about, writing process is completely done”

Quality of 10th GradeWriting Assignments

Sample 10th Grade Writing Assignment • Students were asked to read the book, “I Know Why the Caged Bird Sings,” and write a 5-paragraph essay or a poem about “what freedom means to you” • Learning goals were “I wanted them to write a coherent and focused essay on freedom, or a poem capturing freedom, and go through the steps of the writing process” • Grading criteria were “They were given 2 grades: 1 for their ideas and creativity, and the other for development and mechanics”

Teacher Reaction to the Data Collection

Suggestions for Improving Data Collection Process • More time for completing the assignment packets: • “Give us more time. How about initiating the process at the beginning of the school year instead of a mere three weeks before you expect a completed package back? Give us that professional courtesy please!” • Better timing: • “Please don’t have them due the week of Thanksgiving right after grades were due. Bad timing!”

Explanation for Why the Rubric Was or Was Not Useful • 90% considered the rubric to be useful • “It causes one to really pause and reflect on the tasks one assigns to students. Are they complex? Focused? Are grading criteria explicit and clear? Are learning goals aligned? It’s a motivator for improvement should one be really honest with oneself.” • “It is always good to have a general measure of performance. I want to improve as a teacher and this kind of reflection offers my insight into my work.” • 10% considered it not to be as useful • “It is useful but with all that we do daily in our teaching can we really use this classroom rubric to self-evaluate for every assignment we give? Self evaluation a few times a year would be sufficient.”

Some Future Directions for This Research • Utility of the assignment scoring method for improving teacher self-reflection and practice • Influence on student learning and achievement

American Educational Research Association New Orleans, LA, April 2002