Observing Classroom Instruction

1. Observing Classroom Instruction Sally Atkins-Burnett, Ph.D. Senior Researcher

2. Process for Developing Protocols • Review literature on fidelity • Review classroom observation tools • Review literature on quality instruction • Identify key features of curriculum

3. Observing Classroom Practice • Adherence to instructional practices • Exposure (dose) • Quality of delivery • Student engagement • Program differentiation

4. Methods of Collecting Classroom Data • Rating scales • Checklists • Time samples • Interactive coding systems • Narrative descriptions • Teacher logs or diaries • Teacher assignments • Student work • Interviews and surveys

5. Recording Observations with Interactive Coding Systems • Most widely used • Presence or absence of behaviors in a time sample • Use of a rubric or rating • Low inference • Key to reliable measurement is operational definition

6. Making It Reliable • Describe in behavioral terms –what would you see and hear • Offer both examples and non-examples (is and is not) or differentiate as needed • Use classroom video when training – use range of practice • Discuss disagreements and come to common understanding • Check for rater drift

7. Example of Item: “Asks close-ended question” • Description: Close ended questions are ones in which the teacher accepts only one correct answer. • Example: Q: What is 2+5? (only one answer)

8. Differentiating Categories Sometimes the question may have more than one answer but the teacher only accepts one answer: Q: How do you add 2+5? (more than one possible answer) But teacher only accepts one answer as correct and then moves on to other questions Answer: You start from 5 and add on 2. Others students could have solved it differently but teacher only accepts this one.

9. Areas of Math Instruction to Consider • Teacher-initiated instruction • Teacher responses or feedback • Student to student interaction • Student work • Activity setting • Materials • Types of Representations • Problem-Solving Approaches

10. Quality Instruction • Contingent feedback • Positive climate • Student engagement • Positive behavior management • High productivity • Higher order thinking

11. Identify Key Features • Review Publisher materials • Examine teacher manuals and student materials • Outline key features to assess – may get developer /publisher input • Consider best way to collect information on key features • Triangulate data using multiple-methods

12. References • American Association for the Advancement of Science (AAAS) Project 2061 (Evaluating curricular materials). Retrieved June 2, 2007 from http://www.project2061.org/publications/textbook/default.htm • Baker, Jean A. “Teacher-Student Interaction in Urban At-Risk Classrooms: Differential Behavior, Relationship Quality, and Student Satisfaction with School.” The Elementary School Journal, vol. 100, no 1, 1999, pp. 57-70. • Clare, Lindsay. “Using Teachers’ Assignments as an Indicator of Classroom Practice.” CSE Technical Report 532. Los Angeles, CA: CRESST University of California. November 2000. • Hilberg, R. Soleste, Hersh C. Waxman, and Roland G. Tharp. “Introduction: Purposes and Perspectives on Classroom Observation Research. In Hersh C. Waxman, Roland G. Tharp, and R. Soleste Hilberg (Eds.). Observational Research in U. S. Classrooms: New Approaches for Understanding Cultural and Linguistic Diversity (pp. 1-20). New York: Cambridge University Press. 2004.

13. References • Huntley, Mary Ann. “Operationalizing the Concept of ‘Fidelity of Implementation’ for NSF-Funded Mathematics Curricula.” Presentation at the National Science Foundation K-12 Mathematics, Science & Technology Curriculum Developers Conference. 2005. Retrieved November 1, 2006 from www.agiweb.org/education/nsf2005/speakers.html • Melde, Chris, Finn-aage Esbensen, and Karin Tusinski. “Addressing Program Fidelity Using Onsite Observations and Program Provider Descriptions of Program Delivery.” Evaluation Review, vol. 30, no. 6, 2006, pp. 714-740. • Mowbray, Carol T., Mark C. Holter, Gregory B. Teague, and Deborah Bybee. “Fidelity Criteria: Development, Measurement, and Validation.” American Journal of Evaluation, vol. 24, no. 3, 2003, pp. 315-340.

14. References • National Research Council. “On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations.” Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials. Jere Confrey and Vicki Stohl (Eds.). Mathematical Sciences Education Board, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. 2004. • O'Donnell, C. L. (2006). Fidelity of implementation in scaling up highly rated science curriculum units. In A. Benbow (Ed.) NSF K-12 Mathematics, Science, and Technology Curriculum Developers' Conference 2005: Dealing with Challenges to Effective and Widespread Implementation of IMD Curricula. Alexandria, VA: American Geological Institute.

15. References • O’Donnell, Carol. “Fidelity of implementation in scaling up highly rated science curriculum units.” Presentation at NSF K-12 Mathematics, Science, and Technology Curriculum Developers Conference. 2005. Retrieved November 1, 2006 from http://www.gwu.edu/%7Escale-up/document/ODonnell_Fidelity_NSFIMD.pdf • O'Donnell, C.L. & Lynch, S. (2005, April). Examining the fidelity of implementation of highly rated middle school science curriculum materials. Paper presented at the Annual Meeting of the American Educational Research Association, Montreal, Canada. • Padron, Y. N., and H. C. Waxman. “Classroom observations of the Five Standards of Effective Teaching in urban classrooms with ELLs.” Teaching and Change, vol. 7, no. 1, 1999, pp. 79-100. • Pianta, Robert C., Karen M. La Paro, and Bridget K. Hamre. “CLASS Classroom Assessment Scoring System Manual K-3 Version.” Charlottesville, VA: Center for Advanced Study of Teaching and Learning.

16. References • Rivera, Hector, Roland G. Tharp, D. Youpa, S. Dalton, G. Guardino, and S. Lasky. “ASOS: Activity Setting Observation System coding and rulebook.” Santa Cruz: Center for Research on Education, Diversity and Excellence, University of California. 1999. • Rowan, Brian, Eric Camburn, and Richard Correnti. “Using Teacher Logs to Measure the Enacted Curriculum: A Study of Literacy Teaching in Third Grade Classrooms.” The Elementary School Journal, vol. 105, no. 1, 2004, pp. 75-101. • Stodolsky, S. S. (1990). Classroom observation. In J. Millman & L. Darling-Hammond (Eds.), The new handbook of teacher evaluation: Assessing elementary and secondary school teachers (pp. 175-190). Newbury Park, CA: Sage. • Unrau, Y. A., & Wehrmann, K.C. “Evaluation of a home-based family literacy program.” In Y.A. Unrau, J.L. Krysik, & R.M. Grinnell, Jr. (Eds.), Student study guide for the sixth edition of social work research and evaluation: Quantitative and qualitative approaches (pp. 183-190). Belmont, CA: Wadsworth. 2001. • Woodward, J. and J. Baxter (1997). “The effects of an innovative approach to mathematics on academically low-achieving students in inclusive settings.” Exceptional Children, 63(3), 373-388.