Improving Teachers’ Competency in Classroom-Based Assessment

1. Improving Teachers’ Competency in Classroom-Based Assessment National Evaluation Institute Louisville, KY October 10, 2009 Christopher R. Gareis, Ed.D. Associate Dean for Teacher Education & Professional Services Associate Professor of Educational Leadership crgare@wm.edu 757.221.2319 Leslie W. Grant, Ph.D. Visiting Assistant Professor Curriculum & Instruction lwgran@wm.edu 757.221.2419

2. The Need • Assessment is integral to teaching & learning • Assessment has become high-stakes • An unintended effect of high-stakes, standardized assessments is the narrowing of the curriculum(Clarke, Shore, Rhoades, Abrams, Miao, & Li, 2003) a C I A

3. Many teachers are not adequately prepared in the domain of assessment a C I A State licensure standards • 1983: 10 states explicitly identified assessment as a core competency for teacher licensure (Stiggens & Conklin, 1992) • 2000: 14 states (Atkin et al., 2001) Teacher preparation programs • “Assessment” has typically been taught through courses in methods, tests & measurements, or educational research • 2000: NCATE Standards become performance-based, including contributions to student learning • 2006: Assessment is the weakest competency of first-year teachers(Good, McCaslin, Tsang, et al., 2006) Supervision and Professional Development • 2003: Assessment was the area least adequately documented domain of teaching responsibility among in-service teachers (Tucker, Stronge, & Gareis, 2003) • 2000: Teachers receive little professional development on assessment (Wenglinsky, 2000)

4. lower case “assessment” matters 84th %-ile 60th %-ile + 10 percentile point gain in student achievement Assessment Reform Group (1999)

5. Assessment in the Classroom Make decisions Take actions Communicate Provide feedback Caution! Poorly made assessments lead to erroneous inferences

6. Assessment Literacy Understanding and practically applying on a consistent basisthe concepts of validity and reliabilityas they are relevant to assessment and evaluationin the classroom in order to help students learn

7. A Model for Improving Assessment Literacy Context & Inputs Processes of Professional Development Outcomes for Teachers Impact on Student Learning Experience & Expertise of Professional Developers Explore Alignment C=Ia=A • Unpack Curriculum • Content • Cognitive levels • Experience & Expertise of Participants • - Pre-service teachers • - In-service teachers • Instructional leaders • Administrators Create a Table of Specifications or “blueprint” Psychometric Principles -Translated into practical terms & skills for teachers Critique an assessment for validity & reliability • Explore uses of ToS • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results State Standardized Assessments - de facto Curriculum? State &/or School District Curriculum Create good assessment items, prompts, assignments, & rubrics District Goals, Initiatives, Imperatives, & Context

8. Sixth Grade Science Standard 6.9 Curriculum Framework Unpacking the Curriculum Co Co An Ev An An Co Ev

9. Sixth Grade Science Unit on Policy Decisions Related to the Environment (Virginia Standards) Content of the standard identified Cognitive level of the standard identified

10. A Model for Improving Assessment Literacy Context & Inputs Processes of Professional Development Outcomes for Teachers Impact on Student Learning Experience & Expertise of Professional Developers Explore Alignment C=Ia=A Understand role of C=Ia=A Alignment • Unpack Curriculum • Content • Cognitive levels Understand relationship between assessment & evaluation • Experience & Expertise of Participants • - Pre-service teachers • - In-service teachers • Instructional leaders • Administrators Create a Table of Specifications or “blueprint” Understand & apply concepts of validity & reliability Psychometric Principles -Translated into practical terms & skills for teachers Critique an assessment for validity & reliability • Use a ToS to: • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results • Explore uses of ToS • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results State Standardized Assessments -de facto Curriculum? State &/or School District Curriculum Apply principles to the spectrum of assessment types & practices Create good assessment items, prompts, assignments, & rubrics District Goals, Initiatives, Imperatives, & Context

11. Using a Table of Specifications to Critique and Improve Assessments Under-sampling/ ignoring cognitive level Is this a valid assessment of student learning? Oversampling content and cognitive level Under-sampling the content

12. A Model for Improving Assessment Literacy Context & Inputs Processes of Professional Development Outcomes for Teachers Impact on Student Learning Experience & Expertise of Professional Developers Explore Alignment C=Ia=A Understand role of C=Ia=A Alignment Create assessments that yield valid and reliable data • Unpack Curriculum • Content • Cognitive levels Understand relationship between assessment and evaluation • Experience & Expertise of Participants • - Pre-service teachers • - In-service teachers • Instructional leaders • Administrators Use assessment data to make decisions: - About student learning - For student learning - About assessments - About instruction Create a Table of Specifications or “blueprint” Understand and apply concepts of validity and reliability Psychometric Principles -Translated into practical terms and skills for teachers Critique an assessment for validity and reliability • Use a ToS to: • Create an assessment • Critique and improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results Contribute to the design and development of common assessments • Explore uses of ToS • Create an assessment • Critique and improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results State Standardized Assessments -De Facto Curriculum? State and/or School District Curriculum Provide “opportunity to learn” through aligned instruction Apply principles to the spectrum of assessments Create good assessment items and prompts, assignments, and rubrics District Goals, Initiatives, or Imperatives

13. Using Assessment Results to Make Decisions • Area of Concern for Whole Class • re-teach • re-assess students • Jason responded incorrectly to most of these items • re-teach • re-assess • Poorly worded question • throw out question • adjust scoring • revise question

14. A Model for Improving Assessment Literacy Context & Inputs Processes of Professional Development Outcomes for Teachers Impact on Student Learning Experience & Expertise of Professional Developers Explore Alignment C=Ia=A Understand role of C=Ia=A Alignment Create assessments that yield valid & reliable data • Unpack Curriculum • Content • Cognitive levels Understand relationship between assessment & evaluation • Experience & Expertise of Participants • - Pre-service teachers • - In-service teachers • Instructional leaders • Administrators Use assessment data to make decisions: - About student learning - For student learning - About assessments - About instruction Create a Table of Specifications or “blueprint” Understand & apply concepts of validity & reliability Psychometric Principles -Translated into practical terms and skills for teachers Critique an assessment for validity & reliability • Use a ToS to: • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results Contribute to the design & development of common assessments • Explore uses of ToS • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results State Standardized Assessments -De Facto Curriculum? State &/or School District Curriculum Provide “opportunity to learn” through aligned instruction Apply principles to the spectrum of assessment types & practices Create good assessment items, prompts, assignments, & rubrics District Goals, Initiatives, Imperatives, and Context

15. Creating Common AssessmentsAre we assessing students in an appropriate way?Are we making instructional decisions based on results that are valid and reliable?

16. A Model for Improving Assessment Literacy Context & Inputs Processes of Professional Development Outcomes for Teachers Impact on Student Learning Experience & Expertise of Professional Developers Explore Alignment C=Ia=A Understand role of C=Ia=A Alignment Create assessments that yield valid & reliable data Exposure to the full curriculum (e.g., depth & skill development) • Unpack Curriculum • Content • Cognitive levels Understand relationship between assessment & evaluation • Experience & Expertise of Participants • - Pre-service teachers • - In-service teachers • Instructional leaders • Administrators Use assessment data to make decisions: - About student learning - For student learning - About assessments - About instruction Create a Table of Specifications or “blueprint” Understand & apply concepts of validity & reliability Increased student achievement Psychometric Principles -Translated into practical terms & skills for teachers Critique an assessment for validity & reliability • Use a ToS to: • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results Contribute to the design & development of common assessments • Explore uses of ToS • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results State Standardized Assessments - de facto Curriculum? Deeper, more meaningful learning State &/or School District Curriculum Provide “opportunity to learn” through aligned instruction Apply principles to the spectrum of assessment types & practices Create good assessment items, prompts, assignments, & rubrics District Goals, Initiatives, or Imperatives

17. Aha Moment! Currently used “fun” activities could facilitate and assess higher cognitive levels when properly structured Aha Moment! Ongoing “essential skills” target higher-order thinking

18. A Model for Improving Assessment Literacy Context & Inputs Processes of Professional Development Outcomes for Teachers Impact on Student Learning Experience & Expertise of Professional Developers Explore Alignment C=Ia=A Understand role of C=Ia=A Alignment Create assessments that yield valid & reliable data Exposure to the full curriculum (e.g., depth & skill development) • Unpack Curriculum • Content • Cognitive levels Understand relationship between assessment & evaluation • Experience & Expertise of Participants • - Pre-service teachers • - In-service teachers • Instructional leaders • Administrators Use assessment data to make decisions: - About student learning - For student learning - About assessments - About instruction Create a Table of Specifications or “blueprint” Understand & apply concepts of validity & reliability Increased student achievement Psychometric Principles -Translated into practical terms & skills for teachers Critique an assessment for validity & reliability • Use a ToS to: • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results Contribute to the design & development of common assessments • Explore uses of ToS • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results State Standardized Assessments - de facto Curriculum? Deeper, more meaningful learning State &/or School District Curriculum Provide “opportunity to learn” through aligned instruction Apply principles to the spectrum of assessment types & practices Create good assessment items, prompts, assignments, & rubrics District Goals, Initiatives, or Imperatives

19. We’re in the big leagues.

21. Essential Evaluation & Research Questions • In what ways and to what degree does professional development result in teachers’ improved ability to construct and use valid & reliable assessments? • To what degree does improved assessment literacy correlate with improved student achievement?

22. A Model for Improving Assessment Literacy Context & Inputs Processes of Professional Development Outcomes for Teachers Impact on Student Learning Experience & Expertise of Professional Developers Explore Alignment C=Ia=A Understand role of C=Ia=A Alignment Create assessments that yield valid & reliable data Exposure to the full curriculum (e.g., depth & skill development) • Unpack Curriculum • Content • Cognitive levels Understand relationship between assessment & evaluation • Experience & Expertise of Participants • - Pre-service teachers • - In-service teachers • Instructional leaders • Administrators Use assessment data to make decisions: - About student learning - For student learning - About assessments - About instruction Create a Table of Specifications or “blueprint” Understand & apply concepts of validity & reliability Increased student achievement Psychometric Principles -Translated into practical terms & skills for teachers Critique an assessment for validity & reliability • Use a ToS to: • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results Contribute to the design & development of common assessments • Explore uses of ToS • Create an assessment • Critique & improve an assessment • Create a unit plan assessment • - Plan instruction • - Analyze assessment results State Standardized Assessments - de facto Curriculum? Deeper, more meaningful learning State &/or School District Curriculum Provide “opportunity to learn” through aligned instruction Apply principles to the spectrum of assessment types & practices Create good assessment items, prompts, assignments, & rubrics District Goals, Initiatives, or Imperatives