1 / 28

Research on Assessment

Research on Assessment. 21 st Century Paradigm Shifts NCLB calls teachers and teacher candidates to be accountable for student learning citing research on connection of student learning/achievement to teacher preparation Widening achievement gaps

vin
Télécharger la présentation

Research on Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Research on Assessment • 21st Century Paradigm Shifts • NCLB calls teachers and teacher candidates to be accountable for student learning citing research on connection of student learning/achievement to teacher preparation • Widening achievement gaps • Older Models of Assessment (R.J. Dietel, J.L. Herman, and R.A. Knuth, 1991) • Based on linear model of learning • Instruction was “building blocks” approach • Basic skills learned by rote could be assembled into complex thinking and performance strategies. • Meaningful Learning • Cognitive Psychology Research • Learning proceeds in many directions at once • Important to measure how and whether students organize, structure, and use information to solve problems. Monarch Center 2011

  2. Research on Assessment • Assessment Trends (R.J. Dietel, J.L. Herman, and R.A. Knuth, 1991) • Cognitive View of Assessment • View of Learner: Active, constructive knowledge environment • Scope of Assessment Integrated and cross-disciplinary (alternative views exist – especially those experts in academic language) • Emphasis of Instruction Attention to meta-cognition, formative assessments, effective materials, motivation and self-determination • Characteristics of Assessments Authentic assessments on multiple-choice contextualized problems that are short answer relevant and meaningful, emphasize higher-level thinking, do not have a single correct answer, have public standards known in advance. • Who is Assessed Both individual and group process skills and performances • Frequency of Assessment Samples over time (portfolios, work samples) which provide basis for assessment by multiple stakeholders. Monarch Center 2011

  3. Research on Assessment • Assessment Trends (R.J. Dietel, J.L. Herman, and R.A. Knuth, 1991) • Cognitive View of Assessment • Use of Technology High-tech applications such as administration and scoring sheets computer-adaptive testing, expert systems, and simulated environments • What is Assessed Multidimensional assessment that recognizes the variety of human abilities and talents, malleability of student ability, and that IQ is not fixed • Cognitive View of Assessment • Performance Assessments • Richard Stiggins notes three critical components of performance assessments: • (1) Specification of a performance to be evaluated; • (2) Development of exercises or tasks used to elicit that performance; • (3) Design of a scoring and recording scheme for results. Monarch Center 2011

  4. Research on Assessment • A Combination of Formative and Summative Evaluations best. (C. Mathers, M. Oliva, S. Laine, 2008) • Results • Issues of Performance-Based Tests/Tasks • "Questions of fairness arise not only in the selection of performance tasks but in the scoring of responses. As Stiggins has stated, it is critical that the scoring procedures are designed to assure that `performance ratings reflect the examinee's true capabilities and are not a function of the perceptions and biases of the persons evaluating the performance.' The same could be said regarding the perceptions and biases of the persons creating the test. The training and calibrating of raters is critical in this regard." Monarch Center 2011

  5. Research on Assessment • The primary concerns with performance assessment are: • 1. Time and Content: Performance-based assessments are not able to test as much material as multiple-choice tests in the same amount of time. • Performance-based assessments usually require additional time to administer—this takes away from instructional time. • An inherent trade-off between time and content; the more content the performance-based assessment attempts to cover, the more time it will take to design, administer, and score. Monarch Center 2011

  6. Research on Assessment • 2. Reliability: The main threat to reliability comes from the necessity of having ‘experts’ score the performance-based assessments. Even with a set rubric, there may be variation in scoring among different raters. • In particular, the scoring of performance-based assessment is “susceptible to two general classes of measurement error: random and systematic” (Raymond & Viswervaran, 1993). • Random error is inter-rater reliability. • Systematic error, also called leniency error, “is present when the mean of a rater summed over candidates differs from the mean of all raters” (Raymond & Viswervaran, 1993). Monarch Center 2011

  7. Research on Assessment • 3. Validity: Internal validity is the extent to which each question (or task) on the test measures the objective, skill, etc. that it intends to measure. • External validity, also referred to as generalizability, is the extent to which a student’s performance on a question (or an entire test) represents their overall ability, or the extent to which their performance on one question can be generalized to the domain of knowledge the task represents, or even the extent to which various performance assessments, with the same domain, can be compared. • Performance-based assessments are vulnerable to both internal and external validity threats because it is difficult to design performance-based assessments, and also because time and money constrains the sample one is able to test from the domain (Madaus & O’Dwyer, 1999). Monarch Center 2011

  8. Research on Assessment • Large-scale, performance-based assessments are a favorable alternative to current dominant testing strategies in any subject because they have the potential to • …yield a more complete picture of students’ abilities and weaknesses…support higher quality teaching… increase intellectual challenge in the classroom…[and] can overcome some of the validity challenges of assessing English Language Learners and students with disabilities. (Adamson & Darling-Hammond, 2010) Monarch Center 2011

  9. Performance Assessment of Teacher Candidates • Combination of standardized content knowledge exams and comprehensive performance assessments • Generic and Specific Models • Use of Rubrics • Multidimensional sets of scoring guidelines for specific areas/tasks/projects • Use of Work Samples • An applied performance approach that can be tailored to: • Learning goals • Teaching style • Group & individual student needs • The context of the classroom, school, & community • Increased connection between courses and field experiences Monarch Center 2011

  10. California Teacher Performance Assessment • A performance assessment for teacher candidates with new subject matter standards, new program standards, and new assessment standards; new candidates must complete beginning in 2008 • Alternate assessments permitted must meet California Quality Standards for reliability/validity (i.e., AERA/APA test standards). • Aligned with the California Teaching Performance Expectations (standards) and California Content Standards Monarch Center 2011

  11. California Teacher Performance Assessment • The PACT is a performance assessment, in which teacher candidates plan a series of lessons based on a single standard or set of standards, implement the lessons and collect evidence of student learning, videotape one or more segments and reflect on performance, and reflect on the teaching event as a whole. • All the related documents are assembled into a portfolio. • The rationale for an embedded performance assessment is to assess competence in actual practice. • The logic is that if a teacher preparation program can closely document and then assess the candidate’s performance of an actual teaching event then the program has the greatest likelihood of truly measuring the candidate’s ability to perform under actual teaching conditions Monarch Center 2011

  12. California Teacher Performance Assessment www.pacttpa.org Monarch Center 2011

  13. California Teacher Performance Assessment • Content validity • Development teams, Program directors, Program faculty, & Leadership team • TPE alignment study • Concurrent validity • Evaluation of score validity • Decision Consistency · Holistic vs. Analytic ratings • Bias and fairness review • Construct validity • Factor Analysis(2002-03 Pilot Year): • Reflection & Assessment • Instruction • Planning • Predictive Validity (Carnegie/CT Study) Monarch Center 2011

  14. California Teacher Performance Assessment • Capstone Teaching Event • Planning • Lesson Plans • Handouts, overheads, student work • Lesson Commentary • Instruction • Video clip(s) • Teaching Commentary • Assessment • Analysis of Whole Class Assessment • Analysis of learning of 2 students • Reflection • Daily Reflections • Reflective Commentary • Evidence of Academic Language Monarch Center 2011

  15. California Teacher Performance Assessment • Capstone Teaching Events: Criteria • PLANNING • Establishing a Balanced Instructional Focus • Making Content Accessible • Designing Assessments • INSTRUCTION • Engaging Students in Learning • Monitoring Student Learning During Instruction • ASSESSMENT • Analyzing Student Work From an Assessment • Using Assessment to Inform Teaching • REFLECTION • Monitoring Student Progress • Reflecting on Teaching • ACADEMIC LANGUAGE • Understanding Language Demands • Supporting Academic Language Development Monarch Center 2011

  16. California Teacher Performance Assessment • Embedded Signature Assessments/Assignments • Developed by Each Teacher Preparation Program • Some partnerships with local districts • Within coursework tied to California Teaching Performance Expectations • Case Studies • Analysis of Student Learning • Analysis of Teaching/ Curriculum Implementation Monarch Center 2011

  17. California Teacher Performance Assessment • Teaching Performance Events • TPE 1 · Specific Pedagogical Skills for Subject Matter Instruction • TPE 2 · Monitoring Student Learning During Instruction • TPE 3 · Interpretation and Use of Assessments • TPE 4 · Making Content Accessible • TPE 5 · Student Engagement • TPE 6 · Developmentally Appropriate Teaching Practices • TPE 7 · Teaching English Learners • TPE 8 · Learning about Students • TPE 9 · Instructional Planning • TPE 10 · Instructional Time • TPE 11 · Social Environment • TPE 12 · Professional, Legal, and Ethical Obligations • TPE 13 · Professional Growth Monarch Center 2011

  18. California Teacher Performance Assessment • Specifics of TPE’s • Focus of learning segment & aligned to California content standards • Teaching/learning tasks on video clip(s) • Additional prompts in some content areas (e.g., misconceptions in science, dispositions in mathematics, description of text in English/language arts) • Multiple and single subject specific rubrics • CATs – Content Area Tasks • Benchmarks within subject areas Monarch Center 2011

  19. California Teacher Performance Assessment • Scoring of Teaching Events • Trained and calibrated subject specific assessors • Campus based with central audits & regional scoring • Rubric based scoring in real time (web based platforms) • Organized around dimensions of teaching and guiding questions • Sequentially Scored Monarch Center 2011

  20. California Teacher Performance Assessment • In addition to on-site supervision, teacher candidates must submit a portfolio that comprises: • The candidate’s work planning curriculum, assessment, and instruction. • Further, the portfolio includes evidence of work around implementing instruction, assessing student learning, and then analysis of candidates’ teaching and learning outcome objectives. • All of the work on the portfolio is based on a teaching event of a candidate’s own design. • The portfolio consists of planning documents in the form of pre-designed forms calling for classroom demographic and contextual information, lesson plans, video segments of the teaching event, reflective commentary around the various components of the teaching event, student work samples. Monarch Center 2011

  21. Future Trends in Performance Assessments • Large Scale Performance Assessments • To make performance-based assessments affordable, some promising options are to: • …take advantage of the economies of scale that will accompany states banding together in consortia, tapping the efficiencies of technology in administering tests and supporting scoring, and using teachers strategically in the scoring of performance items. (Adamson & Darling Hammond, 2010) Monarch Center 2011

  22. Future Trends in Performance Assessments • Earlier this year in a speech to state leaders at Achieve’s American Diploma Project Leadership Team Meeting, U.S. Secretary of Education Arne Duncan remarked, “Today is the day that marks the beginning of the development of a new and much-improved generation of assessments for America’s schoolchildren. Today marks the start of Assessments 2.0”  (Duncan, 2010, para 1). Monarch Center 2011

  23. Future Trends in Performance Assessments • New assessments by two large state consortia, the Partnership for Assessment of Readiness for College and Careers (PARCC) and SMARTER Balanced Assessment Consortium • Race to the Top competition (Duncan, 2010). These two consortia will work over the next few years “designing and implementing comprehensive assessment systems in math and English language arts” so that by 2014 the assessments will be ready for use in any state (Duncan, 2010). • Tied to Common Core Standards Monarch Center 2011

  24. Research on Assessment of Special Education Teacher Preparation Programs (L. Goe, 2006) • Teacher Preparation---Teacher Practices----Student Outcomes Relationship in Special Education • Issues • Inclusion Models of Delivery • Co-Teaching • Push-In; Pull Out • Resource Rooms • Who is ultimately responsible for the student gains? Monarch Center 2011

  25. Research on Assessment of Special Education Teacher Preparation Programs (L. Goe, 2006) • Experimenting with Value-added Models • Use of student test data compared to what is “expected” to occur • Imprecise in estimates of student effects • Data needs two-three years minimum • Use of Student Standard Growth Models • Holds constant student variables such as poverty, race, language fluency, and prior test scores. • Not every state has data or systems for perform calculations • Schools Better Equipped to Collect and Analyze Data on School-Level • Less useful for specific teacher effectiveness Monarch Center 2011

  26. Research on Assessment of Special Education Teacher Preparation Programs (L. Goe, 2006) • No Study Adequately Connects Teacher Preparation to Classroom Practice to Student Learning • Pieces of the Puzzle – Five Factors (Carlson, Lee and School, 2004) • Experience, Credentials, Self-Efficacy, Professional Activities, and Selected Classroom Practices (EBP) • Sample of 1475 special education teachers • All contributed to an aggregate measure of teacher quality Monarch Center 2011

  27. Research on Assessment of Special Education Teacher Preparation Programs (L. Goe, 2006) • Grisham-Brown, Collins, and Baird (2000) • Researched pre-service special education teachers explicit connections from what they learned in a course to actual teaching practices. • Miller (1991) Identified as Being Closest to Looking at All Three Components • Integration of Special Education and ELA teacher preparation program • Unit of Study • Actual Pre-Post test measures of student learning Monarch Center 2011

  28. Research on Assessment of Special Education Teacher Preparation Programs (L. Goe, 2006) • Future Directions Needed in Research • Encourage and support research that links teacher preparation---classroom practices--- student learning/outcomes • Extend research to include induction and early professional development • Consider curriculum-based measures of achievement for students with special needs • Include special education instruction in all teacher preparation programs; and encourage collaboration between special and general education teachers • Design research that measures collective contributions to student outcomes • Develop a comprehensive longitudinal teacher database Monarch Center 2011

More Related