Using the Common European Framework of Reference to Report Language Test Scores

1. Using the Common European Framework of Referenceto Report Language Test Scores Spiros Papageorgiou University of Michigan spapag@umich.edu

2. Overview The Common European Framework of Reference (CEFR) The Manual for relating language examinations to the CEFR Standard setting An example of a CEFR standard setting study in Colombia

3. The CEFR • Reference document—not prescriptive • Basis for the elaboration of language syllabi, curricula, examinations, and textbooks • Language objectives: Description of what language learners have to learn to do in order to use a language for communication • Six main levels of proficiency: A1 (lowest), A2, B1, B2, C1, C2 (highest)

4. The Manual for RelatingExaminations to the CEFR It aims to “help the providers of examinations to develop, apply and report transparent, practical procedures in a cumulative process of continuing improvement in order to situate their examination(s) in relation to the Common European Framework” (p. 1).

5. Stages for Relating Test Contentand Test Scores to the CEFR Familiarization Specification Standardization training and benchmarking Standard setting Validation

6. Standard Setting • The decision making process of classifying examination results in a number of successive levels • Performance Level Descriptions (PLD): statements describing what learners can do with language(e.g., CEFR descriptors) • Performance Level Labels (PLL): labels of PLD(e.g., A1–C2) • Cut scores: the boundary between two successive levels • Participation of expert judges (panelists)

7. PLL PLD

8. An Example of a Standard Setting Study in Colombia Reporting scores for the Michigan English Test on the CEFR levels 13 participants from the 9 Binational centers in Colombia Familiarization with the CEFR Training with item difficulty (Pilot Form B) Angoff standard setting method First round of judgments Pilot Form A statistical information Second round of judgments

9. Standard Setting Validity Evidence • Procedural validity: examining whether the procedures followed were practical and implemented properly; that feedback given to the judges was effective; and that documentation was sufficiently compiled. • Internal validity: addressing issues of accuracy and consistencyof the standard setting results. • External validation: collecting evidence from independent sources that support the outcome of the standard setting meeting.

10. The Familiarization Task • A1 = 1, A2 = 2, B1 = 3, B2 = 4, C1 = 5, C2 = 6

11. Procedural Validity:Internalization of the CEFR Correlation of descriptor level judgments withthe CEFR during the Familiarization stage

12. Internal Validity: Method Consistency Standard error of judgments should be ≤ ½of the standard error of the test(Section I 1.71 and Section II 1.74 )

13. Internal Validity: Decision Consistency Calculating agreement coefficient rho(p0; max .98) and kappa (k; max 71)

14. Internal Validity: Intra-judge Consistency Correlation of mean of judgmentswith empirical item difficulty

15. Internal Validity: Inter-judge Consistency Indices of agreement and consistency

16. External Validity: Reasonablenessof the Cut Scores Classification of Pilot Form A test takers(N = 660) into CEFR levels

17. External Validity: Comparison ofLevel Classifications Exact and adjacent level agreement of classifications (N = 302) provided by a test center and the cut score

18. Final Stage Before ReportingTest Scores: Equating A statistical procedure used to allow for comparisons of scores obtained on different test forms Adjustment of differences in test form difficulty(but not content) Scaled scores, not percentages Examinee position on the language ability scale Scores are comparable across different administrations Linked to the CEFR cut scores

19. Reported Scores Both section scores should be taken into account when interpreting the test results for use in decision-making

20. For more information visit www.lsa.umich.edu/eli/testing