The KPG computer adaptive language testing in five languages

1. The KPG computer adaptive language testing in five languages Emerita Professor Bessie Dendrinos, President of the KPG Examination Board The National Foreign Language Exams for the State Certificate of Foreign Language Proficiency in Greece: The KPG multilingual examination suite

2. National Strategic Reference Framework (NSRF) – 2007-2013

3. e-KPG includes • The multilingual CALT platform • Platform for the e-training of oral examiners and script raters • The KPG e-school (http://rcel.enl.uoa.gr/kpgeschool/)

4. Multilingual CALT platform

5. Multilingual standards • Both the computer-based and the paper-based KPG exams contain common: • proficiency scaling (aligned to the 6 level scale of the Council of Europe) • illustrative descriptors (aligned to the CEFR) • accompanying criterial features • (constructivist) view of language-in-use

6. Multilingual framework • The exam suite is based on a generic (portable across languages) methodological framework suitable for describing language performance across proficiency levels.

7. Multilingual operationalization • And… across languages, the exams: • test common competences and strategies • are regulated by common specifications • use a common test-task typology

8. Basic tools of the CALT platform • Authoring tool • Piloting tool • Test player

9. Authoring tool roles Diagram showing the flow of roles, and work required to insert test tasks and give final approval

10. Testing understanding • The KPG CALT is developed to test both comprehension and production: • The first part tests: • reading comprehension • listening comprehension • language awareness

11. The task-based CALT • The CALT is not item but task based. • The smallest unit of the comprehension part is a task, not an item. • Three test tasks comprise a ‘testlet’. • The candidate’s performance in each testlet determines what testlet level s/he will be served next.

12. Composition of testlets • Each testlet is composed of three tasks: • One reading comprehension task • One language awareness task • One listening comprehension task • There is a common (across languages) formula as to the mix of the types of tasks that each testlet contains.

13. CALT task typology • The KPG CALT contains 10 task types giving us in all 12 types of testlets. • Each task is composed of 5 items as follows: • items testing comprehension of a source (written or audio) text • items testing awareness of correct and appropriate use of language

14. CALT testlet typology • 3 tasks comprise a testlet • 3 testlets comprise a testing unit • 3-5 testing units comprise a complete • a complete test for each of the lower levels (Α1-Β1) contains 3 testlets (45 items in total) • a complete test for each of the higher levels (B2-C2 level) contains 4 testlets (60 items).

15. Starting the test and moving through it • The candidate chooses the level of the testlet s/he wants to start with and then if • s/he is successful on 50% of the items, the next testlet is of the same level • his/her success is below 50%, the next testlet is of lower level • If his/her success is above 50%, the next testlet is of higher level

16. Digital e-test typology

17. Reading comprehension (French B2)Multiple choice (MC)-tick

18. Reading comprehension (German C1)Multiple choice (MC)-tick

19. Reading comprehension (English C2)Multiple choice (MC)-tick

20. Language awareness (Italian C1)Multiple match-insert missing item

21. Language awareness (Spanish A2) Matching-drag-’n-drop (Q&A)

22. Language awareness (English B1)Fill in - write in a word

23. Language awarness (German A1) Match-drag’n-drop

24. Language awareness (English B1)writing in a word

25. Listening comprehension (A1) Multiple match (drag-’n-drop letter/number)

26. Listening comprehension (Italian B1) Multiple match (drag-’n-drop letter/number)

27. Testing production • Production performance is also computer based • The test with which production is tested is a leveled test • Its purpose to verify ………

28. Testing leveled performance Testing speaking and oral mediation Testing writing and written mediation

29. Oral production • Similar test tasks across languages are aimed to assess context specific: • oral performance, stimulated by visual or multimodal texts in the target language • oral mediation performance, stimulated by texts in the home language

30. Speaking task (German B1) Diese Fotos stammen aus Ihrem elektronischen Familienalbum. Welche Personen sind auf den Fotos abgebildet? Wann und warum wurde jedes einzelne Foto gemacht?

31. Speaking task (Italian B2) A tuo parere, prima della ripresa di queste due fotografie, che cosa può essere accaduto? E cosa hanno in comune i due avvenimenti?

32. Speaking task (Spanish B2) Dinos cómo crees que se sienten estas personas y por qué.

33. Mediation in English (C2) Tell your friend Edward about two games he knows almost nothing about: backgammon and jenga. Click here to speak

34. Writing production • Similar test tasks across languages are aimed to assess context specific: • writing performance, stimulated by cues, visual or multimodal texts in the target language • written mediation performance, stimulated by texts in the home language

35. Writing task (French B2)

36. Writing task (Spanish A2) Escribe un e-mail (60-70 palabras) a tu amiga Valentina para hablarle deun bonito lugar de tu ciudad donde estuviste ayer. Cuéntale:1. Qué tipo de lugar era2. Dónde está exactamente3. Qué hiciste allí

37. Writing task (English A1)

38. Written mediation in Italian (C1)

39. Test task piloting

40. The Piloting Tool_1 • It’s a management system linked to the Authoring Tool. • Its purpose is to manage trial runs (or the piloting) of each test task, in real-time conditions, with a view to examining the task’s validity.

41. How does the PT operate? • Groups of tasks are packed in groups and piloted by sample population of candidates • Depending on how well the tasks have ‘behaved’, they may be considered: • good enough to be inserted in the Test Bank • partially problematic and in need of amendments • seriously problematic (warranting serious modification or ditching)

42. The piloting tool_2 • The PT has been designed to measure basic indices with an inbuilt statistics system • The functions of the PT are available to the members of the language-specific author team who have been assigned the role of ‘task evaluator’ and/or the role of ‘Administrator’.

43. Managing the test packs • They aremanaged as follows: • The user ‘characterizes’ a total number of tasks on the basis of the task. typology and creates test packages • Token are created for the sample population who will use them. • Indices of statistical interest are worked out for results and ultimately findings.

44. Where do we go from here? • In 2015 the KPG CALT was piloted on a sample of 15-16 year old foreign students of English, French and German from 22 lower secondary schools in different parts of Greece • In 2017 the e-KPG was institutionalized by law • Late 2018 additional funding is expected to allow the e-KPG subsytems, tools and testing centres to become operational for the whole school population • June 2018 the e-KPG CALT will be used to test Greek EFL students in the last class of lower secondary school to certify them as autonomous users of English