Assessment in the Less Commonly Taught Languages: Challenges and Approaches

1. Assessment in the Less Commonly Taught Languages: Challenges and Approaches William P. Rivers, Ph.D. Executive Director Joint National Committee for Languages National Council for Languages and International Studies September 23, 2013

2. Overview • Overview • Operational Definition of LCTLs • LCTL vs. “Critical Language” • Common characteristics of LCTL fields • Basic types of language assessment • Achievement • Proficiency • Why FL has an edge in assessment: Proficiency • Challenges in proficiency testing in the LCTLs • Alternative approaches • About JNCL-NCLIS • Q & A

3. What is a “Less Commonly Taught Language?” • MLA enrollment survey (Furman et al., 2009): • 1.6m FL enrollments in US IHEs in 244 FLs • 8.4 enrollments per 100 students (essentially unchanged since 1974) • 1.2m of these (80%): Spanish, French, German • In effect, all other FLs are LCTLs • 1st tier: 10,000 – 80,000 enrollments: Chinese, Italian, Japanese, Russian, Portuguese, Hebrew, Japanese • 2nd tier: 1,000 – 10,000 enrollments: 9 • 3rd tier: <1,000: 235 FLs

4. Common characteristics of LCTL fields: Capacity challenges • Foundational: • linguistic descriptions may be lacking; SLA research in the TL may be lacking • Few or no Ph.D. programs to train future faculty & researchers • Few or no programs to train K-12 teachers • Infrastructure: • fewer and less-widely distributed disciplinary journals • Smaller or joint conferences • Few professional development programs • Programs • Fewer programs (of course) and • Programs with no advanced level programming • Programs under constant threat of elimination due to low enrollments

5. LCTL vs. “Critical Language” • Each USG agency has its own criteria for determining which languages are critical • Not all LCTLs are critical • Not all CTLs aren’t critical • For external funding programs, determination of “critical language” rests on • Agency needs • Programmatic priorities • funding

6. Basic types of FL Assessment • Achievement: Discrete point testing of specific items • Typically tied to a specific course or curriculum • Proficiency: graduated, structured, progressive testing of language competencies • Curriculum and program independent • Fundamentally communicative • Proficiency testing began at Foreign Service Institute in the 1950s – developed by John Carroll and his students • Based on three years’ observation of language use by diplomats • Divided into 4 skills (L/R/S/W) and five levels • This became the basis for the ILR Proficiency Scale • In turn the basis for the ACTFL proficiency Scale (and most others worldwide use the same approach)

7. Proficiency Scale Review • ACTFL: • Novice • Intermediate • Advanced • Superior • ILR: 0 -5 • Professional language skill generally requires the “Superior” or ILR 3 level • 4 years of College FL results in a median proficiency of Intermediate High /1 +

8. Why we have an edge • FL proficiency testing has revolutionized FL pedagogy in the past 30 years: we now focus on communicative ability as the fundamental goal of FL education • Among the Humanities, FL has a unique capability to show what students can do with their knowledge, because proficiency as a construct is precisely that – what can an individual dowith their FL • 50+ years of practice and R&D in FL testing

9. LCTL Assessment Challenges • Range of languages • 240 + languages • Geographic dispersion of examinees • Low number of examinees • Economic viability of test development • Technical challenges • Validation • Test design

10. An Incomplete Set of Available Tests: Oral Proficiency Interviews Afrikaans, Akan-Twi, Albanian, Algerian, Amharic, Arabic, Armenian, Azerbaijani, Baluchi, Bengali, Bosnian, Bulgarian, Burmese, Cambodian, Cantonese, Cebuano, Chavacano, Czech, Dari, Dutch, Egyptian, English, French, Ga, Georgian, German, Greek (Modern), Gujarati, Haitian Creole, Hausa, Hebrew, Hiligaynon, Hindi, Hmong-Mong, Hungarian, Igbo, Ilocano, Indonesian, Italian, Iraqi, Japanese, Javanese, Jordanian, Kashmiri, Kazakh, Kikongo-Kongo, Korean, Krio, Kurdish Kurmanji, Kurdish Sorani, Lao, Lebanese, Levantine, Libyan, Lingala, Malay, Malayalam, Mandarin, Mandingo-Bambara, Moro, Nepali, Pashto, Persian-Farsi, Polish, Portuguese, Punjabi, Romanian, Russian, Serbian/Croatian, Sindhi, Sinhalese, Slovak, Somali, Spanish, Swahili, Tagalog, Tajik, Tamil, Tausug, Telugu, Thai, Tigrinya, Turkish, Turkmen, Uighur, Urdu, Uzbek, Vietnamese, Wolof, Wu, Yoruba   Source: Language Testing International

11. Challenge: volumeApproach: Self-Assessments • 4 skill (L/R/S/W) • Based on ILR scale • Global statements of proficiency • Detailed assessments by task and proficiency level (Can-do statements) • Regular monitoring of concurrent validity worth formal proficiency tests

12. Self-Assessments: Concurrent Validity • 4 skill (L/R/S/W) • Based on ILR scale • Global statements of proficiency • Detailed assessments by task and proficiency level (Can-do statements) • Regular monitoring of concurrent validity with formal proficiency tests • Peer reviewed (Stansfield, Gao, Rivers, 2010)

13. Self-Assessments: Concurrent Validity • Moderate correlations; acceptable for membership screening

14. Challenge: Range of LanguagesApproach: Expand access to quality tests • ASTM F2889 – 11: National industry standard for proficiency test development • Users and providers develop consensus Standard to ASTM format • ASTM: American Society for Testing and Materials • 30000+ industry standards • Develops but does not enforce standard • Equal representation of users & providers • Participation is open & voluntary • ASTM F43 on Language Services and Products • 120 members from industry, academe, government • 4 standards developed • 6 underway

15. Value of Standard Practice • Leverage • world-leading foreign language proficiency testing (the DLPT system) and practices (the FILR Proficiency Guidelines) to ensure quality procurement of test development and administration • the $15b language services industry’s potential to provide language proficiency testing in more than 150 languages • Enable rapid development of foreign language proficiency tests to meet surge requirements • Enhance compliance with • PL 104-113 §12(d), which requires US Government agencies to maximize the use of industry standards • DoD Standards Program, DoD 4120.24-M • Improve quality assurance through the availability of such standards to procurement and contracting for language proficiency testing

16. Scope of ASTM F2889-11, Standard Practice for Assessing Language Proficiency • Establishes specific requirements for quality and transparency in • Test specification • Test development • Test administration • Incorporates current best practices • FILR Guidelines • DLPT-V testing • Other USG and industry leading test practices • All levels/all modalities are covered

17. Scope of Standard Practice (2) • Incorporates current best practices • FILR Guidelines • DLPT-V testing • Other USG and industry leading test practices • Mandates specific practices, e.g., • Test frameworks • Ethics • Quality control • Quality assurance

18. The Joint National Committee for Languages and the National Committee for Languages and International Studies • Comprise more than 80 organizations in the language enterprise • Develop and advocate policies for language and global expertise • Promote language as a profession in the United States • Support the development of academic, governmental, and industrial standards for language work • Have joined with the language industry and the Globalization and Localization Association to form the American Language Enterprise Advocacy (ALEA)

19. Contact Dr. Bill RiversExecutive Director, JNCL-NCLIS Chair, ASTM Main Committee F43, Language Services and Products wrivers@languagepolicy.org