Adapting Credentialing Exams for Use in Multiple Languages and Cultures

1. Adapting Credentialing Exams for Use in Multiple Languages and Cultures Ronald K. Hambleton University of Massachusetts

2. Three Reasons for Translating and Adapting Exams 1. Knowledge and skills of interest are often the same across language groups--exam adaptation ensures consistency of content and structure. 2. Often more efficient to translate and adapt an exam than to build a new exam. 3. Easier to insure exam equivalence and fairness (than construct new exam in a second language).

3. Goals of the Presentation 1. To describe several myths about test adaptation. 2. To review steps in the test adaptation process. 3. To present the ITC guidelines for test translation/adaptation.

4. Test Translation vs. Test Adaptation? “Test Adaptation” is more descriptive of the process that usually takes place—directions, formats, contexts, etc. “Test Translation” is sometimes only a small part of the process.

5. Examples of Test Adaptations • Intelligence, Aptitude, and Personality Tests • Quality of Life Surveys • TIMSS, OECD/PISA, and More • SAT, GED, NAEP, State Assessments • Credentialing Exams

6. Example 1 Out of sight, out of mind (Back translated from French) invisible, insane

7. Example 2 (IEA Study in Reading) Are these words similar in meaning? Pessimistic -- Sanguine

8. Pessimistic -- Sanguine Adapted to Pessimistic -- Optimistic

9. Example 3 (1995 TIMMS Pilot) Alex reads his book for 1 hour and then used a book mark to keep his place. How much longer will it take him to finish the book? A. ½ hour B. 2 hours C. 5 hours D. 10 hours

10. Example 4 Parker Pen in Mexico: “It won’t leak in your pocket and embarrass.” Became: “It won’t leak in your pocket and make you pregnant.”

11. Four Common Myths About Adapting Credentialing Exams 1. Know two languages and you can be a translator.

12. Selection and Training of Translators • Knowledgeable in the languages • Knowledgeable in the cultures • Knowledgeable in the subject matter • Knowledgeable in the principles of test development, item writing, and scoring rubrics, etc.

13. Four Common Myths About Adapting Credentialing Exams 2. A good translation guarantees validity.

14. Four Common Myths About Adapting Credentialing Exams 3. Judgmental reviews are sufficient to identify problems in a exam translation/adaptation.

15. Four Common Myths About Adapting Credentialing Exams 4. The common strategy of a back-translations design and the use of a bilingual design to compile empirical data is sufficient to justify exam use.

16. 8 Steps for Adapting Exams 1. Checking exam content and format equivalence. -content may not be generalizable -item formats are not universally used

17. 8 Steps for Adapting Exams 2. Deciding on the desirability of a translation/adaptation. -unsuitability of content and/or the exam format? -technical expertise to produce and validate a new exam?

18. 8 Steps for Adapting Exams 3. Choosing translators. -Four qualifications given earlier are important. -Some technical expertise may be especially helpful.

19. 8 Steps for Adapting Exams 4. Translating and adapting the credentialing exam. -Should often be more than a single translator. -Several designs and variations available.

20. Judgmental Designs for Assessing Suitability of a Translation/Adaptation 1. Backward Adaptations 2. Forward Adaptations

21. Backward Translation Design Advantages 1. Some problems can be identified. 2. Researchers do not need to be bilingual. Disadvantages 1. No review of target language version!

22. Forward Translation Design Advantages 1. Judgments made directly about language versions of an exam. Disadvantages 1. Bilingual judges make assessments of an exam for monolingual examinees.

23. 8 Steps for Adapting Exams 5. Reviewing the adapted version of the exam. -Multiple reviewers are often very valuable. -”Smoothing” of language is possible at this step.

24. 8 Steps for Adapting Exams 6. Conducting a small tryout of the adapted exam -item analysis, reliability analysis, candidate review, etc.

25. 8 Steps for Adapting Exams 7. Carrying out a more ambitious study of the adapted exam (that is, checking for exam validity). -construct equivalence, method equivalence, item equivalence

26. Statistical Designs 1. Bilingual Designs 2. Monolingual Designs A. Source Language Examinees Take Original and Backward Adapted Versions B. Monolingual Examinees Take Versions in Their Own Languages

27. Checking for Construct Equivalence:

29. Checking for Item Equivalence: Delta Plots

30. Checking for Item Equivalence: IRT Methods (change in difficulty)

31. IRT Methods (change in discrimination)

33. Checking for Method Equivalence 1. Any biases due to directions, exam format, scoring? 2. Preparation of exam administrators? 3. Equivalence of exam taking skills?

34. 8 Steps for Adapting Exams 8. Documenting the process. -Steps in exam adaptation, details, evidence for validity.

35. International Test Commission Guidelines for Adapting Educational and Psychological InstrumentsPrepared ByInternational Test Commissionand Seven International Organizations(EAPA, ETP, IACCP, IAAP, IEA, ILTA, IUPsyS)

36. Instrument Adaptation Guideline A practice which is judged as important for conducting and evaluating the adaptation or parallel development of psychological and education instruments for use in different populations.

37. Organization of the 22 ITC Guidelines • Context (2) • Instrument Development and Adaptation (10) • Administration (6) • Documentation/Score Interpretations (4)

38. Organization of Each of the 22 ITC Guidelines • Description of the Guideline • Rationale/Explanation • Steps to Meet the Guideline • Common Errors • References for Additional Study

39. D.3 Test developers/publishers should provide evidence that the choice of testing techniques, item formats, test conventions, and procedures are familiar to all intended populations.

40. D.8 Test developers/publishers should provide information on the evaluation of validity in all target populations for whom the adapted versions are intended.

41. D.9 Test developers/publishers should provide statistical evidence of the equivalence of questions for all intended populations.

42. Three Main Conclusions 1. Has progress been made in test adaptation methodology? -Definitely! -Moved from single unqualified translator and bilingual design to considerably more sophisticated methodologies.

43. Conclusions, cont. -Emergence of ITC guidelines; methodological advances in judgmental designs and statistical methods (e.g., DIF, SEM);more resources and time being allocated.

44. Conclusions, Cont. 2. What lies ahead for the field of test adaptations? -I’m very optimistic about the future. -Needed is commitment of resources and time, and implementation of existing methods.

45. Conclusions, Cont. 3. What are the most important areas for improvement? -choose multiple qualified translators and give them sufficient time -aggressively implement empirical methods -build on past experiences and knowledge base

46. For more information about exam translation and adaptation, contact Ron Hambleton at Rkh@educ.umass.edu