Chinese Taipei

1. Chinese Taipei APEC Language Standards for English and Other Foreign Languages A Critical Study of Evaluation Standards for English Programmes at Soochow University Dr. Byron Gong English Department Soochow University, Taipei

2. Current College ELT Testing Practice • Nationally, current evaluation criteria for college graduates’ English proficiency in Taiwan are not based on an island-wide agreed standard. • At tertiary level, the test results from of each university are not comparable due to different criteria. • Many existing “graduation” examinations are progress tests or achievement tests, which can hardly provide comparable, reliable and valid assessment of college graduates’ English language proficiency. • In this respect, Soochow University is no exception.

3. Inadequacy of the English Language Testing System at Soochow University • Current practice is acceptable only at class level for each individual teacher. • But the test results can not be compared with other classes at different levels because the tests are not set on a comparable standard. (Each teacher has 40%.) • Midterm and final exams are simply achievement tests, which are not proficiency tests. • For ELT management at tertiary level, university authorities don’t have reliable and valid data for improving quality ELT programmes or policy making.

4. 2005 First Midterm & Final Scores of English 1

5. 2006 First Final Results of English 1 Reading Listening Total Score • High Level 44.71 31.65 76.36 • Middle Level 40.39 27.70 68.09 • Low Level 31.89 24.16 56.05

6. English Program for Non-English Majors at Soochow University 2006-2007 • The Chapters to be covered for the fall semester are: • Prior to Midterm: Unit 8 Chapter 2 It’s Raining Cats and Dogs, and Frogs! • Prior to Final: Unit 13 Chapter 2 Words to Remember • The Chapters to be covered for the spring semester are: • Prior to Midterm: Unit 7 Chapter 2 Make a Statement—Wear a Tie • Prior to Final: Unit 10 Chapter 1 The Challenges of Space Travel

7. Do we need a standardized English proficiency test at university level? • Our answer is YES! It should not be linked with a particular syllabus, but test students general proficiency of the English language. • A standardized testing system would bring about much beneficial backwash effects on the ELT programmes of this university. • Students would study more (not limited materials in textbooks). • The positive backwash effect can provide better guidance for teachers. • We need without delay a unified English language testing system for quality ELT results.

8. A standardized testing system is to be carried out at this university in 2008 • A small team has been working on the Soochow University English Proficiency Test (SCUEPT) since October 2005 (sponsored by the MOE). • The test will be used as a graduation test. • The purpose is to provide students an easy access to the evaluation of their English competence before graduation. • Tests on the market, such as IELTS, TOEFL (iBT), TOEIC, GEPT, etc. are designed with very different purposes that are not suitable for our students.

9. Our Principles of Test Construction As an important means to evaluate college students’ language competence, our SCUEPT aims at achieving: -- objective and consistent scoring -- positive washback effect on ELT programme -- comparable scoring -- fair test conditions for all test candidates at different places and times -- convenient to conduct the test

10. SCUEPT Design Flowchart • Conduct needs analysis • Make a decision of test content and format • Design test specifications • Design sample tests • Pre-test • Carry out testing • Do scoring and statistic analysis • Establish an Item Bank • Give a test report

11. SCUEPT Test Content (Paper A/B)

12. Receptive Ability and Productive Ability A □In the heat of battle, players have been observed to throw themselves across the court without considering the consequences that such a move might have on anyone in their way. B □I have also witnessed a player reacting to his opponent's intentional and illegal blocking by deliberately hitting him with the ball as hard as he could during the course of play. C □ Does that make any sense? D □It certainly gives proof of a court attitude which departs from normal behavior. Question: Look at the four squares □ marked A, B, C, and D in paragraph 4. Where do you think the following sentence can be added to in this paragraph? Off the court, they are good friends.

13. Reading Speed • Practical readability for upper-intermediate level requires a reading speed of 50 wpm, which means one can read 10 pages per hour. This is practically very useful. • The ratio between time for reading and time for answering questions is designed to be 1: 0.75. Thus, within 35 minutes for the Reading Test, 20 minutes is reading and 15 minutes is for answering. The number of words should be no less than 1100 for reading in our test (EPTSU has 1434 and GEPT has 1096, excluding chart information).

14. Weighting onReading 60% Reading ability is the main training ability and it is also the main means for most college graduates to obtain necessary information. Our survey also suggests that reading is the most needed skill at various work places. So, the weighting of Reading and Listening can be reflected in a ratio of 6:4.

15. Percentage of Items Difficulty of SCUEPT for non-English Majors (15/5/2007) • 16% Difficult, 60% Moderate , 24% Easy ΣCr Item Difficulty (ID) ＝——— N 0.3 ≧ ID > 0 = difficult 0.7 ≧ ID > 0.3 = fine ID > 0.7 = easy

16. 16% Difficult Difficult 24% Easy Moderate Easy 60% Moderate

17. B─Bad Bad 12% A─OK, Acceptable Good 35% F─Fine G─Good OK 29% Fine 24% Percentage of Items Discrimination of EPTSU for non-English Majors

18. Internal Consistency Analysis By SPSS, we conducted internal consistency analysis, and got Cronbach α=0.803. This reliability coefficient is higher than 0.7, a widely acceptable standard, which provides good evidence for reliability.

19. Correlationand Concurrent Validity • Spearman Correlation between the SCUEPT andGEPT to 120 Students at Soochow University shows: • There was a statistically significant positive correlation between the two tests, r = 0.709, p=.000. Statistically, this means they are positively related at high level. • In other words, we may cautiously say that the usefulness of SCUEPT can be partly supported by its high correlation with the GEPT. This is practically very useful for the students.

20. Scoring 百分制 • Scoring: It is suggested that full score be 100, and passing score is 60. This is compatible to the existing scoring system. Raw scores will be first converted into Z score and then T score. Reported score is done by the formula: X − M S = ---------- × 12 + 72 SD S = reported score X = raw score M = mean score SD = standard deviation 12 = SD 72 = norm score

21. A call for a unified English language testing system for college ELT in Chinese Taipei • In order to improve the quality of university ELT programmes, we hold that colleges and universities in Chinese Taipei need without delay a unified standardized English language testing system. • There will be lots of ups and downs in the future, but we are confident we can do a better job !

22. Thank you.