This study

1. Evaluating the Usefulness of Feedback to Immersion Programme Students at the Pre-Programme and Post-Programme Stages Eva Lui Oct 2 2008

2. This study Immersion Programme (4-6 weeks)  Language support system:Language guidance and performance feedback • Usefulness--- advantageous, helpful, or of good effect to students? Eva Lui Oct 2 2008

3. This study • The Phenomenon • Students with varying levels of English proficiency entered the university but most not at a good user level at exit in general New English initiatives to improve students’ English proficiency, e.g. language immersion programmes Language support system (test  reports) to enhance the effectiveness of the initiative • Problem • Language support system is labour intensive and expensive • Significance of the Problem • The system will be a waste of resources if • Students do not work on their weaknesses according to the suggestions of the reports • The feedback is not useful in helping students improve their weaknesses Eva Lui Oct 2 2008

4. This Study Research Objective To ascertain the usefulness* of the Language support system Research Questions • What purposes were served by the system as perceived by: 1.1 the developer (the researcher) 1.2 the feedback users (students) ? • How expensive was the system? 2.1 What was the actual cost of running the tests? 2.2 What was the actual cost of marking and providing feedback to students? • How useful was the feedback to students? 3.1 Did all students refer to the feedback to work on their language weaknesses? If not, why not? 3.2 How did students make use of the feedback? 3.3 Did the system help students improve their language weaknesses? 3.4 Did students find the system useful? • *Usefulness of the Language support system • the system is • serving some purpose • advantageous, helpful, or of good effect to students Eva Lui Oct 2 2008

5. An overview of theLanguage Support System Eva Lui Oct 2 2008

6. Reading report Listening report Pre-programme Tests Writing report Speaking report Reading report Listening report Writing report Speaking report Tests  Feedback Reports Language learning plan Immersion programme University study Updated Language learning plan Post-programmeTests Eva Lui Oct 2 2008

7. Purposes served by the system---From the viewpoint of the developer Eva Lui Oct 2 2008

8. Purposes served by the system • Viewpoint of the developer (the researcher) The language problem and the design of the immersion programme Students’ problems • demanding, impatient and bad at communicating’, weak spelling, grammar • Expectations on the immersion programme and the language support system Learning & teaching • Student-centred • Emphasis on application & knowledge • Focus on learning Motivation • Rewards, threats, externally imposed rules, identified regulation, integrated regulation Responsibility in learning: • ‘learning success and enhanced motivation is conditional on learners taking responsibility for their own learning,being able to control their own learning and perceiving that their learning successes and failures are to be attributed to their own efforts and strategies’ (Dickson, as cited in Dornyei, 2001) Dornyei, S. (2001). Teaching and researching motivation. Pearson Education. Eva Lui Oct 2 2008

9. The lmmersion programme Language enhancement in 4-6 weeks  Hosting universities in Australia, UK, New Zealand -lessons -voluntary work -activities -homestay Eva Lui Oct 2 2008

10. Language support for students in the immersion programme • Students • Individual; learn by experience; communicate by hands-on learning, participation; set short-term goals; manage by following own thinking. • Training style: interactive & multi-modal  • Learning & teaching • Student-centred • Emphasis on application & knowledge • Focus on learning • Motivation & responsibility in learning • Externally imposed rules, identified regulation • Control their own learning What about other features? Our solution: providing diagnostic feedback  Course requirement + Scaffolding support  learner-centred learning   motivations Eva Lui Oct 2 2008

11. Pre-testing & Post-testing: Testing and feedback design Eva Lui Oct 2 2008

12. The test Based on IELTS Eva Lui Oct 2 2008

13. Pre-Test Feedback ReportSubjectively marked papers Eva Lui Oct 2 2008

14. Pre-Test Feedback Report Objectively marked papers Eva Lui Oct 2 2008

15. Post-Test Feedback ReportSubjectively marked papers Eva Lui Oct 2 2008

16. Post-Test Feedback ReportObjectively marked papers Eva Lui Oct 2 2008

17. Purposes served by the system---From the viewpoint of the feedback users Eva Lui Oct 2 2008

18. This Study – Data Collection Eva Lui Oct 2 2008

19. Questionnaire items • Students’ comments on: • Usefulness of the reports---USEFULNESS IN • understanding strengths and weaknesses in the different language skills • helping to draft the language learning plan • being informational • providing tips on what to do to improve own weaknesses Eva Lui Oct 2 2008

20. Questionnaire items • Students’ comments on: • Feedback reports: Writing, Speaking, Reading & Listening---USEFULNESS IN • Commenting on strengths and weakness • Commenting on own performance • Providing examples of errors • Providing suggestions on the weaknesses could be improved • Providing additional links in the report Eva Lui Oct 2 2008

21. Interview items • Students’ comments on: • What they could still remember from the reports • How they used the reports • Whether they found the reports useful in helping them improve their English Eva Lui Oct 2 2008

22. This study– Data collected Only preliminary findings are available for this presentation Eva Lui Oct 2 2008

23. Other data source • Students’ growth data • Students’ performance in the two tests Eva Lui Oct 2 2008

24. Purposes served by the system • Viewpoint of the feedback users (students) • From individual interviews: • ‘give us detailed comments on our strengths and weakness’ • ‘tell us what to work on to improve our weaknesses’ Eva Lui Oct 2 2008

25. Costs Eva Lui Oct 2 2008

26. Cost of the system • Actual cost of running the test (costs incurred) *For this group of app 200 students, just a rough estimation of the costs incurred **At the rate of AUD1.0=HKD6.5 Eva Lui Oct 2 2008

27. Cost of the system • Actual cost of the system (costs incurred) Total cost (rough estimation) for -a student: AUD215 -a group of 200 students: AUD43,000 (HKD2,795,000) Reference: Annual undergraduate tuition fee per student (government funded): app AUD 7000 *For this group of app 200 students, just a rough estimation of the costs incurred **At the rate of AUD1.0=HKD6.5 Eva Lui Oct 2 2008

28. Usefulness of the feedback to students Eva Lui Oct 2 2008

29. Questionnaire findings • Usefulness of the reports- • The report helped me understand my strengths and weaknesses in the different language skills. • The report was useful in helping me draft my language learning plan. Eva Lui Oct 2 2008

30. Questionnaire findings • Usefulness of the reports- • The report was very informational. • The report gave me tips on what to do to improve my weaknesses. Eva Lui Oct 2 2008

31. Usefulness of writing feedback Eva Lui Oct 2 2008

32. Usefulness of speaking feedback Eva Lui Oct 2 2008

33. Usefulness of reading feedback Eva Lui Oct 2 2008

34. Usefulness of listening feedback Eva Lui Oct 2 2008

35. Students’ viewpoint on usefulness of the reports (summary) • Overall: • Usefulness much appreciated by students • Helping them understand their strengths and weakness • Report being very informational • Higher ratings in pre-test reports than post- • Sub-skills • Usefulness highly appreciated by students • Speaking reports (in all areas apart from additional links) • Usefulness much appreciated by students • Writing– Examples of errors & suggestions • Reading and Listening– not as highly appreciated (means: 3.5-3.7) Eva Lui Oct 2 2008

36. Usefulness of the feedback--Comments collected from interviews -’The diagnostic report helped me understand my weaknesses. I tried harder in these areas during the trip.’ -’With the help of the report, I can see my own weaknesses. E.g. I didn’t know that my listening to shapes was weak. The suggestions on my weaknesses in organising my writing were useful. I followed the suggestions to highlight the topic sentences. The suggestions on how to improve were detailed enough.’ -’In the pre-test report, it says I was weak in identifying main ideas in listening. I knew it since I usually too focused on individual words and missed the overall ideas. Now with the report, I worked on the weakness.’ Eva Lui Oct 2 2008

37. Usefulness of the feedback--Comments collected from interviews -I witnessed my own improvement in speaking. The diagnostic report pointed out that I was not fluent. Now, I’m more fluent and less scared to speak. The reports actually confirmed what I knew about myself. The post-test reports helped keep me up with the standard and what I could do in Hong Kong but I’m too busy now to do anything.’ -’The reports were very detailed and specific enough. I could work on the skills clearly.’ -’I read the pre-test reports more in detail but not the post-test.’ -’I treated the post-test report as a progress report, took note of it and happy that I have less unfavourable feedback.’ Eva Lui Oct 2 2008

38. Usefulness of the feedback--Comments collected from interviews -’In the programme, there was more speaking and listening practice. I didn’t have many friends and so spent more time on blogging. My writing is improved.’ ‘-’Additional links? I never saw them.’ -’The Internet access was not useful and so I only relied on the hard copy of the reports’ (hence, missed the additional links) -’Why not many returned their feedback? Some of them are not interested in improving English and so not willing to do too well in the test nor reading the reports in detail…’ Eva Lui Oct 2 2008

39. Usefulness of the feedback • 3.1 Did all students refer to the feedback to work on their language weaknesses? If not, why not? • 3.2 How did students make use of the feedback? • 3.3 Did the system help students improve their language weaknesses? • 3.4 Did students find the system useful? • Yes, especially when it is required by the programme • Some read it and then devise their plan accordingly. A few referred to it closely. Many did not pay full attention to it • -More useful in subjectively marked papers • -Feedback on Speaking more useful than Writing Eva Lui Oct 2 2008

40. Students’ growth data Eva Lui Oct 2 2008

41. Evaluating the Usefulness of Feedback to Immersion Programme Students at the Pre-Programme and Post-Programme Stages Eva Lui Oct 2 2008

42. Evaluation • Costly • Effective if • reference to the report is mandatory • students are motivated enough • More useful if the skills assessed are more relevant to students’ needs and the suggestions are readily adoptable • Most students improved after the programme but unsure with the contribution of the system Eva Lui Oct 2 2008

43. What next • Continue with the evaluation exercise • Improve the system with the aims of • More efficient and effective feedback provision • Lowering the cost of the system Eva Lui Oct 2 2008