The Machine Scoring of Essays: Redefining Writing Pedagogy?

1. The Machine Scoring of Essays: Redefining Writing Pedagogy? Deborah Crusan Wright State University

2. My talk this morning will • focus on one of the programs typically used for classroom writing/writing assessment - MY Access! • define MY Access! • discuss a small study in which a few ELLs used MY Access! • report their attitudes toward MY Access! • wonder about the impact of MY Access! and similar programs on second language writers and writing.

3. But first, we must remember ~ • "For there is nothing either good or bad, thinking makes it so."~William Shakespeare (1564-1616), Hamlet, II.ii • My translation: We are (well at least I am) very good at creating dichotomies of good and bad • So just for now, let’s suspend our biases . . .

4. So . . . What is MY Access? • MY Access! is a “web-based instructional writing product that provides students enrolled in grade 4 through higher education with the opportunity to develop their writing skills within an electronic portfolio-based environment” (Vantage Learning, 2007, p. 1). • MY Access! grades students’ responses to select writing prompts and offers suggestions for improvement to the text in a matter of seconds.

5. MY Access claims • that their program decreases writing instructors’ grading burden • that the almost instantaneous feedback provided by MY Access! motivates students to revise more.

6. Problems • Warschauer & Grimes (2008) found that students rarely use programs like MY Access! to revise anything other than surface errors • Teachers can create their own prompts; however, MY Access! will score only those prompts included in the program • In many cases, longer essays receive higher scores; consequently, MY Access! assumes that length is a proxy for fluency.

7. The study • Graduate students worked with ELLs, asking them to respond three times to two different prompts • Students responded to a survey about their experiences with the program • Additionally, grads interviewed ELLS regarding their attitudes toward MY Access! • For the purposes of this presentation, I’ll highlight comments from six of the students who agreed to be interviewed • Overall, students’ opinions about MY Access! were mixed; students found useful aspects as well as aspects they termed less helpful.

8. Student attitudes - Hannah • Hannah reported that she was excited about the program, but because she had little computer experience, she had difficulty learning the program. She had problems remembering what to do each time she logged on because she didn’t use the program very often. She found that the MY Access! prompts were not very interesting.

9. Xiao • Xiao admitted that it was very easy to find and insert information from other sources into his writing – even if it wasn’t his ideas. He found MY Access! feedback useful for making surface corrections – the program pointed them out, so he could correct them – but he wasn’t motivated to revise his work beyond fixing the surface errors.

10. Heecheon • Heecheon believed that he could get the same kind of information from Microsoft Word. Initially, he liked the instant feedback My Access! provided, but he began to feel overwhelmed as he read through the massive amount of information offered to him by the program.

11. Mohamed • Mohamed believed that MY Access! would be beneficial in the classroom. He found it to be a great tool to use during the writing process because it gave him a variety of tools and activities to aide in the revision of his writing.

12. Farin • After writing and submitting his first draft, Farin thought it was scored too low, so he went to the Internet and found some material loosely related to his topic. When he inserted the material into his essay, his score went up even though the essay was not as coherent as it had been when he first wrote it.

13. Wafa • Wafa felt that MY Access! helped her get into the habit of doing multiple drafts. She liked the editing tools and found some of the feedback very helpful in the revision of her essays.

14. Consensus • Students enjoyed writing with a computer • They liked the immediate feedback provided • However, they often found the feedback overwhelming • Students appreciated help finding grammar errors • Many became bored • Many students did only surface editing, not revision • Several students admitted adding text from the Internet • Students did not use features in MY Access.

15. Should you use MY Access! in your second language writing classroom? • It depends! • Teachers need to ask themselves questions about these programs and how their use might impact their pedagogy • However, this technology is here to stay, so . . . • Teachers must understand this technology and strive to keep assessments locally controlled, context-specific, and accessible