Automated marking of individualised spreadsheet assignments: the impact of different formative self-assessment options

1. Automated marking of individualised spreadsheet assignments: the impact of different formative self-assessment options Paul BlayneyUniversity of Sydney Australia p.blayney@econ.usyd.edu.au Mark Freeman University of Technology Sydney Australia mark.freeman@uts.edu.au

2. Outline • Objectives & motivation • Relevant research • Context & CAA tool • Method • Results • Conclusion

3. Research objectives & motivation for the paper • Objective • To demonstrate how spreadsheets can be used to provide formative feedback to students • To investigate the effects on students of different types of formative feedback (usage restrictions) • Effects on academics and the university • Motive • Higgins 2002 – lack of feedback research • Improve academic productivity • Improve student motivation, knowledge and skills

4. Relevant Literature • IT and learning - no significance difference • Russell (99) • Limited impact when IT introduced into learning if no change in assessment • Biggs (99), Alexander & Mckenzie (98) • New opportunity set for CAA • Scanned paper MCQ • Specialist software (e.g. WebMCQ, QuestionMark) • Standardised course mgmt. system (WebCT)

5. CAA Opportunities • Assign assessment • Clarifying assessment • Complete assessment • Deliver formative feedback • Submit summative assessment • Mark assessment • Generate feedback • Deliver feedback • Evaluate assessment effectiveness

6. The CAA tool • Overview • Worldware …Spreadsheet software (e.g. Excel) • Excel VBA (Visual Basic) incorporated to provide feedback and mark numerical problems • Problem solving application • Financial accounting • Management accounting • Tax • Finance • Etc (maths, stats, physics)

7. CAA Opportunities • Assign assessment • Clarifying assessment • Complete assessment *** • Deliver formative feedback *** • Submit summative assessment • Mark assessment *** • Generate feedback *** • Deliver feedback • Evaluate assessment effectiveness

8. Research Method • Students • Pre and post course surveys • Review of unsolicited comments from online discussion forum • Analysis of learning behaviour from files • Staff • Academics • VBA specialist reflections • Users interviews & surveys • Department/Faculty • Teaching & learning specialist interview • Head of department interview • Uni • IT interview

9. What a student does • Receives weekly assignment file • Completes it • Self-assesses draft answers (different options) • Submits final weekly assignment • Gets mark

10. Demographics of students surveyed • Gender: Male 53% Female 47% • Age: < 21 = 92% 21-25 = 7 >25 = 1% • 1st language: English 49% Other 51% • Working for $’s – hours/week: none 37% 1-5 15% 6-10 23% > 10 25%

11. Student perceptions overall PrePost General • Feedback crucial 94% n/a • Pseudo self-assessment gauge ability 94% n/a • Pseudo self-assessment taken serious 82% n/a VBA • Self-assessment motivated learn n/a 60% • Self-assessment motivated try n/a 82% • Productive way to learn 76% 76% • Non-threatening 76% 79% • Confidence in CAA 64% 55% • Workload for summative value OK n/a 64% • CAA developed accounting problem solving n/a 61% • CAA developed spreadsheet skills n/a 73% • CAA more enjoyable n/a 75% .

12. Demographic analysis of student pre-course perceptions that “feedback is crucial”

13. Demographic analysis of student post-course perceptions • Gender table 3 • No significant differences • Native language – table 4 • No significant differences • Age – table 5 • No significant differences except less confidence in accuracy with older students (p=0.04) • Working hours – not reported in paper • No significant differences except less motivation to learn with working students (p=0.01) and lower confidence in program’s marking accuracy (p=0.08)

14. Analysis of free-response comments solicited from survey

15. Analysis of free-response unsolicited comments • Online discussion forum (Blackboard) • Bulk • Administrative • File location on network? • Digital drop box workings? • Ad hoc • Content • Assignment marking system

16. Student learning behaviour with different self-assessment options Option 1 students performed best: table 7 (p=.0002)

17. Usage of Self Assessment Details of Self Assessment Use (1 student for 1 assignment) 97 times !!

18. What the academic does • Steps to produce self-assessable files • Design question • Create Excel spreadsheet file • Add VBA forms and code modules • Create unique files for students • Update VBA code in Master Marker file • Provide unique files to students • Receive files submitted by students • Run Master Marker file. Post results.

19. Academic staff (developer) reflections and perceptions • The Developer • Accountant by training (academic since 1985) • Lotus 123 user from 1983 to 2000 • Close working relationship with two regimes of IT staff • The CAA Tool • Initial system developed over 10 years ago • Lotus 123 worksheet files and macros • Converted to Excel in 2001 • Used in business education • UG & PG and Professional Education • E.g. Financial Accounting, Management Accounting, Tax, Information Systems • Assignment submission • Email • CMS Digital drop box

20. Academic staff (developer) reflections and perceptions • Benefits • Improve learning • Productivity • Opportunity for reduced teaching load • General support – colleagues and HOD • Opportunity for creativity and challenge • Research opportunity • Contributing to department productivity and reputation • Costs • Exposed to vagaries of IT • Time consuming trouble shooting student errors • When outsourced… • Exposed to poor academic practice • Reliance on others’ expertise

21. Other academic users’ perceptions Free response comments from survey (n=8) support interview feedback

22. Other academic users’ perceptions: potential diffusion of this particular CAA innovation Rogers (95) 5 factors effecting the mainstreaming of innovation (n=8)

23. Effects on Department / Faculty • Benefits • Learning reputation for innovation • Improved productivity • Intra-departmental cooperation • Enhanced morale by supporting innovation • Plagiarism disincentive • Via file individualism • Individual data • Linked cells • Costs • Need to protect against intellectual property exit • “Black box” • Back up re reliance on developer • Exposure to system reliability (CMS, Internet, viruses) • System capacity requirements

24. Effects on University • Benefits • Boundary testing • IT reputation • Learning reputation • Productivity • Breakdown of academic boundaries • Costs • Assessment innovation is high risk • IT assessment innovation is even riskier

25. Conclusions • Confirmation that feedback is crucial • Best learning where students had a limited number of self-assessments • Limitations • Specialist VBA required • Sydney University context • Student propensity to circumvent • Controlling assignment difficulty

26. Conclusions • Future research • Effect of different self-assessment options on summative assessment • Closer analysis of self-assessment and different learner attributes • Relationship between self-assessment assignments and assessment value and assignment frequency • Self-assessment options and learner confidence