Auditing Subject Knowledge in Initial Teacher Training using Online Methods

1. Auditing Subject Knowledge inInitial Teacher Training usingOnline Methods

2. Auditing Subject Knowledge in Initial Teacher Training using Online Methods Project team: Kerie Green Lynne Jones Barbara Kurzik David Longman Technical support: Dan Jones Allan Theophanides

3. Rationale • Subject audits are not new and have been used in our institution to determine existing knowledge for some considerable time. • The introduction of the standards for ITT as described in circulars 4/98 (England) and 13/98 (Wales) was the stimulus for this. • Subsequently, experimental electronic versions were created for core subjects. • These were designed to provide instant assessment and feedback for staff and students. • Recognition of the value and potential of electronic audits led to further developments in the areas of English and mathematics. • The objective of this project was the design and testing of online methods of assessment in these two subjects.

4. Questions: Answers: What? Who? Why? How? Analysis Limitations Observations Progression

5. What do we mean by ‘auditing subject knowledge’? • Is it diagnostic? • Is it assessment for learning? • Is it assessment of learning? • Is it all of these? • Through this project formative and summative assessment methods were explored and developed.

6. Who were to be audited? • First year trainees on the BA (Hons) Primary Studies Programme were the target for the English audit. • Second year trainees on the same programme were subjected to a similar model in the online examination. • First year trainees on the Secondary BSc programme were the target for the mathematics audit.

7. Why do students need to audit knowledge and skills? • An entry requirement for our Primary Studies Programme is that trainees need the equivalent of GCSE English at grade B. • Similarly, trainees are recruited to the Secondary BSc degree on the understanding that they are competent to a level equivalent to a GCSE Higher Tier. • However, trainees enter these courses with shortcomings that need to be identified and addressed.

8. How were the audits created? • The tool of convenience was QuestionMark Perception (QMP). • Audit questions were provided by subject specialist tutors and audits were extensively trialled with colleagues, trainees, teachers and Y12 students. • The English audit and examination used a variety of question formats including multiple choice, and free text entry. • The mathematics audit was presented as multiple choice format. • Audit items were drawn randomly from banks of equivalent questions. • With both audits trainees received instant and automatic feedback on their answers with links to supporting materials on each question. • English audit example • Maths audit exampleTypical audit feedback

9. Analysis • QMP allows for some data analysis but the most productive tool was found to be Excel. • Conditional formatting provided instant visual highlighting of areas for concern. • The usual numerical analysis was, of course, also useful so averages, standard deviations and graphs could be applied where needed. • English examination dataMathematics data

10. Limitations • QMP is designed for the non-specialist to create on-line surveys or tests. It includes a range of fixed template styles and operations. • Our requirements for the display of questions and feedback could not be easily accommodated by QMP.

11. Observations • The need for evaluation and checking of content accuracy and operational efficiency cannot be underestimated. • Question format is a challenge and it takes time and skill to design questions that are unambiguous and test objectively. • The method provides for consistency and accuracy of marking. • The value of the data analysis in terms of highlighting cohort weaknesses has proved compelling as it has in identifying individuals’ subject difficulties. • There is significant potential for the use of formative audits as part of professional development for trainees and to enhance partnership with schools.

12. Progression • Replace QMP with custom made system based on SQL and ASP.NET. Funding constraints have prevented progress so far but the intention remains. • Extend the audit process to cover literacy and numeracy for the same and other programmes (and schools?). This is a current project that is nearing completion. • Incorporate audit procedures in Personal Development Planning. Institutional discussions are taking place but the method has not yet been implemented. • Engage with external agencies such as DCELLS. Meetings to share ideas have taken place with WJEC.

13. Summary • This project was conducted during 2006-7. • The objective was to design and test online methods of assessment in the subject areas of English and mathematics. • In English an existing knowledge audit was further developed to incorporate feedback to individual students with topic specific support links. • Additionally a version of this audit model was adapted for use as an online examination. • In mathematics an audit was created as a formative exercise including links to support materials. • The data generated also informed tutors about areas for development across the cohort. • The findings have clear implications for curriculum design as well as professional development.