SOPHISTICATED TASKS IN E-ASSESSMENT

1. SOPHISTICATED TASKS INE-ASSESSMENT WHAT ARE THEY?AND WHAT ARE THEIR BENEFITS? Andrew Boyle Research & Statistics team QCA

2. Importance of e-assessment • Important in major government initiatives • eLearning strategy • White Paper on 14 – 19 education • QCA’s e-assessment vision • E-assessment commonplace by 2009 • Currently developing KS3 ICT test • CAA well established in UK HE

3. Importance of sophisticated tasks • Substantially different task types in the near future • From pencil-and-paper tests and from early CAA • Tomlinson report/14 – 19 White Paper • E-assessment not limited to multiple-choice testing • Potential to test learners in both structured and unstructured environments • Both short and long answer questions • Ability to use techniques (sic) such as video clips • Assess a wider range of knowledge, skills and understanding than possible at present

4. Possible e-assessment futures • ‘Intelligent measurement’ • Sophisticated assessment methods (e.g. simulations) become more practicable • ‘Reinvented’ learning and assessment • Radically different assessment tasks • Integration of learning and assessment • Highly dynamic and adaptable content

5. Examples of sophisticated tasks • World Class tests • Key Stage 3 ICT test • TRIADs • NAEP-TRE • Concept-mapping tasks

6. Taxonomies of e-assessment tasks • Parshall’s five-dimensional framework • Item format • Response action • Media inclusion • Level of interactivity • Scoring algorithm

7. Towards a definition? • Parshall • Five dimensions likely to be inter-related • Each dimension acts as a continuum • No simple-to-sophisticated cut points • Difficult to find a definition • Cf Mackenzie • Possible to suggest core features

8. Core features of sophisticated tasks • Contain media-rich stimulus material • Would rule out essays – even machine-marked • Test taker interacts with stimulus in various ways • Choosing responses (various ways) • Constructing responses (writing answers, plotting data points) • Directly demonstrating capability (simulations) • Test taker produces complex work • More complex than simple CAA, not necessarily more sophisticated than pencil-and-paper tests

9. Comments on core features • Must contain several features to be ‘sophisticated’ • Notion of ‘sophistication’ always linked to other aspects – not just task design • Test model • Type of data produced • Marking • Reported information • Assessment purpose • Description of tasks as ‘sophisticated’ usually amounts to claim about merits of tasks/test

10. Possible benefits of sophisticated tasks • Measuring different things • Addressing different assessment purposes

11. Measuring different things • Simple items based on a ‘dated form of psychology’ • Learner as a ‘collector of facts and skills’; added to repertoire independently • At odds with recent understanding of learning and experts’ cognition • Conventional test models at odds with test takers’ cognitive strategies • Success in tests depends on at least two dimensions (accuracy and speed) • Test-taking strategies • Models should account for varied sources of difficulty

12. Measuring different things (2) • Curriculum coverage • ‘Higher-order learning outcomes (HLOs)’ • Valued in curriculums, but not easily tested by simple item formats • Simple-item tests might not represent the curriculum adequately • Particularly important if modern curriculums emphasise new types of abilities, etc. • ‘Shared rhetoric in education’ • Emphasis on: Maths and Science; ICT; problem solving and communication

13. Definition of ICT capability • Construct measured in KS3 ICT test • Technical and cognitive proficiency to access, use and communicate information using technological tools • Purposefully applying technology to solve problems, analyse information, develop ideas, create models and exchange information • Discriminating in use of information and ICT tools • Contrasted with ICT skills • Technical competences to carry out tasks using common software applications

14. Challenge for sophisticated-task users • Sophisticated tasks depart from outdated psychological models • Need for new mental models • Significant challenge to researchers • Account for implications of sophisticated tasks • When specifying assessment model • During validation

15. Threats to validityfrom sophisticated tasks • Construct-irrelevant variance • Sophistication and complexity in user interfaces - ‘nuisance variables’ • Detract from validity • Differential impact of new task styles • Learners of differing backgrounds • Different cognitive styles/dispositions

16. Different assessment purposes • Substantial strand of recent research addresses assessment purposes • Formative assessment can improve learning • Formative assessment wide-reaching notion • Applies to formal test instruments and to day-to-day teaching techniques (e.g. questioning styles)

17. Formative e-assessment • Potential of e-assessment to provide useful feedback on aspects of sophisticated tasks • Well-designed formative information more supportive of learning than simple right|wrong • Varied presentation styles • Providing feedback in range of visual and interactive modes • Might include a wider range of learning styles

18. Limits to the optimism (1) • E-assessment is expensive to develop • Costs of e-assessment shifted towards front of cycle • Large budgets necessary for test development • Especially true of sophisticated tasks • Large organisations tend to have most resource to commit to big projects • Generally focused on summative assessment • Often conservative, risk averse • Makes use of sophisticated tasks for innovative formative assessment less likely

19. Limits to the optimism (2) • Writing good test items is difficult. • Good quality items take a long time to produce; iterations of reviewing, editing, etc. • Teachers don’t necessarily have item writing skills • Claim that difficulty in writing good items is particularly pronounced for sophisticated tasks • Sophisticated tasks • Expensive • Slow to develop • Not easily written by non-specialist • May be difficult for teachers and learners to ‘own’ • Lack key characteristic of formative assessment • Assessment closely integrated with classroom practice

20. Discussion • No clear-cut definition of ‘sophisticated tasks’ • Device to distinguish from alleged reactionary use of simple question styles in early CAA • Critique of benefits of sophisticated tasks • Implies a position of informed scepticism • Central assertion is accepted • Sophisticated tasks will soon be ubiquitous • They will bring real benefits • Benefits have been asserted rather than proven • Need to move to informed implementation

21. Discussion (2) – the two benefits • Assessment of HLOs • Formative assessment • Often confounded – they are distinct • Argument stronger in case of HLOs • Empirical evidence in near future • Sophisticated tasks’ ability to facilitate formative assessment – less secure • Difficulty in authoring tasks – continue to be created by large central organisations • Formative assessment ought be to low-tech – usable in everyday classrooms • Use of day-to-day software applications for formative assessment – e.g. track changes in word processors

22. Other work to do onsophisticated tasks • Deriving scores for each task • Measurement models • summarise scores • provide meaningful information about performance/ability, etc. • Validation of tests

23. Mackenzie definition • Advanced Computer-Based Assessment … • goes beyond the simple multiple-choice/response item types into the area of complex question types, adaptive branching, scenarios and simulations of real life situations or problems. In formative mode, feedback may be immediate, extensive, context-sensitive and include annotation of diagrams, links to web resources or full courseware/e-learning tutorials making it an extremely powerful learning tool.