E-Assessment: Evaluation, vision and plan

1. E-Assessment: Evaluation, vision and plan www.jisc.ac.uk John Winkley, e-assessment JISC E-learning Programme

2. Agenda • A vision for e-assessment • Findings from our evaluation • Recent e-assessment activity in the e-learning programme • A shift of emphasis

3. JISC – The structure of the e-learning programme

4. A draft vision for e-assessment (by 2015) • E-assessment is used appropriately and with confidence … in a wide variety of HE curricula. • E-assessment … adds to traditional assessment processes with efficiency improvements, as well as improvements in assessment quality... • E-assessment is used at all stages of the learning programme… • Information from different types of assessment is interoperable... • The day-to-day use of e-assessment is encouraging change in the nature of assessment... E-assessment is embedded in institutional policy… • A wide range of technologies are used to support e-assessment in diverse forms… • Developments in JISC’s infrastructure projects… provide a development platform which allows new technical and pedagogical assessment opportunities to be researched, developed, piloted and exploited efficiently. • A vibrant world-class community of research underpins technical and pedagogical developments in e-assessment….

5. e-learning programme – activity types • Investigates - Institutional-level demonstrators/pilots • Describes - Research, studies, mapping and modelling • Analyses - Review and Evaluate • Creates - Commission Development • Informs - Consult and Inform (Knowledge Exchange ) • Guides - Communication and Dissemination • Project and Programme Support, Programme Management • Recent move toward larger pedagogy-focused programmes with “cross-cutting themes”, eg OER, Curriculum Design and Delivery.

6. eAssessment - Baselining • Programme Management • Underpinning Research • Technology • Content • Embedding and Rollout

7. Programme Management • TODAY • Vibrant but small and specialised. • Too many technologists, not enough educationalists. • Recent efforts to get the two sides talking appear to be working. • Not enough vendors. Slight reluctance from academics. • Hard to maintain but important. Complex education sector, many agencies at the intersection of technology, pedagogy and exams. VISION • Established vibrant, open and mixed community of technical developers, educators and assessment specialists working to form, support and develop JISC’s vision for e-assessment. • Vigorous stakeholder engagement

8. Underpinning Research • TODAY • Anecdotal evidence: • Much adoption, but limited evidence of widescale use of e-assessment in HE. Different elsewhere. • Relies on early adopters and enthusiasts. Poor scaling. • large gap behind the ‘bleeding edge’ and the day-to-day life of departments. • Policy and decision makers are critical. • Ok for Int’l HE, little wider view. VISION • Regular measurement of progress with embedding. • A clear understanding of institutional barriers to take-up and the best methods to fix. • Clear understanding of international best practice.

9. eAssessment - Underpinning Research (2) • TODAY • Patchy. Importance very clear. • E-assessment generally not in HE policy and strategy. • Leading role in IMS. QTIv2 thoroughly explored. Killer business case not found yet. VISION • Technical research underpinned by clear pedagogy. • Alignment of best practice between e-assessment and assessment. • Successful tracking, influencing and adoption of interoperability standards.

10. eAssessment - Technology • TODAY • Successful OSS projects: Reference player, item bank, item dev kit, peer review tools. But often poor takeup and sustainability. • Excellent proof of SOA and OSS in practice. • Needs extending for workflow and enterprise applications. • Human processes • “System” integration for information sharing • Struggling to make e-framework work for practical e-assessment applications. VISION • Successful group of OSS toolkit and demonstrator projects provide inspiration and seedcorn for larger developments. • Technical framework

11. eAssessment - Technology (2) • TODAY • Formative e-assessment pedagogy not quite there, but big strides made recently toward a clear framework to support development. • Wide range of tools and processes under development in a wide variety of settings. VISION • E-assessment embedded in formative assessment, supported by range of OSS tools and components

12. eAssessment - Content • TODAY • The QTI1.x and 2.x specifications are reasonably stable. Untried on large scale in UK. Attempts within FE and schools sector have had mixed success. • The two “killer applications” of content interoperability : avoidance of lock-in and content sharing both require a more widespread use of e-assessment than currently seen. • Some doubt about whether HEIs really wish to share content. (Also some examples of success where external QA is important). • It isn’t yet, in most HEIs. The tools are not yet ready to support it. VISION • Content Interoperability is supported, removing concerns about vendor lock-in and allowing sharing of content. • Content management of assessment content is well understood.

13. eAssessment – Rollout, embedding • TODAY • Significant shortcomings and challenges faced by HEIs. E-assessment commonly run by small groups of “enthusiasts” who spend much of their effort on maintaining a reliable service rather than on managing assessment performance. • Yes, where its use is driven from pedagogy. • E-assessment appears to have a low profile within QAA and HE institutional policies (not fully investigated). • Communities of subject specialists appear to provide the richest community for successful takeup. VISION • HEIs have a clear understanding of how best to measure and manage quality in e-assessment systems including at policy level. • There is clear evidence of the benefits of e-assessment. • E-Assessment is widely used in HEIsand embedded in institutional and sectoral policy: “crossed the chasm”. • Content management of assessment content is well understood.

14. eAssessment – Rollout, embedding (2) • TODAY • Technically possible but difficult to organise. • Empirical benefits very clear. Some examples of good practice. • Likely to be a major focus of next phase of JISC e-assessment work. VISION • Information from assessment including e-assessment is used effectively within learning programmes. Information from different types of assessment is interoperable, allowing it to be combined seamlessly, to provide a rich and dynamic picture of learning to date and planning for future learning, supporting personalised learning programmes.

15. Focus of recent activities • Research to support development of a more diverse set of e-assessment approaches and technologies – “technology-enabled assessments” • Investment in formative e-assessment development • Research to investigate quality processes around e-assessment • Investment in e-assessment as a more integrated part of the ‘learning environment’ • Community development and engagement

16. E-Assessment: Evaluation, vision and plan www.jisc.ac.uk John Winkley, e-assessment JISC E-learning Programme