Enhancing Digital Educational Materials: QuADEM Project Assessment

1. Antwerp - 9 February 2007 Assessing digital educational materials

2. Project Aim Developing a generic method for quality assessment of digital educational materials (professional and academic writing skills). Target audience • students • Instructors • developers

3. Project Partners • University of Antwerp • Ghent University • Higher Institute Antwerp • Radboud University Nijmegen • Vergouwen Overduin • Stroomt • Lodz University • Zürcher Hochschule Winterthur • Swedish TelePedagogic Knowledge Center

4. Project Phases • Research • Development • Evaluating and testing • Dissemination

5. Research methods • Card sorting • Observational analysis • Semi-structured interview • Focus group

6. Calliope & WritingStudio • Semi-structured electronic learning environments • Multilingual • Blended learning • Professional writing and communication skills

7. Matrix usability • The layout of the learning environment is appealing. • The layout of the learning environment is efficient. • It is easy to find specific information. • The integration of multimedia in the learning environment is optimal. • First time users find it easy to use the learning environment. • The digital learning environment meets the needs of the user. • The digital learning environment functions as it should.

8. Card sorting

9. Task analysis

10. Procedures • Card sorting • Very insightful, but not for usability (transfer to content) • Task analysis • Real actions vs. Session actions • Morae recordings prove to be added value • Semi-structured interview • Offers ad hoc explanations • Focus group • Very insightful

11. Conclusions • Card sorting • Select research subject carefully • Task analysis • Valuable technical info • Intuitive responses • Useful input for focus group • Interview • Verbalising explicit info • Focus group • Useful roundup

12. Conclusions • Usability testing allows/forces us to adopt the language of the users. • Usability testing indicates that it is difficult to distinguish between different sub domains. • Example: • Do you think the learning environment is efficient? • Yes, it contains the information I’m looking for.

13. Program • Welcome & Introduction • Online Learning • Theoretical background to QuADEM • How does it work? • DIY • Discussion

14. Introduction • Stands for ‘Quality Assessment for Digital Educational Materials’ • Funded by EU and in collaboration with international partners • Method for the development & review of digital educational materials • Who are you?

15. Online learning

16. Context • Socio-economic aspects • New conceptions of learning • New technology • Innovative new learning environments with computers & internet as new tools

17. Context

18. Blended learning • Blended learning can combine • modes of web-based technology • various pedagogical approaches • any form of instructional technology • instructional technology with actual job tasks (Driscoll 2002)

19. Blended learning • Blended learning and • individualized learning • problem-based learning (Emig, 1977 ; Flower & Hayes, 1980) • narrative learning (Cf. Vygotsky’s zone of proximal development) • social constructivism

20. Blended learning • Valathian • Skill-driven learning“a combination of self-paced learning with instructor or facilitator support to develop specific knowledge and skills” • Behaviour-driven learning“a combination of various events and delivery media to develop specific behaviours” • Competency-driven learning”a combination of performance support tools with knowledge management resources and mentoring to develop workplace competencies”

21. Blended learning • Kerres & De Witt

22. Blended learning • Khan

23. Blended learning • Graham • Activity-level blendingis a combination of face-to-face an computer-mediated activities. • Course-level blendingis a combination of face-to-face and computer-mediated activities as part of a course. • Program-level blendingoften entails one or two models: either the participants choose a mix between face-to-face and online courses, or the combination is prescribed by the program. • Institutional-level blendingoccurs when an institution has made an organisational commitment to blend face-to-face and computer-mediated activities.

24. Blended learning • No prefixed blend: modular approach to criteria • Which criteria matter to you?http://www.writingstudio.eu

25. Development process • QuADEM objective • To develop a method for Quality Assessment of Digital Educational Materials. • Assessment based on preset criteria • Through an iterative process of evaluation and piloting • By domain experts and researchers

26. Development process • Delta phase • Literature study • Framework development • Gamma • Setting criteria • Determining evaluation procedure • Finetuning criteria & evaluation • Beta • Validating evaluation procedure • Finetuning evaluation procedure • Alpha • Fixing criteria

27. Method • Overall composition • several aspects of quality = components • to evaluate one component = unit • core ingredient of a unit = checklist • a checklist = list of criteria

28. Method: components • A component represents one aspects of what constitutes a successful digital learning material. • Content • Assessment • Interculturalaspects • Learning styles • Usability • Pedagogy • Style & language • Multimedia • Questionnaires

29. Method: unit • A unit contains all information you need to evaluate a specific component of an online learning environment • Summary • Fact sheet • Score • Preconditions • Checklist • Methods • Procedure • Manual

30. Evaluation • Starting situation • user is developing a new digital learning materials or revising existing materials • Steps • determine goal & focus • determine assessor • select units • check preconditions • select research methods • select respondents • design procedure • gather rating information • determine scores

31. DIY 1

32. DIY 1 • Usability • Focus group

33. DIY 2

34. DIY 2 • Assessment • Think aloudinterview

35. Disclaimer • The QuADEM project has been funded with support from the European Commission. • This document reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.