Quality Assurance Strategies: Evaluating Learning Resources

1. Quality Assurance Strategies:Evaluating Learning Resources Presented by: Karin Lundgren-Cayrol In collaboration with: Gilbert Paquette and Suzanne Lapointe for CICE & GTN-Normétic I2LOR WORKSHOP ON REPOSITORIES Montreal, November 8 2006

2. Overview • Goal and Objectives • Basic Concepts • Learning Ressources (LR) • LR Repository Types • Quality Assurance Strategy • Quality Dimensions • Integration/Development Processes • LR Life-Cycle and Quality Assurance Strategies • Conclusion

3. Goals and Objectives Provide a set of Quality Assurance Strategies to ensure LOR sustainability • Objectives • Establish quality assurance strategies • Provide pertinent evaluation criteria • Provide guidelines and principles • Repository Managers & Developers • LO designers • LO users

4. Learning Design Electronic book SCORM Object CD-ROM Video Software Image What’s a Learning Resource ? Any entity, digital or non-digital, which is intended to be, or may be used for the purpose of learning, education or training  Granularity

5. Organisational and Technical Decisions Identify Quality Assurance Strategies List of Stakeholders Repository Type User Profile List of Needs Metadata Profile Integrating a Repository • Design and Produce • Integrate into LR • Reuse and Revise Quality Assurance Strategy? ISO 9000: 2000 Quality Assurance is the planned and systematic activities put in place to ensure quality requirements for a product or service will be fulfilled. Document all processes.

6. Quality Dimensions • Instructional Quality Content clarity and conciseness, instructional strategies aligned to the learning objectives, appropriate media according to target audience, etc… • Media and Ergonomical Quality User-friendliness, motivating, visually attractive, built-in accessibility features, etc.. • Technical Quality Technical interoperability and robustness, metadata schema and tagging procedures, conformance to standards

7. The LR Life-cycle and how to identify Quality Assurance Strategies Process preceeding the integration/development of a Learning Object Repository

8. Technological Infrastructure Institutional Support Stakeholder support Policy on copyrights Integrating a LR Define Goal of LR Specify Metadata schema Goal Specify Resource Types Identify Actors Identify Needs Resource Types & Expertise Stakeholder list LISTof User Profile Strategies at the Organisational Level • Repository Type • Stakeholders • User Profile • List of Needs • List of Resource Types • Metadata Profile

9. REVISION All dimensions Pedagogical/Media Quality Technical Quality LR Life-Cycle and Identifying Quality Strategies Life-cycle of a LR After inclusion Before inclusion During contribution Design/ Production Adaptation/ Reuse/Assignment Deposit and metadata referencing

10. LR Repository Types and Main Charactersitics • Private Repositories • Quality = personal usefulness of the LO • Metadata schema • Recommendation systems (social/collaborative tagging/folksomy) • Community repository • Quality = Member ship and peer reviewed LO’s • Domain specific interest, practice or discipline • Protects the consumer trough membership • May use domain ontology • Public Repository • Quality = Annotation and Right’s Management • Metadata validation system and automatic capturing • Propose evaluation criteria for different resources • Provide evaluation, recommendation and annotation systems

11. Summary Graphical designer: Julie Aubin, 2006

12. Give me a chance! Too Much is OverkillToo Little Just Kills Karin Lundgren-Cayrol Suzanne Lapointe Gilbert Paquette CICE, TÉLUQ - UQAM

13. During Contribution Strategies here could be referring to all types of quality dimensions: • Pedagogical and Ergonomical Quality by enforcing Peer Review • Controlled contribution But if Strategies has been applied at the design/production stage, strategies here normally only need to refer to • The quality of the metadata • Interoperability of the Learning Resource and its Metadata Record

14. During Contribution Multi-actor expertise for metadata • Peer Review Strategies: • Involving Content Expert / Author /ID Expert • Title, LR language, key words, description, type, version, contributors, intended end user, learning context • Text mining algorithms can help • Metadata Quality Strategies • Library technician • Version, classification, rights, relations metadata • Overall respect of the standard used • Interoperability • Computer Technicians • Format, size, required conditions, Installation Remarks, meta-metadata, record language General Principle: Reduce form-filling: use wizards, smart automatic or semi-automatic computer agents

15. During Contribution • Demand Membership for contributors: • Gives responsability and motivation; • Make contributors visible by ‘Business Card’ portfolio, chat groups etc. • Suggest domain specific Community of Practices • Make sure that the author provides the following infos: • Degree of Pedagogical Reusability (Rights) • Content quality indications (Method,checklist, evaluation used etc) • Interface quality (idem) • How it might be an efficient learning or teaching tool

16. Media Elements Units of Learning Tools Documents Actors Learning Design as Agregated Objects Basic Resources Operations Scenarios Processes

17. Before Inclusion Strategies that refers to Instructional Design of the Learning Resource and aims at: • Pedagogical Quality • Ergonomical Quality Demands specilized ID methods if LR is a Unit of Learning compliant with : • IMS LD • SCORM • Other like WEBCT, BlackBoard etc..

18. Some Guidance Principles • Design and Production Strategies • Use a solid and adapted ID method • Support Collaborative design, that is let specialists apply their expertise • Clearly identify knowledge and user competencies • Favor pedagogical strategies putting the learner in the center • Apply evaluation criteriaduring development and implement at least one learner/peer evaluation cycle • Be informed about Access4All production principles

19. Instuctional Engineering Method MISA Problem definition 100 Training system 102 Training objectives 104 Target Learners 106 Actual situation 108 Reference documents Instructional Modeling Knowledge Modeling 210 Knowledge modeling principles 212 Knowledge model 214 Target competencies 310 Learning units content 410 Learning instruments content 610 Knowledge and competency management 220 Instructional principles 222 Learning events network 224 Learning units properties 320 Instructional scenarios 322 Learning activities properties 420 Learning instruments properties 620 Actors and group management Delivery Modeling Materials Modeling 240 Delivery principles 242 Cost-benefit analysis 340 Delivery planning 440 Delivery models 442 Actors and user’s materials 444 Tools and telecommunication 446 Services and delivery locations 540 Assessment planning 640 Maintenance / quality management 230 Media principles 330 Development infrastructure 430 Learning materials list 432 Learning materials models 434 Media elements 436 Source documents 630 Learning system / resource management

20. Collaborative Design Note: Interactive Objets are Software and answers to software quality criteria • Content Expert • Instructional Designer • Media Specialist • Delivery Specialist • Project Leader Basic principle: Build or integrate objects that you can quality certify

21. The LORI Evaluation Criteria Nesbit, Belfer & Vargo, 2003 • Usability Ease of navigation, predictability of the user interface, and the quality of UI help features • Accessibility Support for learners with disabilities • Reusability Ability to port between different courses or learning contexts without modification • Standard conformance Adherence to international standards and specifications • Content Quality Veracity, accuracy, balanced presentation of ideas, and appropriate level of detail • Goal Alignment Alignment among learning goals, activities, assessments, and learner characteristics • Feedback and Adaptability Adaptive content or feedback driven by differential learner input or learner modeling • Motivation Ability to motivate, and stimulate the interest or curiosity of, an identified population of learners • Visual Design Design of visual and auditory information for enhanced learning and efficient mental processing

22. LORI instrument Adding: General quality or the rate (X/45) or % ? Other criteria ?

23. Accessibility4All Checklist of Checkpoints for Web Content Accessibility Guidelines 1.0 The TILE PROJECT : http://barrierfree.ca/tile/index.htm

24. After Inclusion in the Repository Refers to strategies that provide: • Retrieval quality • Free text search • Browsing by Classification Schemes or Typologies • Ontological search • Federated search • Easy display of information • Descriptive resumé • Full meta data record • How many used it • What do they think • Peer reviews and evaluations from actual users/reusers. • Simple Annotation systems • Social tagging/bookmarking and Folksomy • Collaborative filtering techniques • Recommendation algorithms

25. After Inclusion Strategies • List of new and innovative high quality LRs • Suggestions for: • Revisions to authors/designers for improvements • Renewal of resource • New resource

26. MERLOT

27. Metadata display

28. PALOMA WEB Federated search displays Source Repository

29. EDNA Online

30. SEARCH ARIADNE DISPLAY

31. Social Tagging Information FORMIST/deLicio.us

32. Technical information Rights information Reviews information SMETE