1 / 21

Reference Model for Evaluating Intelligent Tutoring Systems

This document presents a comprehensive overview of Intelligent Tutoring Systems (ITS) and the methodologies for evaluating their effectiveness. It emphasizes the goal of replicating a competent human tutor's adaptive teaching capabilities. The evaluation can be formative or summative, involving various techniques, including software engineering methods and empirical assessments. Key areas explored include learner models, cognitive strategies, user interaction, and the architecture's influence on educational outcomes. The report advocates for realistic evaluation processes to enhance ITS design and effectiveness.

jennis
Télécharger la présentation

Reference Model for Evaluating Intelligent Tutoring Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Reference Model for Evaluating Intelligent Tutoring Systems Esma Aimeur, Claude Frasson Laboratoire HERON Informatique et recherche opérationnelle Université de Montréal

  2. Introduction • Goal of an ITS (Intelligent Tutoring System): produce the behaviour of an intelligent (competent) human tutor who can adapt his teaching to the learning rythm of the learner • Ability to model and reason about domain knowledge, human thinking, learning processes, and teaching process • building an ITS needs also to evaluate the system • lack of evaluation methodology Laboratoire HERON - TICE 2000

  3. Architecture of an ITS Student Laboratoire HERON - TICE 2000

  4. Evaluation of ITS • Evaluation can be formative or summative • Summative: after the design process • Formative: during the design (define and refine goals and methods) • Evaluation techniques used in • ITS • Software engineering Laboratoire HERON - TICE 2000

  5. Evaluation of some ITS • SHERLOCK : experimental evaluation • VCR Tutor : 4 versions compared • Formative qualitative evaluation • iterative design • formative methods • qualitative : characteristics of a situation • quantitative : finding causes and consequences • case studies Laboratoire HERON - TICE 2000

  6. Lester’s evaluation • On animated pedagogical agents (Design a Plant) --- botanical and physiology • 100 middle school students • Important educational benefits : improved problem solving • Better than less expressive animated agents Laboratoire HERON - TICE 2000

  7. Littman- Soloway • What is the educational impact on students ? • What is the relationship between the architecture of an ITS and its behavior ? • External evaluation of the student model (Proust) • Internal evaluation : architecture, how ITS respond to input values Laboratoire HERON - TICE 2000

  8. Van Lehn • Teaching metacognitive skills to implement and evaluate an ITS (SE-Coach guide self-explanation) • Empirical evaluation • Fundamental questions • How the design tools are used efficiently ? • How the learner improve his knowledge ? Laboratoire HERON - TICE 2000

  9. Software engineering techniques • Product : Boehm’model • software doing what the user want it to do • use resources correctly • easy to learn • well designed, code • easily tested and maintained Laboratoire HERON - TICE 2000

  10. HCI techniques • Evaluating design • cognitive walthrough (how easy a system is to learn) • heuristic evaluation (visibility, consistency, flexibility, helps for the user) • model-based evaluation (GOMS) predict user performance Laboratoire HERON - TICE 2000

  11. Evaluationg Implementation • Empirical and experimental methods • Observational methods (user completes a set of tasks) • Query techniques • interviews of the users about their experience • questionnaires : questions fixed in advance Laboratoire HERON - TICE 2000

  12. Learner model • Layers : • cognitive • affective • inferential • Learning time : speed of knowledge acquisition • Tracability : of learner’s actions Laboratoire HERON - TICE 2000

  13. Levels of knowledge Based on works of Gagné (1985) 7 levels of knowledge 4 principal levels Novice Beginner Intermediate Expert Learner model Laboratoire HERON - TICE 2000

  14. Learning strategies • One-on-one • Co-learner • Learning Companion • Learning by Disturbing Laboratoire HERON - TICE 2000

  15. Learning strategies • Diversity • Adaptability (switching to different strategies) • Modification • Memorization • Feedback • Reduce cognitive load Laboratoire HERON - TICE 2000

  16. Curriculum • Generalisation • Consistency • Knowledge articulation • Reusability • Navigability • Maintenance Laboratoire HERON - TICE 2000

  17. Interface • Intuitive : learner should understand easily the functions to apply • Interactivity : allowing the learner to be active in his learning environment • Matching : using words, phrase and concepts Laboratoire HERON - TICE 2000

  18. General • Productivity rate : the most important factor. Time spent to produce one hour of ITS based course • Learning outcomes : performance obtained by the learner after the session • Ease of use of the tools Laboratoire HERON - TICE 2000

  19. Conclusion • Evaluation process is complex but needs to be realistic • Develop first , evaluate later • Evaluation criteria as a guideline for estimation Laboratoire HERON - TICE 2000

  20. Merci de votre attention Laboratoire HERON - TICE 2000

  21. Stratégies d ’apprentissage Tuteur Co-apprenant Compagnon Apprentissage par explication Laboratoire HERON - TICE 2000

More Related