Approaches to developing more intelligent and adaptive learning tools

1. Approaches to developing more intelligent and adaptive learning tools Adaptive Learning Environments

2. Criticisms of early ITS’s • Mid 1980s: Ohlsson and others criticised ITSs for limited range and adaptability of teaching actions, compared to tactics and strategies used by human expert teachers. • Teachers use variety of communicative activities • e.g. explaining, persuading, arguing, demonstrating, describing… • Narrow focus in ITSs on modelling and diagnosis, not enough on remediation and teaching • Mostly concerned with teaching a procedure, rather than a principle or a concept. • Little support of facilitating and supporting metacognition and providing tools to support planning and reflection Adaptive Learning Environments

3. Progress in various areas.. • Uneven progress - most had restricted repertoire of teaching actions, work concentrated on (ii): • i. Development of a varied repertoire of teaching actions. • ii. Development of effective strategic and tactical means-ends rules for deployment of the teaching actions. • iii. Development of basic, communicative skills and competence as explaining arguing, convincing, cajoling, detecting misunderstandings, dealing with interruptions • iv. Development of theories of motivation and affect that would enable the judicious change of topic, use of a joke, imposition of a threat, offer of praise and so on. Adaptive Learning Environments

4. Later systems attempted to.. • Encouraging explanations from pupils to each other and to the teacher (e.g. Van Lehn et al) • 2. Support group work (led to CSCL): • facilitates metacognitive activities • is itself inherently valuable • 3. Help learners develop better metacognitive skills such as self-explanation and self-evaluation • 4. Involve teachers and learners in the design process • [will look at some of these in later seminars] Adaptive Learning Environments

5. Methodologies for developing more adaptive ALE systems • Aimed to develop more learner centred approach, increase range of teaching tactics and strategies, improve personalisation, make more ecologically valid: • observation of human teachers followed by encoding of effective examples of these teacher’s expertise, typically in the form of rules. • involve teachers and learners in design process • base approach to teaching on learning theory • based on observations of real students or runnable simulation model of student Adaptive Learning Environments

6. Observations of expert human teachers • Early example was Socratic Tutoring (Collins et al., 1975). • provides a number of detailed teaching tactics for eliciting from and then confronting a learner with her misconceptions in some domain. • Generalised to “Inquiry Teaching” (Collins and Stevens,1991). • Lepper et al. (1993) • analysed the methods that human teachers use to maintain students in a positive motivational state with respect to their learning. Adaptive Learning Environments

7. Teacher studies: Graesser • Graesser et al. (2000) studied expert and non-expert teachers and non-expert human tutors: • even untrained human tutors extremely effective • their methods did not correspond to standard methodologies, e.g. Socratic Tutoring (Collins et al., 1975), error identification and correction (Corbett and Anderson, 1992) or sophisticated motivational techniques (Lepper et al., 1993). • Concluded: • “Tutors clearly need to be trained how to use the sophisticated tutoring skills because they do not routinely emerge in naturalistic tutoring with untrained tutors. We believe that the most effective computer tutor will be a hybrid between naturalistic tutorial dialog and ideal pedagogical strategies” (Graesser et al., 2000, page 51) Adaptive Learning Environments

8. Teacher studies: Lajoie et al (2000) • Expert human teachers interweave range of actions that deal with cognitive, metacognitive and affective issues: • observation of an expert medical instructor • noted instructor assigned roles to each medical student when discussing patients • gave both specific and general feedback at same time • Cognitive level: providing scaffolding for the complex, cognitive task of arriving at a diagnosis. • Metacognitive level: the interaction between students and feedback provided models for reflective self-critical examination of how data used and decisions arrived at. • Affective level: encouragement provided when needed. • “Community of practice” level: students apprenticed into medically accepted ways of behaving. Adaptive Learning Environments

9. Derived from learning theory • Learning theories are still being used to inform system design: • e.g Constructivism (Akhras and Self, 2000) and Reciprocal Teaching (Chan and Chou, 1997). • Grandbastien (1999) stresses the need for effective methods to access, organise and use the expertise of the teacher or trainer. • Starting from a model of learning, Winne (1997) suggests how students might be helped to develop better “self-regulated” learning capability (i.e. improve their metacognitive skills). • (see du Boulay and Luckin 2001 for more details) Adaptive Learning Environments

10. Anderson’s learning theory • Development of expertise through declarative knowledge being transformed into procedural knowledge • Transformation supported by ALE (Anderson, 1990). • Attention paid to ensuring learners kept aware of goal and sub-goal hierarchy of the problem solving • The intelligence of these systems in Model Tracing: • representing knowledge of how to do task in terms of production-rules, • keep close track of all student’s actions as problem solved • flag errors as they occur • Adjusts feedback according to the specific problem-solving context in which it is requested. • Knowledge Tracing used to choose the next appropriate problem to move the students through the curriculum. Adaptive Learning Environments

11. Anderson et al ‘85, ‘92, ‘95 • Based on influential (controversial) model of teaching for dealing with errors (also Koedinger et al., 1997, 2000). • one-to-one with fine-grained diagnosis and remediation for multi-step problem solving in domains such as geometry, programming and algebra. • Koedinger et al. (1997) teaching algebra - limitations of tightly constrained solution methods of earlier work circumvented by paying special attention to contextual factors • Involved schools and the teachers using the systems in process and careful thought to how used in classroom • Use of computer tutors by teams of students and with other activities (not just 1-to-1) e.g. presentations to peers Adaptive Learning Environments

12. Derived from studies of real students • Arroyo et al. (2000) categorised students by gender and by level of cognitive development. • How do variations in the style of hints in the context of an arithmetic program interacted with gender and with cognitive development? • Hints varied on two dimensions: • degree of interactivity • nature of the symbolism used. • Looked at reduction in number of mistakes on problem following a hint • Found that “high cognitive ability students do better with highly symbolic hints while low cognitive ability students do worse with highly symbolic hints” • Results enable program to make “macroadaptive” (Shute, 1995) changes to teaching strategy to suit sub-groups of students. Adaptive Learning Environments

13. Derived from studies of simulated students • Typically compare different teaching methods across identical simulated students • build computational model of learner or learning process • derive teaching strategy or constraints on teaching behaviour by observing the model’s response to different teaching actions. • VanLehn et al. (1994) compared two strategies for teaching subtraction to a production rule model of learner: • found “equal additions” strategy more effective (less processing) than widely taught “decomposition” strategy • VanLehn (1987) derived “felicity conditions” for the structure of tutorial examples, e.g. that they should only contain one new subprocedure • Could also model students of differing characteristics and observing how particular teaching method affects them Adaptive Learning Environments

14. Informing and Evaluating Design Adaptive Learning Environments

15. An iterative view of system development(from Waller, 2004) code implementation task analysisfunctional analysis Evaluation prototyping requirementsanalysis conceptual design/ physical design representation From Hix & Hartson (1992) Adaptive Learning Environments

16. Waller (2004) summarises… Identify needs / establish requirements (Re)design Evaluate Build an interactive version Final Product Adaptive Learning Environments

17. Participatory Design: • Will illustrate the various stages with illustrations from: • Conlon, T. & Pain, H. (1996). Persistent Collaboration: A Methodology for Applied AIED. Journal of Artificial Intelligence in Education, Vol 7 No. 3/4 219-252. • And more recent work of Conlon Adaptive Learning Environments

18. What is needed to support learning? • What does the literature suggest is needed? • What are the gaps in the literature? • What has worked and what not? • Methodological flaws? • Untested hypotheses? • What do teachers themselves think they need help with? • hard to know what you want if you do not know what is possible…. • Observational studies • - what works, what does not Adaptive Learning Environments

19. What do teachers want? • Which teachers? • Classroom teachers and University lecturers • Trainers • Head teachers • Curriculum experts • How to find out? • Get them to talk through tasks • Explain and illustrate perceived problems • Interviews: structured/unstructured • Mock ups • Present alternatives • Discuss early ideas • Participatory Design Process Adaptive Learning Environments

20. Case study: Expert System Shells • Knowledge Modelling: learner as 'knowledge engineer’, building, refining and reflecting on knowledge model • Various shells developed for school use (e.g. Primex, Conlon, 1990). • Conlon and Bowman, 1995: • How had these shells impacted upon practice of secondary schools? • - How many schools had bought such software? • - What teaching purposes and learning roles used? • - What affected the software's uptake by teachers? • - How good were pupils at building knowledge bases? • - How skilled were teachers in the use of the software? • Had such shells fulfilled developers' expectations? • What more could be done to improve the prospects of successful knowledge-based modelling? Adaptive Learning Environments

21. Why Expert System Shells? • Can be used to build knowledge models • But commercial ones not suitable for schools • Various rule based shells developed for schools • But used mostly in technology classes - not as modelling tools in the wider curriculum • Led to research by Conlon on why, and how tools might be improved to make them both useful and usable Adaptive Learning Environments

22. Led to INTERMODELLER: • a computer program intended to support children in building classification models; • can be in many domains: a child who is learning about spiders, dinosaurs or planets can build a model of his or her developing knowledge of the domain; • models once constructed can be run as small-scale expert systems that perform interactive classification; • models can be demonstrated and discussed, shared as files across networks, pasted into word processors and graphics programs, printed out to create classroom display material. Adaptive Learning Environments

23. Conlon, 1999 • “.. effective support for constructive thinking in classification can be provided by a model-building environment in which learners create software representations (or models) of classification structures. • The environment provides editors for building these models and an interpreter which can run any model as an interactive classifier.” Adaptive Learning Environments

24. Interchangeable Representations • Classification trees: organise classes or categories hierarchically, with arcs representing subclass relationships • Decision trees: flowchart-like representations which use a branching structure of questions and answers to distinguish between categories • Decision (factor) tables: category named in the right column is defined by a row specifying features as values of attributes named on the header row • Rules: statements of if/then relationships • Graphical tools for Knowledge Base development Adaptive Learning Environments

25. Screenshot of Inter-modeller Adaptive Learning Environments

26. Method: • Conlon and Bowman, 1995: • observed and discussed examples of classroom practice, • questioned a large number of teachers, • gathered and analysed over one hundred knowledge bases constructed by pupils in the 14-18 age group. Adaptive Learning Environments

27. Results: Shells boughtand uptake • How many schools had bought such software? • - Most had at least one expert system shell. • - Shells actively used in IT and Computing classrooms. • - Teachers who used shells were school's IT experts. • - No evidence of use within other subjects. • => knowledge-based modelling failed to develop in the mainstream curriculum. • Factors affecting the software's wider uptake: • - needs for training, • - curriculum support materials, • - improved research skills on the part of learners. Adaptive Learning Environments

28. Purposes, roles and skills in use • Knowledge bases created by pupils were typically: • minor adaptations of the knowledge base files provided as examples by the shell vendor • constructed within one hour of class time. • Pupils who spent 1 to 5 hours with a shell created new knowledge bases that: • contained between three and ten rules, • described informal subjects, • used only most basic shell representation features • IT teachers: learned enough about shells to assist pupils to build knowledge bases of the kind described. • But when presented with more sophisticated knowledge base containing basic errors, few able to identify or correct the errors, • Able to predict correct output for particular test case. Adaptive Learning Environments

29. How decision made: • Interviews with expert KBS technologists suggested 15 types of knowledge-based task (classification, diagnosis, scheduling, etc) from KADS literature. • Curriculum documents analysed to identify the prevalence of such tasks in the curriculum. Adaptive Learning Environments

30. Experts rating of difficulty • 8 KBS experts ranked level of difficulty of task implementation, 1='easiest' to 5='hardest'. Adaptive Learning Environments

31. Eliciting teachers’ views on value and relevance of different tasks • 350 questionnaires to 4 secondary and four primary schools (91 replies) • Asked how often pupils encountered each type of task, five-point scale from 1='never' to 5='very often’ • [Teachers argued ‘tasks’ should be fully explained and illustrated with curriculum-oriented examples: • Researchers felt teacher examples would lead teachers too strongly • Compromise: brief general description of each task with some everyday example of the task's application]. • Analysis of mean responses grouped by stage and subject Adaptive Learning Environments

32. Teachers rating of task frequency Adaptive Learning Environments

33. Task chosen: classification • Classification chosen as the task. • The curriculum documents mentioned it more often than any other task • Schools report that they make use of it in a wide range of contexts • KBS experts identify it as typically amenable to computer implementation Adaptive Learning Environments

34. Formative v. Summative Evaluation • Formative Evaluation: • - throughout design and implementation • - incremental • - assessing impact of changes • - frequently qualitative • Summative Evaluation: • - on completion of each stage • - assessing effectiveness • - frequently quantitative Adaptive Learning Environments

35. Formative Evaluation Methods • Task analysis Observation • Cognitive WalkthroughMock-ups • Protocol analysis Wizard of Oz • Interview (structured/unstructured) • Questionnaire Focus groups • Heuristic EvaluationExpert evaluation • Sensitivity Analysis Self Report • Post-hoc analysis Logging use • Dialogue mark-up and analysis • Manipulation experiment Sentient analysis Adaptive Learning Environments

36. Case Study: Design Principles • Knowledge acquisition techniques for classification selected • General design principles agreed by all parties: • All new tools would be implemented as plug-in extensions to the Primex shell. • Tools should support collaborative model-building, in particular by enabling several smaller models to be combined into one larger one. • Tools should adopt a common set of interface conventions. d) Should be possible to exchange data between tools. • e) All software should run on Macintosh computers of the kind owned by schools. Adaptive Learning Environments

37. Version 1: Pilot testing • Researchers used it for an afternoon with trainee teachers: • twenty-minute demonstration of the software and teachers were then • asked to build knowledge bases on two subjects with which they were familiar: shape classification and river fish identification. • Researchers collected a large number of observations • Observations discussed with technologists who rapidly produced version 2 Adaptive Learning Environments

38. e.g. results of first pilot testing Adaptive Learning Environments

39. Repeated Cycles • Cycle of revision, observation, and discussion was repeated through versions 2, 3, 4 and 5 of the tool. • Each class-tested by the researchers with the same group of B.Ed. Students => more observations and revisions. • Significant feature added to tool: • problem mentioned by the teachers was that pupils' knowledge bases were not easy for teachers to assess. • technologists developed a technique for automatically comparing, and detecting inconsistencies between, two knowledge bases (Conlon 1995). • implemented in version 4 as a new 'Compare' tool. • researchers welcomed the feature, observed it in use, proposed main benefit to stimulate discussion between learners whose models contained different interpretations of the domain. Adaptive Learning Environments

40. Formative Evaluation v 5 • Researchers asked the B.Ed. students to: • assess the change in the quality of the tool relative to version 1; • identify remaining weaknesses in version 5; • say whether they considered version 5 to be ready for classroom use. • Remaining weaknesses identified • Assessed by all as possibly or definitely ready • On the evidence of these observations the researchers initiated school-based trials. Adaptive Learning Environments

41. V5: School based study • 1 teacher, 4-week trial at secondary school • Introduced tool to Graphic Communication and Craft & Design classes • - Explained it was a trial version. • Pupils built models in various domains, including • Selection of Tools, • Choosing a Computer, • Rendering, • Fixing a Bike, and • Buying Football Players • Afterwards some pupils wrote comments on the experience • Researchers were surprised by the sophistication of some of these comments. Adaptive Learning Environments

42. Example pupil comments • 1. Primex allows the user to manipulate the decision tree to the way they want it to. It is able to adopt to a wide number of situations and it is easy to understand and it is extremely user friendly. It allows trees to be created quickly and effortlessly as well as being easy to follow through when completed. • 2. The operation of the tree is very simple and vast improvements have been made over the original interface. I would like to see more improvements: the ability to load expert systems written under the original system; circular questions and the ability to link multiple questions to one decision, or part of another tree; …. the ability to launch saved trees automatically; to be able to hide the tree so as to make it unchangeable, and robust ...... Adaptive Learning Environments

43. Teacher observations • pupils approached their tasks with enthusiasm • pupils seemed to understand program principles • pupils' decision trees rapidly grew large; • large trees sometimes caused the system to fail; • the idea of sub-trees introduced, but not taken up; • 'Compare' tool limited use, more helpful if it could intervene intelligently during tree construction • ...... [Pupils] all thought that this type of work could be utilised in all their classes for an average of about four 40-minute periods. Importantly the work involved in learning how to use the system is minimal compared to the metacognition which undoubtedly must go on. This could be used solely as a consolidating technique (for content revision) as well as for learning new content ...’ Adaptive Learning Environments

44. Summary of Formative Steps • Survey of teachers and experts on what is used, what needed, and what feasible • Implement prototype knowledge modelling tools and carry out formative evaluation. • Evaluate tools in realistic school contexts: • Use classes taught by teachers (not researchers), • Seek comparisons between the productivity of learners using enhanced shell and using standard version. • Build production quality versions of the most successful tools. • Outcome: Intermodeller Adaptive Learning Environments

45. INTERMODELLER: • a computer program intended to support children in building classification models; • can be in many domains: a child who is learning about spiders, dinosaurs or planets can build a model of his or her developing knowledge of the domain; • models once constructed can be run as small-scale expert systems that perform interactive classification; • models can be demonstrated and discussed, shared as files across networks, pasted into word processors and graphics programs, printed out to create classroom display material. Adaptive Learning Environments

46. Interchangeable Representations • Classification trees: organise classes or categories hierarchically, with arcs representing subclass relationships • Decision trees: flowchart-like representations which use a branching structure of questions and answers to distinguish between categories • Decision (factor) tables: category named in the right column is defined by a row specifying features as values of attributes named on the header row • Rules: statements of if/then relationships • Graphical tools for Knowledge Base development Adaptive Learning Environments

47. Adaptive Learning Environments

48. Model building methods • The methodology comprises the following seven steps: • Decide on the purpose of the model • Identify decision factors (factors used to distinguish different categories) • Select a form of representation • Review the design • Start the model • Develop the model • Reflect and evaluate Adaptive Learning Environments

49. Functionality • Learner can switch between representations: • a knowledge base can be switched between representational forms automatically. • In-built machine-learning: • models can be automatically ‘slimmed’ to improve efficiency (uses ACLS induction). • Full expert-system style runtime features: • how and why explanation • certainty handling • consultation review Adaptive Learning Environments

50. Summative Evaluation Intermodeller (Conlon, 1999) • 82 children each aged 15 years undertook a modelling course occupying roughly eight hours of class time. • Analysis of the 632 models that resulted showed: • rule models were almost never as high in quality • (measured by indices of correctness, efficiency, and conciseness) as those built using the alternative factor table, decision tree and classification tree representations. • Questionnaire responses: • - indicated that children least enjoyed working with rules. • That study also reported that: • - children's ability to construct representations of classification improved significantly as a result of the modelling course. Adaptive Learning Environments