ITS Authoring Tools: an Overview of the state of the art

1. AIED July 99, Murray ITS Authoring Tools: an Overview of the state of the art Tom Murray University of Massachusetts & Hampshire College, Amherst, MA www.cs.umass.edu/~tmurray

2. AIED July 99, Murray OR: Cottage industry forms as thousands build intelligent tutoring systems in their basements--NOT YET! OR:ITS construction: How Easy Can It Be? 1. Still iformative stages, though some are comercial or precommercial 2. Overall question is how easy or cost effective can it be to build an ITS1. Still iformative stages, though some are comercial or precommercial 2. Overall question is how easy or cost effective can it be to build an ITS

3. AIED July 99, Murray What is an ITS, such that one can be �authored?� Any CBI system that separates content (what) from strategy (how) Usually makes inferences about �what the student knows� I.E. Contains a �model� of domain, strategy, and/or student --> We�re talking about pretty basic ITSs - This is dif between ITS and CBT - & dynamic generation of sequencing of instr. content - This is dif between ITS and CBT - & dynamic generation of sequencing of instr. content

4. AIED July 99, Murray Purposes of ITS Authoring Tools (Caveat: Authoring shells vs. authoring tools) Cost-effective production of ITSs Decreased skill threshold for authors Insure good quality by content validation or constraining the ITS to a particular model Allow more participation of practicing educators in ITS design and evaluation Provide a test bed for evaluation of alternative strategy or content models ITSs are difficult and expensive to buildITSs are difficult and expensive to build

5. AIED July 99, Murray How many ITS authoring tools have been built? -xamples: over representatin of systems am most familiar with (including the tools developed in our lab at UMass) does not reflect on quality of the tool. -xamples: over representatin of systems am most familiar with (including the tools developed in our lab at UMass) does not reflect on quality of the tool.

6. AIED July 99, Murray How many ITS authoring tools �MORE Many systems listed are only the latest in a lineage of attempts. So there have been many systems. Lets hope we have learned something!Many systems listed are only the latest in a lineage of attempts. So there have been many systems. Lets hope we have learned something!

7. AIED July 99, Murray Overview: Multiple perspectives describing the field What kinds of ITSs have been authored? Authoring the Interface, Domain, Teaching, and Student Models What Authoring/Knowledge Acquisition Methods Have Been Used? How Are Authoring Systems Designed? (Design Tradeoffs & Open Issues) Pragmatics and Use (Are ITS authoring systems �real?�) - dangerous to talk about other people�s systems! Errors and omissions - everything I say about a given system or category of systems is partly wrong or a simplification - alomost unapoligeticlly mention that I will use a dispropotional number of figures from the Eon authoringsystem developed at UMass. - bag of tricks vs. a shelf of tools - more an overview and look to the future than a critique - difficulty in coming with categories. - see my IJAIED paper for more details (**give URL ***) - you might be surprised, you might be dissapointed!- dangerous to talk about other people�s systems! Errors and omissions - everything I say about a given system or category of systems is partly wrong or a simplification - alomost unapoligeticlly mention that I will use a dispropotional number of figures from the Eon authoringsystem developed at UMass. - bag of tricks vs. a shelf of tools - more an overview and look to the future than a critique - difficulty in coming with categories. - see my IJAIED paper for more details (**give URL ***) - you might be surprised, you might be dissapointed!

8. AIED July 99, Murray What kinds of ITSs have been authored? Both pedagogy-oriented and performance-oriented ITSs Seven Types of ITSs Tools constrain ITSs Defiing characteristic: separates content from strategy (skip this slide?? But not title?).Defiing characteristic: separates content from strategy (skip this slide?? But not title?).

9. AIED July 99, Murray Seven Categories of Authored ITSs Strengths, Limits, Variations, student perspective Categories 3, 4, & 6 are mostly �performance-oriented� - no system does everything; - no system does everything;

10. AIED July 99, Murray 1. Curriculum Sequencing and Planning Systems: DOCENT, IDE, ISD Expert, Expert CML Basic and early historical systems Separates content from presentation and sequencing Rules, constraints, or strategies for �intelligently� sequencing content--at the �macro level� (topic level) Usually low fidelity interfaces, canned content, simple student models - rest of categories build open these basics Student perspeive is like CAI- rest of categories build open these basics Student perspeive is like CAI

11. AIED July 99, Murray 2. Tutoring Strategies Systems: REDEEM, Eon, GTE, Smart Trainer AT #1 above PLUS: Micro-level and explicit tutoring strategies Instructional primitives for hints, explanations, examples. reviews, feedback� Instruction can have a more dialogue or conversational feel Some include multiple teaching strategies and meta-strategies Often have low fidelity interfaces, canned content, simple student models

12. AIED July 99, Murray Tutoring strategies category: Example - agin student persopective like CAI- agin student persopective like CAI

13. AIED July 99, Murray 3. Device Simulation and Equipment Training Systems: DIAG, RIDES, MITT-Writer, ICAT, SIMQUEST, XAIDA Micro-world/simulation-based learning environments Most focus on equipment/device operation and maintenance procedures Building the simulation is time consuming, but much of the �tutoring� then comes for free. - operation and component identification are farily generic and ubiquitous tasks. - Perf. Montoiring and feedback tend to be straightforward: that is not the wing pressure cutoff valve; you should have checked the temperature first. - diagnosis is harder and authoring not as far along- operation and component identification are farily generic and ubiquitous tasks. - Perf. Montoiring and feedback tend to be straightforward: that is not the wing pressure cutoff valve; you should have checked the temperature first. - diagnosis is harder and authoring not as far along

14. AIED July 99, Murray Examples from RIDES Tutors

15. AIED July 99, Murray 4. Domain Expert System Systems: Demonstr8, D3 Trainer, Training Express Deep/runnable models of problem solving expertise Fine grained student diagnosis and modeling Building an expert system is very difficult -- but then instruction can come �for free�

16. AIED July 99, Murray D3s Medical Tutor

17. AIED July 99, Murray Demonstr8�s Subtraction Tutor

18. AIED July 99, Murray 5. Multiple Knowledge Types Systems: CREAM-Tools, DNA, ID-Expert, IRIS, XAIDA �Gagne Hypothesis:� There are different types of knowledge --> Each has its own instructional methods and representational formalism Template-like framework for decomposing content into facts, concepts, and procedures Many based on instructional design theory principles Limited so far to relatively simple facts, concepts, procedures - in the �device simulation� categories we see how having a consistent underlyingrepresentation can lead to instructin for free. - Systems in this category capitalkize on this fact even more. - examples: facts are taught with mneumonic devices and drill and practice; concepts are taught with examples and anslogies; proecedrues are taught one step at a time. - for student these are similar to Multiple teachign strategies category: maindifference is for he author,sincle authoring is both easier and leff flexible. - mention �transaction shells� in Merril�s Transaction Theory - in the �device simulation� categories we see how having a consistent underlyingrepresentation can lead to instructin for free. - Systems in this category capitalkize on this fact even more. - examples: facts are taught with mneumonic devices and drill and practice; concepts are taught with examples and anslogies; proecedrues are taught one step at a time. - for student these are similar to Multiple teachign strategies category: maindifference is for he author,sincle authoring is both easier and leff flexible. - mention �transaction shells� in Merril�s Transaction Theory

19. AIED July 99, Murray 6. Special Purpose Systems: IDLE-Tool/IMap, LEAP Authoring Tool Build tutors for a particular type of task Can provide strong authoring guidance and constraints Design and pedagogical principles can be enforced The task, interface, and pedagogy must fit relatively inflexibly to the given model - Takes the ideas just discussed a step further: rather than having templates for knowledge tpyes, create them for specific types of tasks. - more of a production line model - examples: - I�ll say more about special pruose vs generla purpose systm later template for nvestigaand decide Les LAThas conversational grammars ; for to train telephone customer service people - Takes the ideas just discussed a step further: rather than having templates for knowledge tpyes, create them for specific types of tasks. - more of a production line model - examples: - I�ll say more about special pruose vs generla purpose systm later template for nvestigaand decide Les LAThas conversational grammars ; for to train telephone customer service people

20. AIED July 99, Murray Example: IDLE-Tool:Sickle Cell Counselor --

21. AIED July 99, Murray 7. Intelligent/Adaptive Hypermedia Systems: CALAT, GETMAS, InterBook, MetaLinks Similar to Category #1 but also deals with Navigation and (dis)orientation issues Accessibility and UI uniformity benefits associated with the WWW Limited interactivity and learning environment fidelity Potential for making inferences from large numbers of students - sequencing PLUS navigation isses - sequencing PLUS navigation isses

22. AIED July 99, Murray Example: InterBook

23. AIED July 99, Murray Authoring the Interface, Domain, Teaching, and Student Models Interface Domain Model Curriculum knowledge structures Simulations of Devices and Phenomena Expert Systems Teaching Model Student Model

24. AIED July 99, Murray 1. Authoring the Interface Systems with interface authoring tools: RIDES (below), Eon, SIMQUEST - most ITS authr tools don�t allow the author to build the interface design highly interactive screens from scratch - most mune because its what COTS authoring systems do well - but important.. - Items on screen are objects; animation and other visual effexcts are achievd by attaching properties of the object, such as its location, bitmap, or color, to dynamic simulation parameters.- most ITS authr tools don�t allow the author to build the interface design highly interactive screens from scratch - most mune because its what COTS authoring systems do well - but important.. - Items on screen are objects; animation and other visual effexcts are achievd by attaching properties of the object, such as its location, bitmap, or color, to dynamic simulation parameters.

25. AIED July 99, Murray Example 2

26. AIED July 99, Murray 2. Authoring the Domain model Curriculum knowledge and structures Simulations/models of devices and phenomena Domain Expertise models (expert system)

27. AIED July 99, Murray Authoring Curriculum Knowledge and Structures Topics/KUs Relationships (e.g. prerequisite) Knowl. Type (concept, procedure�) Objectives Importance Difficulty Visual and textual (next slide) methodVisual and textual (next slide) method

28. AIED July 99, Murray Example: IRIS Note: multi knowledge types; objective type; difculty and abstraction level - separating the learning objectives from the knowledge - Merril�s Content types and - Uses Gagnes taxonomy of learning objectives Note: multi knowledge types; objective type; difculty and abstraction level - separating the learning objectives from the knowledge - Merril�s Content types and - Uses Gagnes taxonomy of learning objectives

29. AIED July 99, Murray Example: CREAM Tools - A connection has to be made between the abstract world of topics and concepts to the concrete world of the screens and digital resrouces that illustrate or (make them contrete) - separating the domain relationships (part-of) from the learning pedagogical relationshis (prerequisite); - Capabilities C, Objectives ), resources R, pedagogy P - Resorce model; Ax abstraction, CP particular case; An analogy; - Resouce types inmiddle; resource linking tool on right - A connection has to be made between the abstract world of topics and concepts to the concrete world of the screens and digital resrouces that illustrate or (make them contrete) - separating the domain relationships (part-of) from the learning pedagogical relationshis (prerequisite); - Capabilities C, Objectives ), resources R, pedagogy P - Resorce model; Ax abstraction, CP particular case; An analogy; - Resouce types inmiddle; resource linking tool on right

30. AIED July 99, Murray Authoring Simulations of Devices and Phenomena - graphic objects (e.g. pump) have alternate graphics representations. The representaiont (open or closed valve) is set according to an author defined ate variable. The author then defines rules like if linePressure > 3000 then control vallve position - open. - Theoryof operations in XAIDA--aircraft maintenance domain- graphic objects (e.g. pump) have alternate graphics representations. The representaiont (open or closed valve) is set according to an author defined ate variable. The author then defines rules like if linePressure > 3000 then control vallve position - open. - Theoryof operations in XAIDA--aircraft maintenance domain

31. AIED July 99, Murray Example 2 RIDES - - constraint based and event based e simulain vent processing -more complicted simulaitns than XAIDA,but harder to author- - constraint based and event based e simulain vent processing -more complicted simulaitns than XAIDA,but harder to author

32. AIED July 99, Murray Authoring Domain Expertise (Expert systems): D3 Trainer

33. AIED July 99, Murray 3. Authoring the Teaching Model Example: REDEEM - most Atools odn�t allow for authoring the teaching strategy� -- this is a structured interview tecnique- most Atools odn�t allow for authoring the teaching strategy� -- this is a structured interview tecnique

34. AIED July 99, Murray Authoring the Teaching Model Example 2: Eon - like authorware or icaon authors - more powerful andlfexible bt less usable- like authorware or icaon authors - more powerful andlfexible bt less usable

35. AIED July 99, Murray 4. Authoring the Student Model - only system with customizable student model- only system with customizable student model

36. AIED July 99, Murray What Authoring/Knowledge Acquisition Methods Have Been Used? 1. Scaffolding knowledge articulation with models 2. Embedded knowledge and default knowledge 3. Knowledge management 4. Knowledge visualization 5. Knowledge elicitation and work flow management 6. Knowledge and design validation 7. Knowledge re-use 8. Automated knowledge creation Authoring is: design, knowledge acquisition, and fabrication, Authoring is: design, knowledge acquisition, and fabrication,

37. AIED July 99, Murray 1. Scaffolding knowledge articulation with models Ex. 1: Templates: IDLE-Tools Ex. 2: Ontology-Aware tools: SmartTrainer AT

38. AIED July 99, Murray

39. AIED July 99, Murray 3. Knowledge management Topics/KUs Lesson Objectives Interface objects & screens Exercises, examples, pictures Teaching Strategy actions

40. AIED July 99, Murray 4. Knowledge visualization LEAP-AT - many of the tools involve helping authors visualize. - helping authors visualize conversational grammars- many of the tools involve helping authors visualize. - helping authors visualize conversational grammars

41. AIED July 99, Murray 5. Knowledge elicitation and work flow management Author: �What do I do next?� � Where do I start?� Prompts in ID-Expert and DNA: �Which of the following describes what the student will learn: a. What is is? B. How to do it? C. How does it work?� Top down vs opportunistic design DNA: Semi-structured interactive dialog has prompts with choices REDEEM: Agenda mechanism for authoring tasks - interactive knowlsdge elicitation - stepping the author through the design process by asking questions - previously we had VISUAL and Strucural organization; this is PROCEDURAL scaffolding; next slide is CONSTEAIN-BASED- interactive knowlsdge elicitation - stepping the author through the design process by asking questions - previously we had VISUAL and Strucural organization; this is PROCEDURAL scaffolding; next slide is CONSTEAIN-BASED

42. AIED July 99, Murray 6. Knowledge and design validation Opportunistic & Open ended --> more flexibility & more errors Constraint-based advice: �The estimated time for all Lesson-2 topics exceeeds the estimated time for Lesson-2� �The engine maintenance procedure has no sub-steps defined� �Lesson-3 objectives include procedural and conceptual knowledge, but there are no conceptual topics linked to Lesson-3.�

43. AIED July 99, Murray 7. Knowledge re-use Libraries of Content, Graphics, Strategies, etc. Flexible reconfiguration of components

44. AIED July 99, Murray 8. Automated knowledge creation Example-Based programming Inferring a general procedure/rule from an example procedure/rule

45. AIED July 99, Murray

46. AIED July 99, Murray Suggestions for a Full-Featured Authoring System Visual reification of conceptual and structural elements Assistance: design steps or agenda; constraint-based validation Content reusability and object libraries Scriptable and customizable WYSIWIG editing, Opportunistic design, Easy design/test iteration (interpreted vs compiled), Reasonable default values for rapid prototyping We�ll revisit scriptable and customizable later \SKIP THIS?We�ll revisit scriptable and customizable later \SKIP THIS?

47. AIED July 99, Murray How Are Authoring Systems Designed?Design Tradeoffs & Open Issues The space of design tradeoffs General vs. special purpose authoring systems Who are the authors? Who should author ITS instructional strategies? Meta-Level Authoring - so far descibed a wide variety of features and teqniques. - to AT has all of them. - each has strenghts a weaknesses - sometimes due to amount of effort or attention payed to certain components, but aslo due to philosophical differences.- so far descibed a wide variety of features and teqniques. - to AT has all of them. - each has strenghts a weaknesses - sometimes due to amount of effort or attention payed to certain components, but aslo due to philosophical differences.

48. AIED July 99, Murray The Space of Design Tradeoffs - strong in one area tends to be weak in some other - strong in one area tends to be weak in some other

49. AIED July 99, Murray General vs. special purpose authoring systems E.G. special purpose systems: LAT and IDLE-Tool Greater usability, fidelity, depth -- but only for design goals that match the tools. Does the �demand� balance the inflexibility? How to make more customizable while maintaining ease of use? Types of abstraction/specialization? 1. Real-world tasks 2. Abstract tasks 3. Knowledge types - ther are a number of hot issues; areas of freiendly controversey - ( add: )Worse breadth, flexibility - like chandrasakaren�s abstract task types - ther are a number of hot issues; areas of freiendly controversey - ( add: )Worse breadth, flexibility - like chandrasakaren�s abstract task types

50. AIED July 99, Murray Abstracting ITSs for special purpose authoring systems 1. Abstracting real-world tasks: Investigate & Decide; Evidence-Based reporting; Run an Organization 2. �Abstract tasks:�  Equipment operation & maintenance (RIDES); Conversational Grammars (customer service; LAT) 3. Knowledge types: Facts, concepts, procedures, principles (CREAM-Tools, DNA, ID-Expert, XAIDA) - ther are a number of hot issues; areas of freiendly controversey - ( add: )Worse breadth, flexibility - ther are a number of hot issues; areas of freiendly controversey - ( add: )Worse breadth, flexibility

51. AIED July 99, Murray Who are the authors? What level of skill & training should be expected? Authoring skill sets: instructional design, classroom pragmatics, graphics/UI, domain knowledge, knowledge engineering, script-level programming... IDLE, XAIDA, REDEEM: try to allow authoring by teachers and �off the street� domain experts with minimal training - some think we can built systems an order of magnitude more powerful than traditional CAI with an order of magnitude less effort. I am hopeful yet skeptical. Seems only possible for template-based systems. - should be able to xpect authors to have some degree of training, sophisitcation, and dedication- some think we can built systems an order of magnitude more powerful than traditional CAI with an order of magnitude less effort. I am hopeful yet skeptical. Seems only possible for template-based systems. - should be able to xpect authors to have some degree of training, sophisitcation, and dedication

52. AIED July 99, Murray Suggested authoring scenario Effort level: Building an ITS is more like writing a book than creating a greeting card! Skill level: Skill level equivalent: Accounting applications, CAD, spreadsheet macros, 3-D modeling, advanced Photoshop�-- Special training but reasonable Sophistication level: Authors need to look at the big picture and do ongoing quality assessment of what they build ITSs are built by design teams, not individuals (distributed skill sets)

53. AIED July 99, Murray Who should specify/author ITS instructional strategies? - who is going to tell these machines how to teach? (other suggestions: students, market analysis, legislative committees�) SKIP?- who is going to tell these machines how to teach? (other suggestions: students, market analysis, legislative committees�) SKIP?

54. AIED July 99, Murray Meta-Level Authoring Custom/extensible interface widgets Customizable descriptive vocabulary Pre-configured tutoring strategies and student models - leave themost idfficult authoringto the experts. - one way to deal with the tradeoffs between usabilty and power - use a general purpose AT to build special purpose ones - leave themost idfficult authoringto the experts. - one way to deal with the tradeoffs between usabilty and power - use a general purpose AT to build special purpose ones

55. AIED July 99, Murray Use & Pragmatics (Are ITS authoring systems �real?�) Authoring system Use Authoring system Productivity Authoring system Evaluation

56. AIED July 99, Murray Authoring Tool Use Examples: � XAIDA domains: equipment operation and maintenance, algebra, medicine, computer literacy, biology � IDLE-Tool: three informal trials with 21, 8, 8 grad student and grade school teacher authors - category 1: toy domains - category 4: next slide- category 1: toy domains - category 4: next slide

57. AIED July 99, Murray (Relatively) Heavily Used Authoring Tools Build a dozen or more ITSs Many ITSs used in real educational settings Robust enough for use independent of original design team RIDES: many project spin-offs and diverse domains CALAT: over 300 Web-based courses (used at NTT) Rovust: usu. included user documentatin, some level of suport etc. - not sure how mnay were built with SIMQUEST or Trainins express (and lwhether they were used)Rovust: usu. included user documentatin, some level of suport etc. - not sure how mnay were built with SIMQUEST or Trainins express (and lwhether they were used)

58. AIED July 99, Murray Authoring Tool Productivity For traditional CAI: Estimated 300:1 ratio of development to instruction time ITS authoring: goals and some spotty evidence ID-Expert�s goal: 30:1 XAIDA�s goal 10:1; evidence of a first time user at 16:1 KAFITS Physics tutor w/ six hours of instruction: 100:1 CALAT: ITS development in about the same time as traditional instruction REDEEM: 2:1 to segment CAI content & make intelligent Implication: AI Knowledge Representation does provide ITSs with inherent efficiencies - Xaida figure: including trianing time - Demonstr8 informal study: moel tracing multi-column addition or subtractin tuutor built in < 20 minutes. - Summary: inconclusive estimates but all indications are that non-model tracing tutors can be authored in same time as traditioanl multimedia instruction - since building an ITS presumably takes additional effort for knowledge engineering (to acquire a model of the domain and/or teaching strategies) over and above traditional CAI, we must assume that there are real efficiencies involved in the modular and reusable nature the knowledge in ITSs. & generation of content on the fly - - Xaida figure: including trianing time - Demonstr8 informal study: moel tracing multi-column addition or subtractin tuutor built in < 20 minutes. - Summary: inconclusive estimates but all indications are that non-model tracing tutors can be authored in same time as traditioanl multimedia instruction - since building an ITS presumably takes additional effort for knowledge engineering (to acquire a model of the domain and/or teaching strategies) over and above traditional CAI, we must assume that there are real efficiencies involved in the modular and reusable nature the knowledge in ITSs. & generation of content on the fly -

59. AIED July 99, Murray Authoring Tool Evaluations Existence proofs: Usability; Productivity; Breadth Examples: XAIDA REDEEM IDLE-Tools, COCA, LAT, KAFITS, DNA - �Existence proofs,� (see authoring tool USE above ) as alternative to evaluation - how do you evaluate something like an authoring tool? - no clear methods or metrics, but �- �Existence proofs,� (see authoring tool USE above ) as alternative to evaluation - how do you evaluate something like an authoring tool? - no clear methods or metrics, but �

60. AIED July 99, Murray Evaluations of XAIDA Eight authoring field studies with average of 10 instructor participants each 13 studies of students using the built tutors Data: Learnability: abilities assessment, self-report skills, cognitive assessment, task-based performance Acceptability: open-ended questionnaire Productivity: use analysis Usability: questionnaire - mostly the physical characteristcs part of the XAIDA shell Brenda wenzel Henry Half and colleagues- mostly the physical characteristcs part of the XAIDA shell Brenda wenzel Henry Half and colleagues

61. AIED July 99, Murray XAIDA Evaluation

62. AIED July 99, Murray Proficiency Using XAIDA

63. AIED July 99, Murray REDEEM Evaluation: 1 SME author, 3 teacher authors, 7 �virtual students� Data: authoring sub-task time, variations among authors, appreciation of added �intelligence� Time spent by teacher practitioner:

64. AIED July 99, Murray Some Formative Evaluation Results (And see Productivity and Use above): Authors� cognitive model of the domain had a structure closer to SME�s after tool use (XAIDA) Considerable difference between authors in content structure, strategy specification, categorization of students (REDEEM) Teacher reactions in general positive but difficulty with complex relationships among content pieces Teachers thought AI technology could simulate reasonable teaching strategies(COCOA) Tools needed to give users abstract view of the content (IDLE-Tools) -- to give an idea of the types of results researchers are getting. The individual results are suggestive only, as most have not been reproduced. -- to give an idea of the types of results researchers are getting. The individual results are suggestive only, as most have not been reproduced.

65. AIED July 99, Murray Some Formative Evaluation Results (cont.) Including examples for design steps/information was very helpful (IDLE-Tools) Graphic representations for knowledge elicitation much less error-prone than text-based (LAT) Overestimated of the level of expertise authors would gain in a short amount of time (LAT) Authors have difficulty conceptualizing non-linear, modular content (KAFTIS) Comparing automated knowledge elicitation to coded-by-hand task analysis: automated method covered most of the domain knowledge in a small fraction of the time (DNA)

66. AIED July 99, Murray Summary Many types of ITSs have been �authored� Wide variety of knowledge acquisition and authoring methods have been used. Too early to know when each is most appropriate. Some tools have significant use and a few are in commercial or near-commercial form Promising results in from evaluations of usability and productivity, with more rigorous evaluations just starting What are the foreseeable limits?

67. AIED July 99, Murray Conclusions--How Easy Can It Be? There are limits! Limited use of cookie-cutter special purpose authoring tools-- too restrictive for most authors Limited ability to reduce ITS authoring to easy, small, independent steps (recipes) Authors need to think about the big picture and need skills and tools to do this

68. AIED July 99, Murray ...Back to the Future Customizability requirements will usually lead to the author specifying BEHAVIORS (choices, rules, algorithms) as well as static information This requires ability to RUN, test, and modify these behaviors This is (simple) PROGRAMMING Debugging skills and tools will be needed! (Tracing, stepping, inspecting states, etc.)

69. AIED July 99, Murray ------------------------------------

70. AIED July 99, Murray ITS Authoring Tools: an Overview of the state of the art Tom Murray University of Massachusetts & Hampshire College, Amherst, MA www.cs.umass.edu/~tmurray

71. AIED July 99, Murray People #1

72. AIED July 99, Murray People #2

73. AIED July 99, Murray