Cognitive Tutor Development Kit TDK Tutorial

1. Cognitive Tutor Development Kit (TDK) Tutorial Cognitive Modeling & Intelligent Tutoring Systems Ken Koedinger Vincent Aleven

2. Overview Basics of TDK Notation Matching: $ and = variables Modeling alternative strategies �Interface constraints� for model tracing Control flow and production chains Writing hints and bug productions Creating a curriculum

3. Production Rules in the Tutor Development Kit (TDK) How do you go about writing a production rule? The TDK Production System Working Memory: Make up of �Working Memory Elements� (WMEs) Production Rules Interpreter (Match, Conflict Resolution, Fire)

4. Implementing a Production Rule Model in TDK Simple example: a model for single-column addition without carrying! How would you define: Working Memory representation for the problem states Production rules that transform working memory Anderson and Pelletier paper on the tutor development kit says that you develop the user interface first. In this course, most people will probably not get to the point that they will implement a user interface. However, it is still very important, when designing a production rule model, that one knows what the interface will be. (Perhaps this point should be made later.) Anderson and Pelletier paper on the tutor development kit says that you develop the user interface first. In this course, most people will probably not get to the point that they will implement a user interface. However, it is still very important, when designing a production rule model, that one knows what the interface will be. (Perhaps this point should be made later.)

5. A Notation for Working Memory Elements (�WMEs�) in TDK PROBLEM4+3> ISA SINGLE-COLUMN-ADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT nil DONE nil

6. Working Memory Transitions PROBLEM4+3> ISA SINGLE-COLUMN- ADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT nil DONE nil Nothing complicated. T means TRUE in Lisp NIL often means FALSE In developing a more complex model, you would probably not typically write out all transitions beforehand. But nonetheless you need to have a mental picture of how you would represent problem states.Nothing complicated. T means TRUE in Lisp NIL often means FALSE In developing a more complex model, you would probably not typically write out all transitions beforehand. But nonetheless you need to have a mental picture of how you would represent problem states.

7. TDK Production Rule Notation (defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2 ==> !eval! =sum (+ =num1 =num2) =problem> result =sum ) ADD (English version) IF The goal is to do =problem, a single-column addition problem And no result has been found yet And the first addend is =num1 And the second added is =num2 THEN Set =sum to the sum of =num1 and =num2 Write =sum as the result

8. TDK Production Rule Notation

9. TDK Production Rule Notation In Anderson�s addition model in Rules of the Mind, the sum of two digits is not computed but rather retrieved from working memory by means of productions. Simple rule, so only one WME pattern in the condition part and action part.In Anderson�s addition model in Rules of the Mind, the sum of two digits is not computed but rather retrieved from working memory by means of productions. Simple rule, so only one WME pattern in the condition part and action part.

10. TDK Production Rule Notation Variables - will be bound to whateverVariables - will be bound to whatever

11. Matching a Production Rule to Working Memory�Find Values for Each Variable

12. Summary�Components of a TDK Production Rule Model Working Memory is a collection of WMEs wme-name> isa wme-type slot1 value1 slot2 value2 Production rules specify working memory transitions WME patterns in the IF-part are matched against Working Memory =wme-variable> isa wme-type slot1 value-pattern1 slot2 value-pattern2 WME patterns in the THEN-part specify changes to Working Memory

13. Overview Basics of TDK Notation Matching: $ and = variables Modeling alternative strategies �Interface constraints� for model tracing Control flow and production chains Writing hints and bug productions Creating a curriculum

14. WME Representation Initial State PROBLEM26+51> ISA MULTI-COLUMN-ADDITION-PROBLEM COLUMNS (COLUMN2 COLUMN1) DONE nil COLUMN1> ISA COLUMN FIRST-ADDEND 6 SECOND-ADDEND 1 RESULT nil COLUMN2> ISA COLUMN FIRST-ADDEND 2 SECOND-ADDEND 5 RESULT nil Each transition in the state space has to be mirrored/represented by a change (or changes) in working memory.Each transition in the state space has to be mirrored/represented by a change (or changes) in working memory.

15. Production Rules Set New Goals & Perform Actions Setting goals happens in the head. Actions are external actions. Observable actions. Or they are internal. Production rules not only set new goals, they also respond to goals. They may also modify existing goals.Setting goals happens in the head. Actions are external actions. Observable actions. Or they are internal. Production rules not only set new goals, they also respond to goals. They may also modify existing goals.

16. Goals in Production Rules FOCUS-ON-FIRST-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column of the problem And there are numbers to add in C And there is no result yet in C THEN Set a subgoal to process column C FOCUS-ON-NEXT-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column with numbers to add and no result THEN Set a subgoal to process column C WRITE-RESULT IF There is a goal to process column C THEN Set Sum to the sum of the addends in column C Write Sum as the result in column C DONE IF The goal is to do a multi-column addition problem And there is no column left with numbers to add and no result THEN Mark the problem as done Subgoal story. Breaks down theproblem in a way that seems to correspond to the skil decomposition. Are these good subskills? Seems so. If you have learned to add numbers in the rightmost columns, it would seem that you also know how to add numbers in the For example, much better than if you had written a rule that completely takes care of �Hand simulation� ? How to do this quickly?Subgoal story. Breaks down theproblem in a way that seems to correspond to the skil decomposition. Are these good subskills? Seems so. If you have learned to add numbers in the rightmost columns, it would seem that you also know how to add numbers in the For example, much better than if you had written a rule that completely takes care of �Hand simulation� ? How to do this quickly?

17. Figuring Out Which Column FOCUS-ON-FIRST-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column of the problem And there are numbers to add in C And there is no result yet in C THEN Set a subgoal to process column C FOCUS-ON-NEXT-COLUMN IF The goal is to do a multi-column addition problem And C is the rightmost column with numbers to add and no result THEN Set a subgoal to process column C WRITE-RESULT IF There is a goal to process column C THEN Set Sum to the sum of the addends in column C Write Sum as the result in column C DONE IF The goal is to do a multi-column addition problem And there is no column left with numbers to add and no result THEN Mark the problem as done Subgoal story. Breaks down theproblem in a way that seems to correspond to the skil decomposition. Are these good subskills? Seems so. If you have learned to add numbers in the rightmost columns, it would seem that you also know how to add numbers in the For example, much better than if you had written a rule that completely takes care of �Hand simulation� ? How to do this quickly?Subgoal story. Breaks down theproblem in a way that seems to correspond to the skil decomposition. Are these good subskills? Seems so. If you have learned to add numbers in the rightmost columns, it would seem that you also know how to add numbers in the For example, much better than if you had written a rule that completely takes care of �Hand simulation� ? How to do this quickly?

18. Example Production Rule

19. TDK Production Rule Notation

20. TDK Production Rule Notation At the first place where it occurs, the variable will be bound during the matching process. At latter places, the value of a bound variable in a pattern must be matched in the WME that the pattern is matched against.At the first place where it occurs, the variable will be bound during the matching process. At latter places, the value of a bound variable in a pattern must be matched in the WME that the pattern is matched against.

21. TDK Production Rule Notation At the first place where it occurs, the variable will be bound during the matching process. At latter places, the value of a bound variable in a pattern must be matched in the WME that the pattern is matched against.At the first place where it occurs, the variable will be bound during the matching process. At latter places, the value of a bound variable in a pattern must be matched in the WME that the pattern is matched against.

22. Matching a Production Rule to Working Memory�Find Values for Each Variable (1)

23. Value to be matched: (COLUMN4 COLUMN3 COLUMN2 COLUMN1) Pattern: ($ =next-column =previous-column $) Match 1 Left $ () =next-column COLUMN4 =previous-column COLUMN3 Right $ (COLUMN2 COLUMN1) Match 2 Left $ (COLUMN4) =next-column COLUMN3 =previous-column COLUMN2 Right $ (COLUMN1) Match 3 Left $ (COLUMN4 COLUMN3) =next-column COLUMN2 =previous-column COLUMN1 Right $ () Pattern Matching with Lists�Patterns with$ variable(s) often create multiple matches

24. TDK Production Rule Notation �==>� separates if-part & then-part If- & then-parts are made up of WME patterns of form: =wme-variable> isa wme-type slot1 value-pattern1 slot2 value-pattern2 An atom starting with either �=� or �$� is a variable Ex: �=num� �$columns� �=� vars match atoms �$� variables match 0 or more elements of a list �=� or �$� by itself is just an unnamed (dummy) variable

25. The TDK Production System Interpreter Match phase: Attempt to match each WME pattern with 1 or more WMEs in working memory Variables are �bound� to a value. Subsequent references to those variables must have the same value. A set of �bindings� for all variables in the if-part is called a �instance� of a production. If a consistent set of bindings is found, the production �matches�, otherwise it �fails�. A production can match working memory in more than one way, that is, it can have multiple instances (or rule instantiations). Conflict resolution phase: In stand-alone mode: Determined by hand or arbitrarily In tutor mode: Production priorities & model tracing heuristics Act phase (fire a production): Use the variable bindings to modify or create the WME indicated in the then-part. If �isa� slot is present, create a WME, otherwise modify.


27. Modeling Alternative Strategies Consider multiple solution paths Reordering of steps Write sum first or write carry in next column first Alternative strategies Left to right strategy Use calculator and write result strategy Shortcuts Implicit carrying Which of these will the model cover? Motivate choices pedagogically! Point out that this is a general issue Perhaps the alternative strategies that are listed are not entirely serious, but in other domains they are. Our choices and motivation are: Point is, this influences the design of the cognitive modelPoint out that this is a general issue Perhaps the alternative strategies that are listed are not entirely serious, but in other domains they are. Our choices and motivation are: Point is, this influences the design of the cognitive model

28. Execution Space Diagram Illustrating Flexibility


30. Model Tracing 1 � Using a Cognitive Model to Evaluate Students� Solution Steps Cognitive model is an expert system that is psychologically plausible accounts for all reasonable solutions accounts for commonly occurring errors (�bugs�) To verify solution step by student Use model to generate correct and buggy next steps If student step matches Correct step: Accept update Working Memory to reflect new problem state Buggy step: Give error-specific feedback No step: Reject without feedback


32. Communicating About Student Solution Steps between Interface and Tutor Represent each observable action in the user interface as a selection-action-input triple. Typically, each interface element would be identified uniquely by the selections.Typically, each interface element would be identified uniquely by the selections.

33. Execution Space Diagram with Selection-Action-Input Triples

34. Interface Constraints�Matching Student Steps Against Model Steps Production rule writer can attach �model-tracing� or �interface constraints� to rules. Specify tests on the selection-action-input triples. During model-tracing TDK will not consider productions whose constraints are violated.

35. Example of Interface Constraints in English WRITE-SUM IF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been added THEN Write Sum as the result in column C And remove the goal

36. Example of Interface Constraints in English WRITE-SUM IF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been added THEN Write Sum as the result in column C And remove the goal

37. More Detail on Interface Constraints In Tutor Mode When evaluating student input TDK will not consider productions whose constraints are violated. When responding to a help request Tries to find the production path with the least severe constraint violation. from least to most severe: input, action, selection. In Stand-Alone Mode, interface constraints are ignored Convenient for development�can run model without having to supply input.

38. TDK Notation for Interface Constraints (defproduction write-sum addition (=problem) =problem> isa addition-problem . . . =subgoal> isa write-sum-goal . . . ==> =column> result =sum =problem> subgoals ($sg1 $sg2) ; the remaining subgoals :nth-selection 0 =column :action 'write-result :input =sum #'equal-value-p)

39. Syntax of TDK Notation for Interface Constraints :nth-selection n value [test-fn] :action value [test-fn] :input value [test-fn]


41. Creating a Control Flow Diagram Control Flow Diagram elements: Boxes are goals or actions Arrows are production rules A �path� or �chain� through the diagram starts at a goal and ends with an action Crucial idea: Each chain should produce exactly one action When you �cycle� the TDK production system a �chain� of productions fire


43. Chaining of Production Rules�Multiple Productions per Step

44. Important Design Principle Design your production rule model so that there is exactly one cycle per observable (student) action. Even though we are running the model in stand-alone mode, all the time we have to be thinking about how it will be used for tutoring.Even though we are running the model in stand-alone mode, all the time we have to be thinking about how it will be used for tutoring.


46. Example Hint Sequences for Addition Tutor Start with the column on the right. This is the "ones" column. You need to add the two digits in this column. Adding 4 and 6 gives 10. The sum that you have, 10, is greater than 9. You need to 10 of the 10 into the second column. And you need to write the rest of the 10 at the bottom of the first column. Write 0 at the bottom of the rightmost column. You need to complete the work on the first column. Write the carry from the first to the next column. Write 1 at the top of the second column from the right. Now move on to the second column from the right. This is the "tens" column. You need to add the two digits in this column. Adding 6 and 1 gives 7. There is a carry into this column, so you need to add the value carried in. This gives 7 + 1 equals 8. Write 8 at the bottom of the second column from the right

47. Design of Hint Messages What makes a good hint sequence? Short Clear (minimum of jargon, short sentences) Tells student what to do But also why/when that action is appropriate Goes from the general to the specific Writing hints can help in designing the production rule model. Hint sequences often combine hints from multiple productions in a chain.

48. (defproduction focus-on-next-column addition (=problem) =problem> isa addition-problem . . . =previous-column> isa column . . . name =name position =pos !eval! (or =num1 =num2 =carry) ==> . . . !chain! addition (=problem) :messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.))) Example Hint Template Examples? What are good help messages? Have some slides about this, no? Have had to add some slots to the columns to store words (strings) that can be used to fill the hint templates. Have had to change problem representations. Have had to change rules. Some now have more conditions, in order to extract information for help templates.Examples? What are good help messages? Have some slides about this, no? Have had to add some slots to the columns to store words (strings) that can be used to fill the hint templates. Have had to change problem representations. Have had to change rules. Some now have more conditions, in order to extract information for help templates.

49. Example hints 1. Now move on the the second column from the right. This is the �tenths� column. 2. Next big event: the second column from the right!!! Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.) `(Next big event #\: the #\space ,=pos column from the right #\! #\! #\! ) ) 2nd Example Hint Template



52. Example hints from this template 1. Now move on the the second column from the right. This is the �tenths� column. 2. Next big event: the second column from the right!!! Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.) `(Next big event #\: the #\space ,=pos column from the right #\! #\! #\! ) ) 2nd Example Hint Template

53. Sequences Combining Hints from Multiple Productions Start with the column on the right. This is the "ones" column. You need to add the two digits in this column. Adding 4 and 6 gives 10. The sum that you have, 10, is greater than 9. You need to move 10 of the 10 into the second column. And you need to write the rest of the 10 at the bottom of the first column. Write 0 at the bottom of the rightmost column. You need to complete the work on the first column. Write the carry from the first to the next column. Write 1 at the top of the second column from the right. Now move on to the second column from the right. This is the "tens" column. You need to add the two digits in this column. Adding 6 and 1 gives 7. There is a carry into this column, so you need to add the value carried in. This gives 7 + 1 equals 8. Write 8 at the bottom of the second column from the right

54. How could you model students who don�t carry? Instead of doing the addition correctly: Can you model a student who writes: Can you write a �bug rule� that models this error?


56. How to come up with a bug rule? Intuition: Need to model the �forgetting� of the subgoal to carry into the next column. Normal processing: If you then You will pretty soon be in situation:

57. Solution 1: Explicit Forgetting Rule ;; BUG-FORGOT-WRITE-CARRY-GOAL ;; IF The goal is to do an addition problem ;; And there is a pending subgoal to write a carry ;; THEN Delete that pending subgoal (w/o carrying it out). ;; (And carry on with the addition problem in the usual manner.) (defproduction-bug BUG-FORGOT-WRITE-CARRY-GOAL addition (=problem) =problem> isa addition-problem subgoals ($sg1 =write-carry-goal $sg2) =write-carry-goal> isa write-carry-goal ==> =problem> subgoals ($sg1 $sg2) ; Remove the write-carry-goal. !chain! addition (=problem) :messages (bug `(First #\, you need to write the carry from the previous column #\. ) ))

58. ;; BUG-FORGOT-WRITE-CARRY-GOAL ;; IF The goal is to do an addition problem ;; And there is a pending subgoal to write a carry ;; THEN Delete that pending subgoal (w/o carrying it out). ;; (And carry on with the addition problem in the ;; usual manner.) (defproduction-bug BUG-FORGOT-WRITE-CARRY-GOAL addition (=problem) =problem> isa addition-problem subgoals ($sg1 =write-carry-goal $sg2) =write-carry-goal> isa write-carry-goal ==> =problem> subgoals ($sg1 $sg2) !chain! addition (=problem) :messages (bug `(First #\, you need to write the carry from the previous column #\. ) )) Solution 1: Explicit Forgetting Rule


60. Setting Up a Curriculum Organize curriculum early on during the development of your tutoring system. Organize the tutor curriculum Sequencing of skills Break down curriculum into lessons and sections Each section involves a handful of �skills� Assign problem sets to each section Say how the curriculum breaks down into lessons and sections, and TDK will do the rest - it will do the bookkeeping for you. That is, it keeps track of where each student is in the curriculum so that students Breaking down occurs early on in the development process. Say how the curriculum breaks down into lessons and sections, and TDK will do the rest - it will do the bookkeeping for you. That is, it keeps track of where each student is in the curriculum so that students Breaking down occurs early on in the development process.

61. Setting Up Knowledge Tracing and Mastery Learning Indicate how to group the crucial productions into traced skills. Each traced skill corresponds to one or more productions. For each skill, provide values for the four knowledge tracing parameters. For each problem, indicate which productions are needed to solve it. Enables tutor to select problems that involve un-mastered skills. Say which productions you want to trace and TDK takes care of the rest - it does the math for you.Say which productions you want to trace and TDK takes care of the rest - it does the math for you.

62. Example: Addition Skills Start � Start in the rightmost column FOCUS-ON-FIRST-COLUMN Add � Add the addends in a column ADD-ADDENDS Write � Write the sum in a column WRITE-SUM ADDITION Done � Determine that all columns are done DONE ADDITION Next � Focus on the next column FOCUS-ON-NEXT-COLUMN Regroup � Realize that 10 goes to next column MUST-CARRY Write Carry � Write the carry WRITE-CARRY Add Carry � Add the carry to the sum ADD-CARRY ADDITION Each production corresponds to a skill.Each production corresponds to a skill.

63. Example from Addition Tutor Structure curriculum, based on skills Section1 � one column, no carry new skills: Start, Add, Write, Done Section2 � two columns, no carry new skills: Next Section3 � two columns, possible carry new skills: Regroup, Mark Carry, Add Carry Section4 � three columns, possible carry new skills: - Based on skills or problem type? The two go hand in hand of course - but you are somewhat constrained by the problem type - they determine the most usual combinations. One lesson. Note how this tracks the sequence of production rule models. Omitted one-column with carry.Based on skills or problem type? The two go hand in hand of course - but you are somewhat constrained by the problem type - they determine the most usual combinations. One lesson. Note how this tracks the sequence of production rule models. Omitted one-column with carry.

64. Excerpt from Lesson File (Make-Section :Required-Problems '(Problem11 Problem12 Problem15) :Remedial-Problems '(Problem13 Problem14 Problem16 Problem17 Problem18 Problem19) :Skills '(((Focus-On-First-Column Addition) :Description "Start") ((Done Addition) :Description "Done") ((Add-Addends Addition) :Description "Add") ((Write-Sum Addition ) :Description "Write") ))

65. Set Knowledge-Tracing Parameters Parameter (setf *p-initial* 0.33) (setf *p-learn* 0.20) (setf *p-guess* 0.20) (setf *p-slip* 0.10) (setf *p-known-p* 0.95) Represents probability that the student has learned the skill prior to the instruction the student learns the skill when successfully applying it the student gets the answer right even though the skill is not mastered the student gets the answer wrong even though the skill is mastered Represents mastery threshold

66. Provide Production Counts For problem (or any other 2-column problem without carrying) (set-problem-production-counts problem26+51 '(((FOCUS-ON-FIRST-COLUMN ADDITION) . 1) ((FOCUS-ON-NEXT-COLUMN ADDITION) . 1) ((ADD-ADDENDS ADDITION) . 2) ((WRITE-SUM ADDITION) . 2) ((DONE ADDITION) . 1) ))))

67. Collecting Logs ofStudent-Tutor Interactions in TDK Create �protocol collectors� when starting tutor (make-instance 'global protocol-collector :broadcast-station broadcast-station) (make-instance 'protocol-collector :broadcast-station broadcast-station) ; add to function ; make-addition-tutor Set directory parameters (setf *global-protocol-directory* "Home:Globals;") (setf *protocol-directory* "Home:Protocols;") Flip the switch (setf *collect-protocols* t)

68. Example Protocol from Addition Tutor *000:00:01 START STUDENT; "Student3", USER-DATA; NIL, PROBLEM; PROBLEM15, NPRESENTED; (0 0), START-TIME; "07/18/01 21:57", FILE-NAME; "Macintosh HD:Desktop Folder:Vincent:CIRCLE Summer School:TDK:Protocols:LESSON1:Student3:1%PROBLEM15%0.p-0", LESSON; LESSON1, LESSON-PRINT-NAME; "", SECTION; SECTION155, SECTION-PRINT-NAME; "", . *000:00:24 INITIAL-PROBABILITIES p/p-1; (0.6762857412475228 (ADD-ADDENDS ADDITION)), p/p-2; (0.9451767563766541 (DONE ADDITION)), p/p-3; (0.6762857412475228 (FOCUS-ON-FIRST-COLUMN ADDITION)), p/p-4; (0.6762857412475228 (WRITE-SUM ADDITION)), . *000:00:40 FOCUS SELECTED-WMES; (COLUMN1), . *000:14:926 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; (COLUMN1), ACTION; WRITE-RESULT, INPUT; "5", .

69. Example Uses of Log Data To improve model and tutor Is skill decomposition accurate? If �learning curves� for skills do not follow power law, crucial difficulty factors may have been overlooked. Tune knowledge tracing parameters Fit the parameters to the log data, use the new parameter values next time around. What kind of errors do students make? Any use for new bug messages? Do we have enough different problems? For research Relation between help use and learning outcomes. Compare learning rates in comparative studies. Interesting aside. The fact that the learning curves in the Lisp tutor were smooth was used as important evidence for the ACT-R theory - namely for the point that the productions were the right level of granularity to look at skill acquisition. Another aside is that the cognitive tutors were built originally as much as an application of Interesting aside. The fact that the learning curves in the Lisp tutor were smooth was used as important evidence for the ACT-R theory - namely for the point that the productions were the right level of granularity to look at skill acquisition. Another aside is that the cognitive tutors were built originally as much as an application of

70. END

Cognitive Tutor Development Kit TDK Tutorial

Cognitive Tutor Development Kit TDK Tutorial

Presentation Transcript

Cognitive Development

COGNITIVE DEVELOPMENT

Cognitive Development

Cognitive Development

Cognitive Development

Cognitive Development

Cognitive development

Cognitive Development

COGNITIVE DEVELOPMENT

Cognitive development

Cognitive Development

Cognitive Development

Cognitive Development

Cognitive Development

Cognitive Development

Cognitive Development

Cognitive Development: Cognitive Processes

Cognitive Radio Kit Tutorial

Cognitive Development