1 / 50

The interaction plateau

The interaction plateau . CPI 494, April 9, 2009 Kurt VanLehn. Schematic of a natural language tutoring systems, AutoTutor. Step end. T: Tell. Step start. Only if out of hints. T: Elicit. S: Correct. Remediation:. T: Hint or prompt. S: Incorrect.

walterperkins
Télécharger la présentation

The interaction plateau

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The interaction plateau CPI 494, April 9, 2009 Kurt VanLehn

  2. Schematic of a natural language tutoring systems, AutoTutor Stepend T: Tell Stepstart Only if out of hints T: Elicit S: Correct Remediation: T: Hint or prompt S: Incorrect

  3. Schematic of other natural language tutors, e.g., Atlas, Circsim-Tutor, Kermit-SE Stepend T: Tell Stepstart Only if out of hints T: Elicit S: Correct Remediation: T: What is…?S: I don’t know.T:Well, what is…S:…T:… S: Incorrect Often called a KCD: Knowledge construction dialogue

  4. Hypothesized ranking of tutoring, most effective first • Expert human tutors • Ordinary human tutors • Natural language tutoring systems • Step-based tutoring systems • Answer-based tutoring systems • No tutoring

  5. Hypothesized effect sizes

  6. Hypothesized effect sizes Bloom’s (1984) 2-sigma: 4 weeks of human tutoring vs. classroom Classroom

  7. Hypothesized effect sizes Kulik (1984) meta-analysis of CAI vs. classroom  0.4 sigma Classroom

  8. Hypothesized effect sizes Many intelligent tutoring systems: e.g., Andes (VanLehn et al, 2005), Carnegie Learning’s tutors… Classroom

  9. My main claim: There is an interaction plateau

  10. A problem and its steps • Suppose you are running in a straight line at constant speed. You throw a pumpkin straight up. Where will it land? • Initially, you and the pumpkin have the same horizontal velocity. • Your throw exerts a net force vertically on the pumpkin. • Thus causing a vertical acceleration. • Which leaves the horizontal velocity unaffected. • So when the pumpkin falls, it has traveled the same distance horizontally as you have. • Thus, it lands in your hands

  11. A dialogue between a human tutor (T) and human student (S) • Suppose you are running in a straight line at constant speed. You throw a pumpkin straight up. Where will it land? S: Behind me. • T: Hmm. Let’s think about that. Before you toss the pumpkin and are just carrying it, do you and the pumpkin have the same speed?S: Yes • T: Good. When you toss it up, is the net force on it exactly vertical?S: I’m not sure.T: You exert a force on the pumpkin, right?Etc.

  12. Schematic of dialogue about a single step Stepend T: Tell Stepstart T: Elicit S: Correct Remediation: T: Hint, or prompt, or explain, or analogy, or … S: Incorrect

  13. Comparisons of expert to novice human tutors Stepend T: Tell Novices Stepstart Experts T: Elicit S: Correct T: Hint, or prompt, or explain, or analogy, or … S: Incorrect Experts may have a wider variety

  14. Schematic of an ITS handling of a single step Stepend T: Tell Stepstart Only if out of hints S: Correct T: Hint S: Incorrect

  15. Major differences • Low-interaction tutoring (e.g., CAI) • Remediation on answer only • Step-based interaction (e.g., ITS) • Remediation on each step • Hint sequence, with final “bottom out” hint • Natural tutoring (e.g., human tutoring) • Remediation on each step, substep, inference… • Natural language dialogues • Many tutorial tactics

  16. Conditions(VanLehn, Graesser et al., 2007) • Natural tutoring • Expert Human tutors • Typed • Spoken • Natural language dialogue computer tutors • Why2-AutoTutor (Graesser et al.) • Why2-Atlas (VanLehn et al.) • Step-based interaction • Canned text remediation • Low interaction • Textbook

  17. Human tutors(a form of natural tutoring) Stepend T: Tell Stepstart T: Elicit S: Correct T: Hint, or prompt, or explain, or analogy, or … S: Incorrect

  18. Why2-Atlas(a form of natural tutoring) Stepend T: Tell Stepstart T: Elicit S: Correct A Knowledge Construction Dialogue S: Incorrect

  19. Why2-AutoTutor(a form of natural tutoring) Stepend T: Tell Stepstart T: Elicit S: Correct Hint or prompt S: Incorrect

  20. Canned-text remediation(a form of step-based interaction) Stepend T: Tell Stepstart T: Elicit S: Correct Text S: Incorrect

  21. Experiment 1: Intermediate students & instruction

  22. Experiment 1: Intermediate students & instruction No reliable differences

  23. Experiment 2:AutoTutor > Textbook = Nothing Reliably different

  24. Experiments 1 & 2(VanLehn, Graesser et al., 2007) No significant differences

  25. Experiment 3: Intermediate students & instruction Deeper assessments

  26. Experiment 3: Intermediate students & instruction No reliable differences

  27. Relearning Experiment 4: Novice students & intermediate instruction

  28. Experiment 4: Novice students & intermediate instruction All differences reliable

  29. Relearning Experiment 5: Novice students & intermediate (but shorter) instruction Add Add

  30. Experiment 5: Novice students & intermediate instruction No reliable differences

  31. Experiment 5: Low-pretest students only Aptitude-treatment interaction?

  32. Experiment 5, Low-pretest students only Spoken human tutoring > canned text remediation

  33. Experiments 6 and 7 Novice students & novice instruction Was the intermediate text over the novice students’ heads?

  34. Experiments 6 and 7 Novice students & novice instruction No reliable differences

  35. Interpretation = Can follow reasoning only with tutor’s help (ZPD) predict: Tutoring > Canned text remediation = Can follow reasoning without any help predict: Tutoring = Canned text remediation Experiments 1 & 4 Content complexity Experiments 3 & 5 Experiments 6 & 7 High-pretest Low-pretest Intermediates High-pretest Low-pretest Novices

  36. Original research questions • Can natural language tutorial dialog add pedagogical value? • Yes, when students must study content that is too complex to be understood by reading alone • How feasible is a deep linguistic tutoring system? • We built it. It’s fast enough to use. • Can deep linguistic and dialog techniques add pedagogical value?

  37. When content is too complex to learn by reading alone: Deep>Shallow? Why2-Atlas is not clearly better than Why2-AutoTutor

  38. When to use deep vs. shallow? Use both Use deep Use locally smart FSA Use equivalent texts

  39. Results from all 7 experiments(VanLehn, Graesser et al., 2007) • Why2: Atlas = AutoTutor • Why2 > Textbook • No essays • Content differences • Human tutoring = Why2 = Canned text remediation • Except when novice students worked with instruction designed for intermediates, then Human tutoring > Canned text remediation

  40. Other evidence for the interaction plateau (Evens & Michael, 2006) No significant differences

  41. Other evidence for the interaction plateau (Reif & Scott, 1999) No significant differences

  42. Other evidence for the interaction plateau (Chi, Roy & Hausmann, in press) No significant differences

  43. Still more studies where natural tutoring = step-based interaction • Human tutors • Human tutoring = human tutoring with only content-free prompting for step remediation (Chi et al., 2001) • Human tutoring = canned text during post-practice remediation (Katz et al., 2003) • Socratic human tutoring = didactic human tutoring (Rosé et al., 2001a • Socratic human tutoring = didactic human tutoring (Johnson & Johnson, 1992) • Expert human tutoring = novice human tutoring (Chae, Kim & Glass, 2005) • Natural language tutoring systems • Andes-Atlas = Andes with canned text (Rosé et al, 2001b) • Kermit = Kermit with dialogue explanations (Weerasinghe & Mitrovic, 2006)

  44. Hypothesis 1: Exactly how tutors remedy a step doesn’t matter much Stepend T: Tell Stepstart T: Elicit S: Correct What’s in here doesn’t matter much S: Incorrect

  45. Main claim: There is an interaction plateau Hypothesis 1

  46. Hypothesis 2: Cannot eliminate the step remediation loop Stepend T: Tell Stepstart Must avoid this T: Elicit S: Correct S: Incorrect

  47. Main claim: There is an interaction plateau Hypothesis 2

  48. Conclusions • What does it take to make computer tutors as effective as human tutors? • Step-based interaction • Bloom’s 2-sigma results may have been due to weak control conditions (classroom instruction) • Other evaluations have also used weak controls • When is natural language useful? • For steps themselves (vs. menus, algebra…) • NOT for feedback & hints (remeditation) on steps

  49. Future directions for tutoring systems research • Making step-based instruction ubiquitous • Authoring & customizing • Novel task domains • Increasing engagement

  50. Final thought • Many people “just know” that more interaction produces more learning. • “It ain’t so much the things we don’t know that get us into trouble. It’s the things we know that just ain’t so.” • Josh Billings (aka. Henry Wheeler Shaw)

More Related