Topics = Domain-Specific Concepts

1. Concept-Level Topic Analysis of Tutoring Dialogs Amruta Purandare, Intelligent Systems Program Dr. Diane Litman, Department of Computer Science • Intelligent Tutoring SPOKEn dialog-based system • Interactive Dialog Sessions teaching Qualitative Physics • Backend: Why-2 text-based physics tutoring [VanLehn et.al.] • Tutor’s Role: • Present a problem • Clarify missing points, Correct misconceptions • Ask Q’s to check understanding • Request to revise answer • Student’s Role: • Write Essays • Answer Q’s • Topics = Domain-Specific Concepts • Online Physics Encyclopedia ‘Eric Weisstein's World of Physics’ • Contains total 3040 terms including multi-word concepts such as “Frictional Force”, “Newton’s second law” etc. • Present analysis limited to single-word concepts • Ambiguity: “current problem” Vs “electric current” • 36 physics concepts observed in our data • 10 concepts appear in ideal answers are marked as ‘Critical’ • Analyze number of times each concept is repeated per student • Tutor: You need to show even more of your reasoning in your essay. What is the name of the principle of physics that you could apply to determine the difference in acceleration between the car and truck given that they both experience the same force? • Student: I don’t know • Tutor: Newton's Second Law. We know that Newton's Second Law expresses a relationship between force, mass, and acceleration. What is the equation for that relationship? • Student: Force equals mass times acceleration • Tutor: Fine. Applying this principle, if you apply equal force to two objects of differing masses, you can use the relative accelerations of the two objects to determine the relationship between the respective masses of the two objects. In this case, which vehicle will have a smaller acceleration? • Student: Truck • Tutor: Uh huh. So then you know that during the impact, the magnitude of the acceleration of the truck is less than the magnitude of the acceleration of the car for every time instance. Try to add something about this to your essay. • Students take tests before (pre) and after (post) tutoring • We study partial correlations (R) between post-test scores and concept-metrics, controlled for pre-test scores 3. Methodology 1. Background: ITSPOKE 4. Preliminary Results Metrics 4. What are the forces acting on the pumpkin? 6. Good! Revise your essay now Automatic Topic Extraction • #physics-concepts, #critical-concepts discussed per student • #dialog turns that discuss n concepts (where n = 0,1,2,3+,1+) • Average #physics-concepts per turn ^^^^^^^^^^^ ^^^^^^^^^^^^ 2. Done! 3. You need more details Word Level Correlations 5. Gravity? 1. Solve this problem ^^^^^^^^^^^^ ^^^^^^^^^^^ Turn Level Correlations Example • Shallow Metrics (Automatic) [Litman et. al. ITS04] • Dialog Length (#turns, #words, words/turn) • Distinguish Student and Tutor Contributions • Deep Metrics (Manually Labeled) • Student and Tutor Actions [Forbes-Riley et. al. AIED05] • Misses the central theme of discussion: “Topics” or “Content” • Show: Topics correlate to Learning • Automatically identify Topic Structures in Dialogs • Granularity • Simple, Complex, Abstract Topics? • 1 topic per turn/sentence? • No Manual Annotations 2. Motivation: What Affects Learning? Bold indicates correlations significant at 0.05 level Bold and Italics indicates correlations significant at 0.01 level S = Student (spoken), T = Tutor (spoken), E = Essay (written) Previous Work Conclusions • Physics concepts show better correlations than all words • #Dialog turns with 0 and 3+ concepts show no impact • Student contributions matter! Our work • Include Multi-Word Concepts e.g. “Newton’s Second Law” • Do non-physics words have any role in learning? • Analyze Discourse-Level Topic Structures and Segments 5. Current Directions Evaluation Challenges