Asynchronous Online Tutoring: Using Multimedia to Teach Difficult Concepts

1. Asynchronous Online Tutoring: Using Multimedia to Teach Difficult Concepts Presented at the Seventh Sloan-C International Conference on Online Learning, November 16-18, 2001 Prof. Richard Larson and Laura Koller Center for Advanced Educational Services Massachusetts Institute of Technology 77 Massachusetts Ave. Cambridge, MA 02139

2. Why PIVoT? • Start with a brilliant educator • Broadcast video Help Sessions over MIT cable TV • Convert TV help to asynchronous, web-based “virtual office hours”

3. Motivation… • Newtonian Physics: most difficult subject for MIT freshmen • In a lecture hall with 100’s of other students, how much real interaction between student and professor? • Wouldn’t each student like to have the professor as a tutor, 24/7, anywhere?

4. Revising our Thinking about Teaching and Learning… • Teaching Learning • Teacher Mentor or Coach • Student Learner • Synchronous Asynchronous • Passive Active • Linear Nonlinear • Scheduled On-demand • Teaching material Accomplishing a Goal

5. An Intelligent Matching of Pedagogy and Technology to Content and Learners… Specific Knowledge Domain Pedagogy College: 18-26 yr.olds Technology Post- graduate K-12

6. Fundamental Hypothesis… • PIVoT can increase the equivalent face-to-face contact between learner and mentor by an order of magnitude or more.

7. Related Literature… http://www.free-clip-art.com

8. Taxonomy of Scientifically-Oriented Educational WebsitesNachmias and Turi, 2001 • Descriptive dimension (e.g., target population, site developers, language) • Pedagogical dimension • (instructional model, instructional means, cognitive demands) • Representational dimension (representational structure and means, navigation tools) • Communication dimension (links configuration, distant learning modes) • Scientific content dimension (content level, visualization means, historical background)

9. Descriptive dimension(target population, site developers, language)College freshmen taking Newtonian physics, CECI, web-based languages with database

10. Pedagogical dimension(instructional model, instructional means, cognitive demands)- Nonlinear student-directed inquiry into various media - Lecture, demo, problem solutions, concept discussions, simulations, tests - Reflection and integration of physical world and math world depictions

11. Representational dimension (representational structure and means, navigation tools)- Streaming video of lectures and problem sessions- Textbook- Tests- Animations- Various search options (key word, syllabus)

12. Communication dimension(links configuration, distant learning modes)- Links to animations and various key words - No difference between on-campus asynchronous learning and the distance learning option

13. Development of PIVoT • $1M in grants • Start: April 1998 • Design features – acquire content – build • Beta release, Fall 1999 • Evaluate beta, Spring 2000 • Final deployment: Fall 2000 • Final evaluation, Spring 2001

14. Spring 2000 PIVoT Evaluation Results (Prof. Lewin taught the course)… • 98% registered to use it • 90% had a positive experience • 87% accessed it from living group • 52% used it 1-10 times; 48%, 11-2 times According to a student survey (45% of registered users responded):

15. Spring 2000 Evaluation Results (cont’d.)… Student survey says: • 84% watched help session videos • 83% worked on problem sets • 78% watched lectures • 64% used it to study for exams • 63% used it to learn concepts • 52% watched exam reviews

16. Some student comments... • “A lot of people think it is the only reason they are passing 8.01, because it’s a tutor available 24 hours a day….” • “PIVoT is what makes MIT, MIT.” • “I’ve never spoken to a fellow freshman who hasn’t used and benefited from PIVoT.” • “Students are very thankful for the existence of PIVoT.” • “...even more helpful than lectures because it simulates a one-on-one learning environment.”

17. Spring 2001 Evaluation (Prof. Lewin did not teach the course)… • PIVoT was used in entry-level physics classes during the Fall 2000 semesterat • MIT • RPI • Wellesley

18. PIVoT was used in different ways… MIT: • Traditional large lecture/recitation format • Voluntary educational supplement RPI: • Studio physics format • Required for problem set homework Wellesley: • Small lectures • Required for problem set homework

19. The Study had Two Main Goals: • To learn about PIVoT functionality, student usage patterns, and student attitudes toward PIVoT in different educational contexts • To see if the use of PIVoT enhanced conceptual learning

20. Methodologies… • A student survey was administered to registered PIVoT users at all three campuses. • Data were collected and analyzed at RPI, where a controlled educational experiment was conducted: • four out of nine entry-level physics classes used PIVoT, while five did not.

21. Evaluation Instruments… • Two pre- and post- diagnostic tests were administered to assess students’ improvement in conceptual learning: • Force Concept Inventory (FCI) • assesses understanding of basic Newtonian physics concepts • most widely used physics instrument • Force Motion Conceptual Examination (FMCE) • assesses conceptual learning gains

22. Findings… • On the FCI, PIVoT users had significantly higher gain scores than non-users (data on following 2 slides). • In regard to the FMCE gain scores, there were few differences between the two groups. This may be because the FCI tests topics that PIVoT covers, but the FMCE does not.

23. Table 1. RPI Class Scores by PIVoT Use PIVoT Non- Users Users Total Means FCI Pre-Test 53 50 52 FMCE Pre-Test 42 43 42 FCI Post-Test 68 *** 56 63 FMCE Post-Test 71 67 70 FCI Post-Pre Difference 14 * 7 11 FMCE Post-Pre Difference 28 25 28 FCI Class Gain 31 13 23 FMCE Class Gain 50 43 48 FCI Indiv, Student Gain 33 * -1 19 FMCE Indiv. Student Gain 49 42 46 (T-test level of statistical significance *** p<.001, ** p<.01, * p<.05)

24. Table 2. RPI Class Scores by PIVoT Use PIVoT Non- Users Users Total Exam 1 80 79 79 Exam 2 75 74 75 Final Exam 76 73 75 Homework 84 84 84 In-Class Activities 96 97 96 Quizzes 95 92 93 Final Numerical Grade Score 83 82 83 Percent Distribution A (91-100) 28% 33% 31% B (81-90) 42% 36% 39% C (71-80) 21% 17% 19% D (61-70) 6% 9% 8% F (60 or lower) 3% 4% 4%

25. PIVoT Evaluation Report… • Available online at: http://caes.mit.edu/research/pivot/ PIVOT_report1.pdf

26. PIVoT Architecture… Web server Client Database server Video server information flow

27. PIVoT Navigation… • Free text search • Keyword index • Topic tree • Lectures • Textbook Multiple nonlinear navigation modes:

28. PIVoT Features… • 35 Lectures • 20 hrs. Help Sessions • 600-page Textbook • FAQs + Answers • Practice Problems • Java Simulations • Discussion Board

29. Helping Signposts for Students…

30. Signposts =PIVoT’s “Personal Tutor” • Follows learner as she acts as spelunker through the web site • Makes suggestions based on key words and topics being studied

31. PIVoT Web Accessibility… • Research focus: • Making complex scientific web sites accessible to deaf and blind users • Methods: • video captions • audio descriptions • overall web site design guidelines • Collaboration with WGBH-NCAM • Funded by NSF and Mitsubishi

32. More information on the Access to PIVoT Project… • MIT: http://caes.mit.edu/research/ access_pivot/index.html • WGBH National Center for Accessible Media: http://ncam.wgbh.org/webaccess/pivot/

33. Selected References • “Integrating Multiple Teaching Methods into a General Chemistry Classroom,” J. S. Francisco, G. Nicoll and M. Trautmann, J. Chem. Education, 75 (2) 1998 210-213. • “Taxonomy of Scientifically Oriented Educational Websites,” R. Nachmias, I. Tuvi, J. Science Education and Technology, 10 (1) 2001 93-104. • “Using the Internet to Enhance Student Understanding of Science: The Knowledge Integration Environment,” M. C. Linn, P. Bell, S. Hsi, Interactive Learning Environments, 6 (1) 1998 4-38. • “Lifelong Science Learning: A Longitudinal Case Study,” M. C. Linn and B. S. Eylon, Proc. Of CogSci96 (pp. 597-602), 1996 Lawrence Erlbaum Associates. • “Creating Lifelong Science Learners: What Models Form a Firm Foundation?” M. C. Linn and L. Muilenburg, Educational Researcher 25 (5) 1996 18-24. • "A Three-School Comparative Analysis of Student Usage Patterns and Attitudes toward PIVoT" by Dr. Alberta Lipson, MIT Teaching and Learning Laboratory, July 2001.

34. Selected References, cont’d… • Devlin, Maureen, Richard Larson and Joel Meyerson (eds.), The Internet and the University 2000 Forum, EDUCAUSE, 2001 Boulder, CO. • Larson, Richard C. and Joel W. Meyerson, “We’ve Got the Steam Locomotive – Now Let’s Design the Railroad Cars!” Preface in The Internet and the University 2000 Forum. 2001 • Larson, Richard C. and Glenn P. Strehle, “Edu-Tech: What’s a President to Do?” An invited book chapter in Technology Enhanced Learning: Opportunities for Change, edited by Paul Goodman of Carnegie Mellon University and produced by Lawrence Erlbaum publishers, 2001.

35. URLs and Contact Info… To request a guest account, go to the PIVoT web site: http://curricula2.mit.edu/pivot/ MIT Center for Advanced Educational Services: http://caes.mit.edu/ Prof. Richard Larson, Principal Investigator: rclarson@mit.edu Laura Koller, Project Manager: lkoller@ceci.mit.edu