Enhancing the Classroom Learning Experience with Web Lectures

1. Enhancing the Classroom Learning Experience with Web Lectures Jason Day & Jim Foley {dayja, foley}@cc.gatech.edu Past Contributor: Remco Groeneweg

2. Introduction • By taking the lecture out of the class onto the web, time in class can be spent in more engaging ways

3. Motivation • Use class time for more learning by doing, less learning by listening • A lot of lecture material to be covered…learning-by-doing activities take a lot of time

4. Research Areas • Educational: • Educational framework for web lectures + meaningful in-class learning activities • Ways to motivate students to watch web lectures • Which in-class activities work well with web lectures? • Technological Affordances: • Ways to make web lecture watching more engaging (delivery mechanism, interactive elements, etc.)

5. Technology • Web lecture: studio-recorded, condensed lecture

6. Web Lecture Production Workflow PowerPoint slides Streamed video + HTML web lecture Microsoft PowerPoint Microsoft Producer Streaming Server Web Browser Import Publish Narrate slides with audio/video View lecture

7. Web Lecture Studio Design Total equipment cost: ~ $3000

8. Pedagogy • Accepted educational approaches as inspiration for the design of in-class learning activities • Learning activities stimulate the social process of articulation and reflection on shared public artifacts • Learning activities grounded in real-world examples or anchored in group-project activities

9. Related Work • Tutored Video Instruction @ Stanford (Gibbons et al. 1977) • eTeach @ UWisconsin (Moses, 2002) • iCampus @ MIT • Internet Learning Environments @ Edith Cowan University (Oliver, 2001) • HWebs @ GT (Collard et al. 2002) • eClass @ GT (Abowd et al. 1999)

10. Research Context • CS/Psych 4750: Senior elective course on Human Computer Interaction • Typically 25-35 students • Semester-long UI design project • 3 to 5-person teams • Gather requirements, prototype, build, test • Midterm, final, homeworks

11. Formative Evaluation, Fall 2003 • Student feedback guided us to optimal level of web lecture production quality • Notable improvements • Audio quality • Animate slides to help viewer focus • Include body and hands in video • ‘Teleprompter’ and slide advancement

12. Pilot Study, Spring 2004 • 35 students enrolled • 27 class meetings (down from 30) • 17 lectures (down from 25) • 13 web lectures, totaling 277 minutes (about 3.5 80-minute class meetings) • 7 new constructivist-inspired learning activities

13. Example Class Activities • Project anchored: • Requirements Gathering presentations • Prototype poster session (old) • Evaluation Plan presentations • Cognitive Walkthrough presentations • Critiquing: • Professor critiquing • Small group critiques • UI Hall of Fame/Shame • Use and critique of physical interfaces

14. Evaluation Methods • Four surveys • Survey 1: Second week of class • Survey 2: Mid-Semester • Survey 3: Group vs. Individual • Survey 4: Final • Three focus groups • Focus group 1: Second week, 5 students • Focus group 2: Sixth week, 5 students • Focus group 3: Week before finals, 4 students • Three observations

15. Pilot Study Results • Satisfactory web lectures can be created with modest faculty time and simple, inexpensive equipment • Students desire some form of explicit motivation to watch web lectures • Students prefer watching web lectures individually over group watching • Students find class activities educational and enjoyable

16. Pilot Study Results (2) • Slight preference for this course format over traditional lecture format (avg. 3.36 out of 5) • Web lectures not as useful for exam review as students predicted on first and second surveys • Covered same amount of material and added 7 learning activities…with 3 fewer class meetings • Informal judgment that educational outcomes were same as in Fall ‘03

17. Quasi-experimental Study, Spring 2005 • Two sections of 4750 • One taught traditionally (n=18) • One using our web lecture / class activity format (n=29) • Hypothesis • Web lecture intervention as good or better based on educational outcomes and subjective attitudes

18. Study Design • Matched two sections on: • Instructor teaching the course • Topics covered • Lecture slides used in class or integrated into web lecture • Assigned reading • LHWs, homeworks, and semester project • Mid-term and final exams • Time on task: • Control section: time spent in class • Experimental section: time spent in class plus total running time of assigned web lectures • Counterbalanced and blind grading • Exception: Projects were graded together with same grading criteria

19. Lecture Homeworks (LHWs) • Short homeworks associated with each web lecture / lecture • Blind to instructor • Served as • explicit motivation (15 LHWs worth 1% each) • discussion guide for next class meeting

20. Study Details – Experimental Section • 21 class meetings (compared to 28) • 3 lectures (compared to 25) • 27 web lectures, totaling 537 minutes (about 7 80-minute class meetings) • 13 Q&A / discussion / learning activity classes

21. Preliminary Results – Educational Outcomes • So far, experimental section average grades have been higher for every graded assignment and exam

22. Preliminary Results – Educational Outcomes (2) • Experimental section average LHW grades across all LHWs is significantly (p<0.01) higher than control section.

23. Preliminary Results – Educational Outcomes (3) • Experimental section average project grades are significantly higher for Phase 1 (p<1.8E-7) and Phase 3 (p<0.03); Phase 2 grades were higher, but not statistically significant (p=0.06).

24. Preliminary Results – Subjective Attitudes • If you compare the new course format of web lectures and in-class activities to the traditional in-class lecturing format, how would you rate the new course format? • Average Responses • Interim Survey after 11 web lectures: 3.11 • Midway Survey after 25 web lectures: 3.61 • Statistically significant increase (p<0.04)

25. Preliminary Results – Subjective Attitudes (2) • Both sections reported LHWs helped them focus on and learn the material presented • Both sections came in with very positive attitudes towards the relevance of HCI to education and career • Experimental section has slightly increased • Control section has slightly decreased

26. Future Work • Look for correlation between survey responses and digital library use • Implement and evaluate technological affordances of web lectures • Question submission mechanism, built-in interactive learning activities, discussion forum, FAQs, variable playback speed, etc. • Development new in-class activities, current activities will be further improved • Use of this class format in other courses

27. The End • Web lectures available at: http://hcc.cc.gatech.edu/videolectures • Questions/comments: • Stop by the recording ‘studio’ TSRB 377 • Email dayja@cc

28. Web Lecture Host Infrastructure Streaming Server Windows Media Services 9 Series Web Server Apache/1.3.29 TCP/IP Internet HTTP MMS Client Machine Web browser + Media Player

29. Production Quality How would you rate the production quality (i.e. audio/video quality) of the web lectures?

30. Perceived Usefulness of In-class Activities

31. Perceived Usefulness of Other Possible Technological Affordances

32. Human-Centered Computing Education Digital Library Ed Clarkson, Jason Day, Jim Foley, Aarjav Trivedi

33. IntroductionHuman-Centered Computing (HCC) Education Digital Library • Support the new GT College of Computing Ph.D. program in Human-Centered Computing • Expose Georgia Tech as a leader for HCI/HCC educational content • Leverages one of the largest HCI faculty in the world! • Enhance the overall learning experience via rich set of instructional material

34. A Question • What resources are there for people who want to learn or teach HCI/HCC?

35. Existing Answers • HCI/HCC-specific: • HCI Bibliography (Gary Perlman) • SIGHCI Teaching Resources • Colleagues • General • Google • Educational Digital Libraries (e.g., )

36. Common Issues • HCI-specific: • Materials not specifically for education • Not active • Not universal • General: • Wide target audiences • Problems with linking • Browsing? • Quality control

37. Requirements Gathering • Focus groups • Online Surveys • HCI Education Workshop (CHI ’05) • http://hcc.cc.gatech.edu/chi2005workshop.htm

38. Derived Requirements • High-quality, locally-stored content • Syllabi, lectures, tests & exams, homework and project assignments, videos • Quality assurance • Filtering over reviewing • Controlled reviews over free-for-all

39. Derived Requirements (cont’d) • Browse vs. search • “I would like to see the detailed level of granularity because many instructors of HCI are thrown into the task rather then being trained in it.”’ • Scalability/sustainability

40. Prototype Implementation • Demo the live site…

41. Future Work, Research Goals and Questions • Version 2.0 in progress • Dynamic, database-backed • What can aggregate user behavior over time tell us? • Web lecture • Automated hierarchy reorganization

42. Future Work, Research Goals and Questions • Scalability/sustainability • What mechanisms (both organizational and technological) are needed for a sustainable educational knowledge repository? • How well does Wikipedia model transfer? • How does the taxonomic organizational scheme scale? What are alternatives?

43. Questions?