Bridging Research and Practice in Distance Education

1. Bridging Research and Practice in Distance Education A communication management approach to a state-of-the-art Online Learning Environment. Erin Shaw CARTE/DEN Collaboration Viterbi School of Engineering

2. Goals and Progress DEN OLE goals • Feature innovative teaching and learning facilities • Increase learner-centered participation and collaboration • Encourage instructor-initiated experimentation and assessment • Bridge research and practice in distance education CARTE progress toward goal • Deploy ISI discussion board • Automate registration via single sign on from Blackboard • Render development at ISI independent of production at DEN • Integrate discussion and artifacts (course doc, webcasts) • Allow students to attach and share documents • Focus discussion and assessment • Expand communication capabilities • Support IM (mobile), RSS feed, summarization and activity watches • Extend document capture capabilities • Generate text, html, and pdf file formats and metadata • Parse, segment, tile, and make searchable

3. Integrated System Overview DEN Blackboard System Student Collaboration Tools ISI Discussion Board Agent Response Agent Analysis DEN File System Instructor Assessment Tools ISI Document Database Completed In Development Planned

4. DEN Blackboard Link to ISI Discussion Board Native Blackboard discussion board link replaced with link to ISI board

5. ISI Discussion Board Blackboard sends user profile and discussion board automatically registers user

6. Current State Current state • In use since Spring 2005 • Two DEN courses currently online (CSci577, CSci402) • Four archived courses available (AME599, 3xCSci402) • Extensive processing of 402 course materials completed Current focus • Utilize discussion board for research • Data benefits learning, mining, ped. agent and NLP research • Spawned AIED paper, HTL demo, and AERA, IUI submissions • Experimental work (UCLA intern and CSci577 class) • Proposal planning Now is a critical time! • Funding has ended (funds slated to run out ~12/2005) • Minimal funding would assure continued support for • current course use • research evaluation

7. Project Rationale Addresses instructional challenges • Instructors want to experiment but have limited support • E.g. Can maintain own web site but not discussion board • registration, security, and maintenance are obstacles • Students expect to use current communication technologies • Evaluation of ‘Learner centered’ model difficult but necessary Addresses technical challenges • Building own OLE, duplicating DEN system impractical • Blackboard’s Building Blocks solution incomplete • API incomplete, integration too tight • ISI development within DEN production environment problematic Our solution • Encourages adoption (maximizes ease-of-use) • Enables development at ISI (minimizes disruptions) • Uses open source development model (maximizes code maintenance) • Allows for evolution from Blackboard • 18 open source CMSs currently available (EduTools, August, 2005)

8. Discussion Board Deployment Software • phpBB, open source, latest version, running at ISI • http://www.phpbb.com, http://melodic.isi.edu/phpbb2 • ISI modifications adhere to community development standard • modularized and self-documenting • we maintain changes, phpBB dev community maintains core Courses online (2005) • Spring: AME599, CSCI402 • Summer: CSCI402 • Fall: CSCI555, CSCI402 • Archived 2004 discussion (CSCI402 and CSCI555) Usage statistics • ~130 past students, ~130 current students • Over 2000 messages

9. Summary of Accomplishments Main accomplishments • Single sign on from Blackboard (fully integrated) • Document capture (Blackboard-integrated user interface) • Successfully used in five courses Discussion board details • Auto-creation of lecture discussions • Integration of webcasts and lecture notes to follow • Annotation system • Instant messaging/mobile learning integration • Answer mining of archived discussions • Archiving features Document capture details • Automate parsing of uploaded document • Creates PDF, HTML, text and metadata files for all documents • Makes documents searchable via discussion board

10. Summary of New Research Research thrusts • SPADE rhetorical analysis (Shaw) • Classummary evaluation (Liang Zhou, GRA) • Q/A processing pipeline (Donghui Feng, GRA) • 800 messages manually annotated for network analysis • Ontology creation (in planning with Kim) Data share tools • API for message export • XML schema for message formatting • API for agent message insertion Evaluation tools • Statistics viewer • Questionnaire creation tool (auto-send to users) • Instrumentation for Q/A evaluation (planned)

11. Screen Shots

12. Artifact-Based Discussion Documents and images displayed next to discussion Supports all file formats, including PDF

13. Topic-Based Annotation Simple markup of HTML version of document directs processor to create links to auto-generated topics

14. Two way communication via instant messaging BBS sends message Notification User queries BBS for details

15. AT&T Samsung x427 Cingular Nokia 3595 Working Assets Sprint LG 1200 T*Mobile Sidekick Mobile Communication via IM AOL Instant Messenger communication Worked on all phone types tested

16. Statistics Viewer Accessible via ‘Administrator Panel’ Allows instructors to access ISI data

17. Question Answering Research Text is incoherent, non-factoid Questions barely identifiable, often include multiple sentences or paragraphs Answers require explanations

18. Question Answering Placeholder for results screen shot Answer-bot refers questioner to most similar discussion and document segment.

19. XML Schema for Message Export

20. Classummary Tool (Liang Zhou)

21. Classummary Archive

22. Classummary Notification Scheduling Interface

23. Classummary Notification Email

24. Research Potential

25. Discourse Analysis (Shaw) Apply SPADE (Soricut & Marcu) rhetorical parser to discussion board data (spring 2005) • Run on over 1000 messages (Open University corpus) • Used to validate differences in tutor and student messages • AIED 2005 paper Assessing and Scaffolding Collaborative Learning in Online discussions Generalize work (to be proposed) • Adapt for real-time analysis • Run on DEN data • Evaluate different types of messages • Question vs. comment vs. answer • Tutor-instructor vs. student • Develop message classifier (done manually now) • Develop rules for promoting discussions

26. Pedagogical Interventions (Beal) Analyze discussion board communication (spring 2005) • Analysis of small corpus of messages (Beal’s UMass course) • [More information here] Generalize work (proposed in AIED poster, Beal, Kim, Shaw) • Amalgamation of agent and ontology work, NLP, and discourse analysis • [More information here]

27. Learning Ontologies (Jihie Kim) Analyze discussion board communication (summer 2005) • Annotation of small corpus of messages (CSci555) • Classify on-topic messages using keywords • Compare technical vs general discussion messages Generalize work (to be proposed) • Acquire keywords automatically • Identify keywords in messages • Assess students’ contributions