Project 1 Pre-Final Presentation Quiz Preparation in an online management system

1. Project 1Pre-Final PresentationQuiz Preparation in an online management system Kenneth Compass Sanjay Jauhar Camille Mercado Kristen Wakamoto

2. Methods Summary Phase 1 • Contact and set up either a face to face, telephone or Skype meetings with past instructors who have used Moodle v1.97 • Assess Moodle 1.97 using HCI Guidelines Primary •  Gain insight of the quiz preparation system through a walkthrough with the users • Think aloud procedure • Done on Moodle 1.97 • Questionnaires are also given out; Unstructured Interviews  Secondary •  For users that were not able to have scheduled meetings with us • Sent out questionnaires/surveys about their usage; questions based on interviews with primary interviewees

3. Methods Summary Phase 2 • Formulate structured interviews • Re-administer structured interviews to new respondents  • Based on Moodle 2.0 Primary • Look into differences between 1.97 and 2.0 • Ask test subjects to try out 2.0 and ask them for feedback • Structured Interviews and re-evaluation Secondary •  Look into differences between 1.97 and 2.0 • Ask secondary users about differences found through questionnaires and see what they think about it • Analyze data and make recommendations

4. Methods Summary Phase 3  • Contacted users who did not respond to the first attempt • Determined their availability. • Sent out surveys to those who were under time constraints • For others, we set up either face to face, Skype, or phone interviews • Think out loud walk through's of Moodle 1.97 • Ask users to look at Moodle 2.0 and then send questionnaires regarding their thoughts

5. Preliminary Results Initial Interviews (done with Moodle 1.97): Phone, Skype, and  In-Person Think Out Loud Walkthroughs •     Major usability issues identified and confirmed with   pre-assessment of system by team. •     Turn editing on/off   •     Many options - confusing and daunting •     Adding questions does not go directly to quiz • Clunky interactions to correctly add a question to a quiz

6. Preliminary Results Demographic Interviewed: • Got in touch with 15 instructors • Most were in their 40s and 50s • Taught mostly Management, Engineering, or Computer Science classes at the University Extension • Have used the Moodle system for at least 1.5 years • Create their own quizzes • Have not really had to use any other learning management systems aside from Moodle

7. Preliminary Results In Person and Skype Interviews (1.97) • Screen recording with walkthroughs of the Moodle System • Found the initial set up of the quiz system to be easy • e.g. Creating the title, description, etc. • Did have issues concerning the help buttons • Still had concerns about arbitrary processes •  Main roadblock came with the question bank feature • "Why is it that you have to set it up with the question bank?" • How do you edit questions? In the question bank or the quiz itself? • Interface is found to be too "crunched up"

8. Preliminary Results Phone Interviews (1.97): • One concern brought up was how to prevent cheating • Issues with the question bank feature as well • Keeping the students updated (if he/she misses a quiz or if the instructor modifies the quiz) Surveys (1.97) • In general, instructors say that the interface is good for conducting online classes • Most brought up the point of functionality vs. usability • "Yes there was a steep learning curve, but eventually, each time I use it I discover more" • Compare it to Blackboard (has a better quiz/exam feature)

9. Compiled Results Common Usability Issues: • Turn editing on - Why? If needed - Move to a more noticeable area • Drop down Menu's contain too much information • Need more commonly used default values • Question bank Interface - Completely confusing • Needs directions or more intuitive flow • All around - needs more on screen instructions, not just help actions Some users liked the system as is, but still felt it was hard to learn

10. Compiled Results Moodle 2.0 Addresses most usability issues in 1.97 • Turn editing on - Moved to more "eye popping" position • Drop down menu content re-vamped (slightly better) • Adding questions to quizzes improved • Quiz question bank improved • Can add questions directly to quizzes and use in a bank

11. Adding questions in Moodle 1.97

12. Adding a question in Moodle 2.0

13. Moodle 2.0 Question bank

14. Compiled Results • Many options - needed for high user customization - but they appear to complicate the system • Still compiling results • Almost all tests indicate an upgrade is needed • Demo runs of Moodle 2.0 were positive and enthusiastic Preliminary Recommendations: • Leaning towards recommending quiz system be upgraded to newer version of Moodle • Do not feel like customization is needed

15. Updated Schedule • Week Five (04/24 - 04/30) • Meet up with users; run usability tests (all members) • First Phase of the usability tests with the primary group of users (all members) • Create relevant features "list" Moodle 2.0 that might address usability issues. (Kenny)   • Week Six (05/01 - 05/07) • Analyze data received from usability tests (Camille and Kristen) • Proceed with second phase of testing (all members) • Publish web survey and notify all participants to participate(Sanjay). • Week Seven (05/08 - 05/14) • Analyze data received from usability tests (Sanjay and Kenny) • Re-evaluate system(all members).

16. Updated Schedule • Week Eight (05/15 - 05/21) • Contact a user set for suggestions and opinions on redesign (Sanjay, Camille) • Conduct Final remaining interviews (Sanjay, Kenny, Camille) 2. Weeks Nine and Ten (05/22 - 06/04) • Conduct Final remaining interviews (Sanjay, Camille) • Refine and render collected data (Sanjay, Kristen) • Produce Final Report (Team) • Final Presentation (Sanjay, Camille, Kenny)

17. Thank you Questions ?