MHCI PSLC Data Shop Project

MHCI PSLC Data Shop Project Final Design Presentation

The Team Sandi Lowe Jason Hum Sam Zaiss Jeff Wong Meghan Myers MHCI PSLC Team | Summer 2005

Outline • Background: PSLC & The Data Shop • The MHCI Project • Use Case Scenario • Prototype Demo • Implementation Timeline • Wrap Up MHCI PSLC Team | Summer 2005

What is the PSLC? PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free- Form Export MHCI PSLC Team | Summer 2005

PSLC Goals • Further current education research • Enable new education research • Support collaboration • Support 7 LearnLab courses MHCI PSLC Team | Summer 2005

What is the Data Shop? PSLC LearnLab Data Shop Collect Process Access Pre- Defined Free- Form Export MHCI PSLC Team | Summer 2005

The MHCI Data Shop Project Access Pre-Defined Free-Form Learning Curves Error Report Problem Profile Help FX Behavior Graph Timeline Viz Session Browser Data Export Export MHCI PSLC Team | Summer 2005

Project Requirements • High-Fidelity Proof-of-Concept Prototype • In-Depth Research Including Weekly User Testing • 13 Contextual Inquiries • 8 Requirements Interviews • 12 Competitive Analyses • 37 User Tests • Deliverables • Current Prototype • Requirements Document • Design Specification • Supporting Data • Design Iterations MHCI PSLC Team | Summer 2005

MHCI Project Timeline Start of Summer Summer Workshop End • Began With Low Fidelity Paper Prototypes • Gradually Added Features & Increased Fidelity • Weekly User Testing Throughout UserTesting Iteration Hi-Fi Prototype Paper Prototypes Mid-Fi Prototypes MHCI PSLC Team | Summer 2005

Project Themes • Context Matters • Facilitate Inter-Report Navigation • Create Specialized Reports • Emphasize Visual Communication MHCI PSLC Team | Summer 2005

ITS Background • Intelligent Tutoring Systems (ITSs) used: • To help students learn • To gain insight into how students learn • Consist of a series of problems in a particular subject. • Order of problems is random • Each problem composed of a number of steps; each of which test one or more knowledge components MHCI PSLC Team | Summer 2005

Meet Dave Jargenson Name: David Jargenson, Ph.D. Age: 35 Affiliation: Research Scientist, Wisconsin Center for Education Research • Interested in how students learn Algebra. • Has been doing education research for 10 years. • The PSLC’s newest member, he has already run a study with the Center. • Trying out the Data Shop with a couple basic studies. MHCI PSLC Team | Summer 2005

Dave’s Data Exploration Determine How Students Are Performing Overall Compare Performance of 2 Samples Select Certain Transactions from the Data Shop For Further Analysis Determine Most Frequent Unexpected Error MHCI PSLC Team | Summer 2005

Error Report • Supporting Data: • Requirements Solicitation (VanLehn, Koedinger, Ritter) • General Research Contextual Inquiries (U3, U8, U10) • LearnLab Research Contextual Inquiries (U11, U12) • Goal: • View all mistakes that students made, by frequency, steps and knowledge components MHCI PSLC Team | Summer 2005

Error Report "Here we are. Errors by classification. Hmm, unanticipated? Oh, I can click it. Okay, the most common one is miles.“ - U30 MHCI PSLC Team | Summer 2005

Learning Curves • Supporting Data: • Requirements Solicitation (Koedinger, Aleven, Ritter) • General Research Contextual Inquiries (U2) • LearnLab Research Contextual Inquiries (U13) • Goal: understand students’ performance over time particular knowledge components MHCI PSLC Team | Summer 2005

Learning Curves/Problem Profiles • “Oh! I don’t need to see the [transaction table], it’s right here in the graph.” (-U28) • “I like being able to see the curves without punching in the formulas.” (-U1) • “I love that the a’s and the b’s come right up.” (-U36) • “I liked the Problem Profile. Leave it as it is.”(-U28) MHCI PSLC Team | Summer 2005

Sample Selector • Supporting Data: • Contextual Inquiry Research: U3, U15 • Requirements Solicitation: Aleven, Koedinger • User Testing Pre-test Questions: U15, U16, U19-U24, U26, U28-U34, U36 • Goal: Define multiple groups of students and compare their performance throughout the standard reports within the Data Shop MHCI PSLC Team | Summer 2005

Sample Selector • "so the only option I have is 'all students.' Ah, but I can edit this list.“ (U27) • “Oh, maybe I need to create a new sample.” (U31) • “making the samples was fairly easy.” (U29) MHCI PSLC Team | Summer 2005

Data Export • Dave’s Goal is to: • Select a subset of his data • Export it to a file for further analysis • Supporting Data: • Requirements Solicitation (Koedinger, VanLehn) • General Research Contextual Inquiries (U5, U8) MHCI PSLC Team | Summer 2005

Data Export Questions MHCI PSLC Team | Summer 2005

Implementation plan Estimated Implementation Time (in Days) MHCI PSLC Team | Summer 2005

Context Matters • Intimate Knowledge of Tutors Required • Dig a little deeper, right away MHCI PSLC Team | Summer 2005

Inter-report Navigation • Reports are useful when they are connected. • Carry context between reports when possible. MHCI PSLC Team | Summer 2005

Specialized Reports • More tailored than a stat package • Eliminates grunt work MHCI PSLC Team | Summer 2005

Use Visual Communication • Get familiar with data quickly • Identify points of interest MHCI PSLC Team | Summer 2005

Evolution of Error Report MHCI PSLC Team | Summer 2005

Evolution of Error Report "I really don't understand what it [error report] means. When Ken was showing it to us [earlier in the day at the Data Shop demo] it made sense but on my own I wasn't sure." –User 13, Summer School MHCI PSLC Team | Summer 2005

Evolution of Error Report "This wasn't very helpful... probably just the layout - it's hard to decipher….this is very difficult to read… I’m not sure what these errors mean." – User 13 MHCI PSLC Team | Summer 2005

Evolution of Error Report MHCI PSLC Team | Summer 2005

Acknowledgements Bob Kraut Andrea Knight Ken Koedinger Kurt Van Lehn Vincent Aleven Shipra Kayan Carolyn Rose Polo Chau Michael Bett Alida Skogsholm Peter Centgraf Braden Kowitz Bonnie John Ben Billings MHCI PSLC Team | Summer 2005

The End “It’s like having my very own grad student!” – User 21 MHCI PSLC Team | Summer 2005

Backup Slide: Data Visualization • Information Visualization (Card, 2003) says that Data Visualization improves cognition in 6 ways: • Increasing the memory and processing resources available to users • Reducing the search for information • Using visual representations to enhance the detection of patterns • Enabling perceptual inference operations • Using perceptual attention mechanisms for monitoring • Encoding information in a manipulable medium MHCI PSLC Team | Summer 2005

Backup Slide - Error Report • Horizontal Stacked Bars • Option to take hints as errors of omission • Allows them to compare “down the line” • Error names fit better horizontally • Visualization provides better performance than tables MHCI PSLC Team | Summer 2005

Problem Profile • Supporting Data: • General Research Contextual Inquiries (U1, U3, U7, U8, U9, U10) • LearnLab Research Contextual Inquiries (U11, U12) • Course Committee Survey (Chem) • Think Aloud Pilot (U2) • Goal: Understand students’ performance on a particular problem, and the problem’s context MHCI PSLC Team | Summer 2005

Multi-Selection(in long lists) • A more standard method of indicating multiple-selection • Highlight helps users quickly spot which items are selected if scrolling the list. “…if there were check boxes on the side I would have known I could select more than one” – (U15) MHCI PSLC Team | Summer 2005

“Scrubbing” A method to quickly compare across knowledge components “Oh interesting. It [the next curve] pops up.“ – (U35) “ooh…that's so cute….I'm going to click on that point to see why it jumped back up.” – (U36) MHCI PSLC Team | Summer 2005

Why Not Just Add Condition? • Within-subject experiments require the capability for students to be assigned to multiple conditions • Sample Selector allows for multiple groupings based on individual researchers’ units of analysis MHCI PSLC Team | Summer 2005

What About Behavior Characteristics? • The Sample Selector can build groups of students based on any characteristic in the database • Student behavior characteristics are not currently fields in the database • Once explicitly defined and included in the database, any behavior characteristic can be added and then used to build samples MHCI PSLC Team | Summer 2005

Student Characteristics MHCI PSLC Team | Summer 2005

Problem Characteristics MHCI PSLC Team | Summer 2005

MHCI PSLC Data Shop Project