Download
1 / 26
260 likes | 434 Vues
Emulating human essay scoring with machine learning methods Darrell Laham Tom Landauer Peter Foltz Cognitive Systems: Human Cognitive Models in System Design June 30, 2003. Marcia Derr, Ph.D. Scott Dooley Terry Drissell Dave Farnham Peter Foltz, Ph.D. Shawn Frederickson Brent Halsey
E N D
Emulating human essay scoring with machine learning methodsDarrell LahamTom LandauerPeter FoltzCognitive Systems: Human Cognitive Models in System DesignJune 30, 2003
Marcia Derr, Ph.D. • Scott Dooley • Terry Drissell • Dave Farnham • Peter Foltz, Ph.D. • Shawn Frederickson • Brent Halsey • Pat Hilton-Suiter • Darrell Laham, Ph.D. • Tom Landauer, Ph.D • Karen Lochbaum, Ph.D. • Dian Martin • Jeff Nock • Jim Parker • Randy Sparks, Ph.D. • Lynn Streeter, Ph.D
Taxonomy of essay assessment • Writing Assessment Types • Composition (Language Arts) • Does the writer write well? • Exposition (Content Areas, e.g. history) • Does the writer understand the topic? • Levels of Assessment • 1. Holistic Scoring • 2. Trait and Componential Scoring • 3. Annotation • 4. Situated Value Judgments • Which levels are open to automated scoring?
Taxonomy of essay assessment Level 1 Level 2 Level 3 Level 4 Analytics Annotations Language Arts (composition) Local Errors Trait Scores Levels of Assessment Situated Value Judgments Holistic Score Content Areas (exposition) Truth Values Knowledge
Architecture of scoring systems • Intelligent Essay Assessor™ technologies • Latent Semantic Analysis for scoring quality of content and providing tutorial feedback • Style & Mechanics measures for scoring and validation of essay as appropriate for task • Student essays written to directed prompts • Constructed-response alternative to multiple-choice for domain knowledge assessment • Directed essay questions or summaries • Reliable, objective, consistent and immediate • Used as second reader, formative evaluations, diagnostic tutorials, interactive textbooks
Architecture of scoring systems Customized Reader Overall Expert Scored Essays CONTENT % Content Score % Style % Mechanics variance VL Coherence STYLE Confidence PLAGIARISM MECHANICS Misspelled Words Char Count And / Or VALIDATION
Latent Semantic Analysis • LSA is both a psychological theory of knowledge representation and a computational modeling and application tool • LSA learns the relationships between text documents and their constituent words (terms) when trained on large numbers of background texts (thousands to millions) • Each term, document, or new combination of terms (new document) is represented as a point in a high dimensional “Semantic Space” (300-500 dimensions, not 2 or 3) • LSA effectively measures semantic content against prescribed standards of quality based on human judgments • Extensive and varied research shows LSA judgments of similarity agree well with human judgments
Meaning Based Representation LSA is NOT simple co-occurrence Over 99% of word pairs whose similarities are induced never appear together in a context (paragraph) Synonyms are rarely seen in the same context LSA is NOT simple keyword matching LSA operates on the deep level (latent) meaning of words rather than the surface characteristics (exact matches)
What features of LSA are most important? • It is a fully automated model of memory • Training data of same magnitude as human experience • It begins with first-order local associations between a stimulus and other temporally contiguous stimuli • Represents concepts and contexts (episodes) in same way • Conjointly learns about concepts from their natural contexts and contexts from their constituent concepts • No explicit hand coding of rules or features • Induction stage for generalization • High dimensional vector mathematics offer neurologically plausible computations • Not claimed to be a comprehensive model
What features of LSA are ad hoc? • Based on performance in applications, not requirements of cognitive models… • Singular Value Decomposition (SVD) as induction mechanism • Many other candidate algorithms have emerged • SVD can solve (750K X 10M matrix for 300 dimensions on 8 node Beowulf in 20-30 hours) • Emphasis on easily parsable symbol systems, e.g. text • Text is relatively easy to work with compared to visual data • Now applied to other symbol systems, e.g. genetic codes • Text pre-processing specifics • Local log, global entropy weighting • Similarity metrics (Cosine, Euclidean Distance, etc.)
Performance assessment of system • Focus is on quality of content as judged by people rather than on measures of surface features & keywords • Uses background knowledge of domain in assessment in addition to previously scored essays • Measures what students are saying rather than just how well they are saying it • Does best when linked to course student learning materials – providesformative assessment of domain knowledge with tutorial feedback rather than just a simple overall score • Requires fewer training essays (100 vs. 500) • More difficult to ‘coach’ student in ways to receive artificially high score (e.g. “use semi-colons” or say “Thus and Therefore”) • Models do NOT use any count variables (Word count, etc.)
Performance assessment of system MAINFRAMES Mainframes are primarily referred tolarge computers with rapid, advanced processing capabilities that can executeand perform tasks equivalent to many Personal Computers (PCs) machines networked together. It is characterized with high quantity Random Access Memory (RAM), very large secondary storage devices, and high-speed processors to cater for the needs of the computers under its service. Consisting of advanced components, mainframes have the capability of running multiple large applications required by many and most enterprises and organizations.This is one of its advantages. Mainframes are also suitable to cater for those applications (programs) or files that are of very high demand by its users (clients). Examples of such organizations and enterprises using mainframes are online shopping websites such as Ebay, Amazon, and computing-giant Microsoft. MAINFRAMES Mainframes usually are referred those computers with fast, advanced processing capabilities that could perform by itself tasks that may require a lot of Personal Computers (PC) Machines. Usually mainframes would have lots of RAMs, very large secondary storage devices, and very fast processors to cater for the needs of those computers under its service. Due to the advanced components mainframes have, these computers have the capability of running multiple large applications required by most enterprises, which is one of its advantage. Mainframes are also suitable to cater for those applications or files that are of very large demand by its users (clients). Examples of these include the large online shopping websites -i.e. : Ebay, Amazon, Microsoft, etc.
