1 / 22

Matthieu Hermet, Stan Szpakowicz

Matthieu Hermet, Stan Szpakowicz. Automated Analysis of Students ’ Free-text Answers for Computer-Assisted Assessment University of Ottawa, Canada. CAA for CALL. To address the specificity of CALL… → where student material contains syntactic and orthographic errors

violet
Télécharger la présentation

Matthieu Hermet, Stan Szpakowicz

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Matthieu Hermet, Stan Szpakowicz Automated Analysis of Students’ Free-text Answers for Computer-Assisted Assessment University of Ottawa, Canada

  2. CAA for CALL • To address the specificity of CALL… → where student material contains syntactic and orthographic errors • …with minimal pre-encoded material : • Content validation : simple • Form validation : difficult → A good case for automating based on Natural Language Processing

  3. Text comprehension • The uOttawa project: CALL solutions for helping French-as-a-Second-Language students to enhance autonomous reading comprehension → master the structure of text in order to understand the author’s discursive intention → guess the meaning of unknown words → develop reformulation and synthesis capabilities

  4. DidaLect …is a FSL tool aimed at teaching autonomous reading skill (designed for intermediate- and advanced-level students) • Is an instance of eLearning Intelligent Tutoring System: • adaptation to individual student’s skills and agenda • access to external resources (dictionaries) • built to reflect the cognitive concerns such as matching feedback to the student’s behaviour

  5. Intelligence in DidaLect • DidaLect begins its operation with a placement test to determine a student’s initial level: • varying order of questions to pick up the best of a student’s skill • the implementation includes fuzzy logic methods • A separate element of DidaLect is the processing of free-text answers: • need of a robust CAA component • a trade-off between symbolic processing and Machine Learning techniques

  6. Free-text answer assessment • The problem is to know in advance what material to expect in student answers. • Usually implemented as a classification problem: a student answer must match reference answer(s). → Size and form of reference material affects the process • Here, a reference answer is the text itself → A case for trying symbolic processing using techniques of Computational Linguistics

  7. Expected limitations • No possibility of modifying the size and form of the reference material, except by automatic processing to control reformulation. • Therefore, this only works for limited forms of questions. • Strong need to ground selection of questions in a firm didactic theory. • Questions on texts for Text Comprehension (didactics offers a classification of question types).

  8. Text-Explicit: based on a single sentence « d'habitude, l'hermaphrodisme frappe surtout les mâles, qu'elle dote de simulacres d'appareils génitaux féminins. » Q : Qu’arrive-t-il aux ours mâles lorsqu’ils sont frappés d’hermaphrodisme ? Ex R : « Ils ont les génitaux féminins » Text-Implicit : based on two co-referenced sentences « le détecteur de décélération situé à l'avant du véhicule génère instantanément un courant électrique, qui déclenche une amorce, qui elle-même enflamme un mélange allumeur. Ce dernier met finalement le feu à l'agent propulseur responsable du gonflement du coussin. » Q : Quelle est la réaction en chaîne qui se produit lorsque survient un impact ? Question types 1

  9. Question types 2 • Identification, cause-effect, goal, comparison, definition, instrumental… • These categories express linguistically through lexical connectors • Goal : for, so that, in order to… • Cause–Effect : because, therefore… • So, the control of reformulation can be automated

  10. Processing • Find lexical differences between the student’s answer S and reference R • Parse S and R, produce dependency relations • Process different words (using a dictionary) to detect synonyms • Control of syntax in S • Control of reformulation in S wrt R

  11. Tools • A robust parser that enables partial recovery from errors in student’s answer • A dictionary of synonyms • A derivational dictionary • Locally derived resources: • State and action verbs • Ensemble of typical errors, set of syntactic and reformulation structures

  12. Semantics and synonyms • Examine word set differences and commonalities in search for: • Common words • Reformulated words • Different words • Detect synonyms accross parts of speech: • Derive forms for a word lemma • Search synonyms for each form and look for a match in Word Sets

  13. Syntax and reformulation • Correct syntactical structures to verify syntax of student’s answer • Lexicalized reformulation structures to verify discourse conformity Ex : pollution has increased with the rise of transportation Q : Why has pollution increased ? Ans : With the rise of transportation is partially wrong → Because of the rise of transportation OR it has increased due to the rise of transportation ETC.

  14. Parsing and tree-building S : Le cardio-vasculaire d'un rat s'approche à une personne humain. SUBJ(<approche^approcher:53>, <cardio-vasculaire^cardio-vasculaire:48>) OBJ(<approche^approcher:53>, <personne^personne:56>) VMOD_POSIT1(<approche^approcher:53>, <une^un:55>) NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>, <rat^rat:51>) PREPOBJ(<une^un:55>, <à^à:54>) PREPOBJ(<rat^rat:51>, <d'^de:49>) DETERM(<rat^rat:51>, <un^un:50>) The above are incrementally recomposed, based on lexical selection that maximizes promise of discovering material which diverges from R. That material is processed in parallel, in a similar fashion: SUBJ( <OBJ(<approcher>, <NMOD(<être>, <humain>)>)>, <NMOD(<NMOD(<système>, <cardio-vasculaire>)>, <rat>)>)

  15. Main types of heuristics • To address syntactic correctness and/or equivalence between S and R: the same sense but different structures → bank of typical errors and correct structures • To address discursive variations, detected as supplementary material → bank of state and action verbs: action verbs must be present, possibly reformulated • To address, partially, errors in S → word replacement to relaunch parsing when stopped due to lexical mistakes

  16. Reformulation rules • Examples : • Abstraction : incidence sur le temps de gestation → incidence sur la possibilité d’avoir une gestation écourtée (words like fact, chance, etc.) • Cause-Effect : Le plasma augmente et dilue les paramètres chimiques → L’augmentation du plasma dilue les paramètres chimiques • Is-A : Le rat est un animal qui + S → Le rat + S • Attribute : Le rat possède un système cardio-vasculaire → Le système C-V du rat

  17. Assessment • Must give student feedback on: • Agreement and orthography • Syntax: signal errors and provides correction via display of a correct structure • Semantics: signals error and provides admissible words • Completeness of content with respect to R

  18. R: Et puisque le rat est un animal qui possède un système cardio-vasculaire très semblable à celui de l’humain, il est donc permis de tirer les mêmes conclusions pour l’humain. Q: Pourquoi peut-on tirer les mêmes conclusions pour l'humain et pour le rat ? S: Le cardio-vasculaire d’un rat s’approche à une personne humain. Start by creating wordlists : Words of S absent in R → s’approcher, personne Words of R absent in S → animal, posséder, système, semblable Common words → rat, cardio-vasculaire, humain Example

  19. S: SUBJ(<approche^approcher:53>, <cardio-vasculaire^cardio-vasculaire:48>) OBJ(<approche^approcher:53>, <personne^personne:56>) VMOD_POSIT1(<approche^approcher:53>, <une^un:55>) NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>, <rat^rat:51>) PREPOBJ(<une^un:55>, <à^à:54>) PREPOBJ(<rat^rat:51>, <d'^de:49>) DETERM(<rat^rat:51>, <un^un:50>) R: SUBJ(<est^être:2>,<rat^rat:1>) OBJ_SPRED(<est^être:2>, <animal^animal:4>) OBJ(<possède^posséder:6>, <système^système:8>) COREF_REL(<animal^animal:4>, <qui^qui:5>) NMOD_POSIT1(<système^système:8>, <cardio-vasculaire^cardio-vasculaire:9>) NMOD_POSIT1(<système^système:8>, <semblable^semblable:11>) NMOD_POSIT1(<celui^celui:13>, <humain^humain:16>) ADJMOD(<semblable^semblable:11>, <celui^celui:13>) PREPOBJ(<humain^humain:16>, <de^de:14>) PREPOBJ(<celui^celui:13>, <à^à:12>) DETERM(<système^système:8>, <un^un:7>) DETERM(<animal^animal:4>, <un^un:3>) DETERM_DEF(<rat^rat:1>, <le^le:0>) CONNECT_REL(<possède^posséder:6>, <qui^qui:5>) Parse (partial output)

  20. Comparison (partial) SUBJ(< OBJ(<approche^approcher:53>,<personne^personne:56>) >,<NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>,<rat^rat:51>)>) SUBJ(<OBJ_SPRED(<est^être:2>,< COREF_REL(<animal^animal:4>,< CONNECT_REL(< OBJ(<possède^posséder:6>,< NMOD_POSIT1(< NMOD_POSIT1(<système^système:8>,<cardio-vasculaire^cardio-vasculaire:9>) >,<ADJMOD(<semblable^semblable:11>,< NMOD_POSIT1(<celui^celui:13>,<humain^humain:16>) >)>)>) >,<qui^qui:5>) >) >)>,<rat^rat:1>) • Consider structures to assess for syntactic correctness • Heuristics to put some structures into equivalence : → Here, «Le SCV du rat» and «Le rat est un animal qui possède un SCV» are equivalent, but expressed in different syntactic structures

  21. - <RESULTS> - <DEF> - <E L="fr"> <W>semblable</W> <SW ENC="n">00730065006D0062006C00610062006C0065</SW> <P>adj.</P> </E> - <DF N="1"> <W>qui ressemble à; comparable, similaire.</W> </DF> </DEF> - <DEF> ….. - <WD L="fr" W="2"> <W L="fr">approchant</W> <SW ENC="n">0061007000700072006F006300680061006E0074</SW> </WD> To retrieve a synonymy relation : Produce derivations for all words in List 1 Find matches in a synonymy basis under entries for words of List 2 The search process can be repeated at most once, using <DEF> lexemes → semblable = approchant OR → ressemblerà = s’approcher de 4. In this way we can catch both synonyms and attached prepositions. Synonyms

  22. Conclusions and future work • Automation is possible, with 2 main restrictions : • « bad faith » answers • Lexical errors based on homonymy → as long as S contains elements of answer, S can be evaluated • Future Work : to assemble the parts through software engineering !

More Related