EISS - Usability field study

1. EISS - Usability field study Suzanne Kabel Roeland Stoter EISS - Usability field study

2. TOC • Today • Evaluation • Context • Demo • Next week • Carry out field study, 13.00 room 9.01 EISS - Usability field study

3. Intro • Applied evaluation of an indexing-vocabulary • In Kabel’s evaluation framework: • Keystroke • Process • System image • Mental model • System content • Efficacy EISS - Usability field study

4. Goal of the evaluation • To determine to what extent the use of indexes can be standardized with ontologies, and to what extent people use and apply indexes in the same fashion. EISS - Usability field study

5. Indexing • What is it? Describing information or metadata about a certain domain. E.g. library of scientific articles – writer, date, topic, keywords, etc. • Of what? Of elements or fragments of content on the Internet or in a database. • Why? To re-use and share information. • How? By using standardized indexing vocabularies expressed in XML. EISS - Usability field study

6. Markup languages • Technical implementation of metadata • generic coding movement (late 60’s) separate document content from format (IBM, publishing) • SGML (standard, around 1985) • HTML fixed tag-set to describe layout • XML user defined tag-set to describe content EISS - Usability field study

7. Real Estate Music Transportation Financial Research Farming & Agriculture Bioinformatics Accounting XML vocabularies in practice EISS - Usability field study

8. XML vocabularies in Education • IMS Global Learning Consortium’s Metadata specification imsproject.org (US) • ARIADNE’s Learning Object Metadata ariadne.unil.ch (Europe) • ADL’s SCORM adlnet.org (Defense) • IMAT European project imat.swi.psy.uva.nl (UvA and others) EISS - Usability field study

9. Indexing voc. and ontologies • Merge of SCORM and IMAT • In IMAT: indexing with ontologies • An ontology is an explicit specification of a conceptualization • Here: A fragment is a frame with slots that need to be filled. An ontology serves as a fixed vocabulary to fill the slots. EISS - Usability field study

10. Design of the evaluation • Domain: JAVA • Target use: • Build database of indexed JAVA fragments • Compose lesson material from JAVA fragments • Target group: JAVA teachers • Domain knowledge (JAVA and instruction) • Knowledge of computers • Experience with efficacy and process features of the of the tool: the indexing vocabulary EISS - Usability field study

11. Design of the evaluation (2) • 25 subjects (you) conduct the indexing task • Index JAVA fragments with the indexing tool • Answer questionnaire • Analyze results: Compare indexes • Define percentage correspondence • Categorize differences • Overlapping concepts • Gap in the vocabulary • Incomplete concept definitions • … EISS - Usability field study

12. Fragment 3 Correspondence Difference Category: Educational 85 % 15 % Category: Content 78% 22 % Category: Domain 80 % 20 % Example analysis e.g. learning context 6 %, due to incomplete indexing vocabulary Next step: adapt indexing vocabulary EISS - Usability field study

13. Example JAVA fragment EISS - Usability field study

14. Example JAVA fragment if (Character.isUpperCase(aChar)) { System.out.println("The character " + aChar + " is upper case."); } else { System.out.println("The character " + aChar + " is lower case."); } System.out.println("The value of aBoolean is " + aBoolean); EISS - Usability field study

15. Example JAVA fragment The direct manipulation of an object's variables by other objects and classes is discouraged because it's possible to set the variables to values that don't make sense. For example, consider the Rectangle class from the previous section. Using that class, you can create a rectangle whose width and height are negative, which, for some applications, doesn't make sense. Ideally, instead of allowing direct manipulation of variables, a class would provide methods through which other objects can inspect or change variables. These methods ensure that the values of the variables make sense for objects of that type. Thus, the Rectangle class would provide methods called setWidth, setHeight, getWidth, and getHeight for setting and getting the width and the height. The methods for setting the variables would report an error if the caller tried to set the width or the height to a negative number. The other advantage of using methods instead of direct variable access is that the class can change the type and the names of the variables it uses for storing the width and the height without affecting its clients. EISS - Usability field study

16. Example index • Learning resource type : narrative text • Learning context : University • Difficulty : medium • Instructional role : explanation • Domain topic : JAVA Class EISS - Usability field study

17. Indexing vocabulary • General aspects, applicable to the complete set of JAVA fragments • General category • Lifecycle category • Content, educational and domain aspects, applicable to each individual JAVA fragment • Content category • Educational category • Domain classification category EISS - Usability field study

18. General category This category groups the general information that describes this resource as a whole. Apply this category to the complete set of fragments. • Title Name given to this resource. • Description A textual description of the content of this resource being described. EISS - Usability field study

19. Lifecycle category This category describes the history and current state of this resource and those who have affected this resource during its evolution. Apply this category to the complete set of fragments. • Contribute This sub-category describes those people or organizations that have affected the state of this resource during its evolution (includes creation, edits and publication). • Role Kind of contribution. (Predefined vocabulary) • Centity The identification of and information about the people or organizations contributing to this resource, most relevant first. • Date This sub-category defines the date of the contribution. EISS - Usability field study

20. Content category This category describes the characteristics of the content of this resource. Apply this category to each fragment individually. All predefined vocabularies. • Learning resource type Specific kind of resource, most dominant kind first. • Semantic density This element defines a subjective measure of this resource's usefulness as compared to its size or duration. • Description type The perspective from which the resource was originally written. • Description scope The nature of the resources size. • Knowledge type The kind of knowledge involved in the resource. EISS - Usability field study

21. Educational category This category describes the key educational or pedagogic characteristics of this resource. Apply this category to each fragment individually. All predefined vocabularies. • Interactivity type The flow of interaction between this resource and the intended user. • Interactivity level This element shall define the degree of interactivity between the end user and this resource. • Learning context The principal environment within which the learning and use of this resource is intended to take place. • Difficulty This element defines how hard it is to work through this resource for the typical target audience. • Instructional role The way the resource is used in instruction or instructional material. EISS - Usability field study

22. Domain classification category This category describes a taxonomic path in a specific classification system. Each succeeding level is a refinement in the definition of the higher level. • Is-a : Leaf-node in the JAVA Is-a ontology • Part-of : Leaf-node in the JAVA Part-of ontology EISS - Usability field study

23. Demo … EISS - Usability field study