Evaluations in information retrieval

1. Evaluations in information retrieval

2. Evaluations in information retrieval:summary • The following gives an overview of approaches that are applied to assess the quality of • information retrieval systems, and more concretely of search systems • the resulting set of records obtained after performing a query in an information retrieval system • Note: This should not be confused with assessing the quality and value of the content of an information source.

3. Evaluations in information retrieval:introduction • The quality of the results, the outcome of any search using any retrieval system depends on many components / factors. • These components can be evaluated and modified to increase the quality of the results more or less independently.

4. Evaluations in information retrieval:important factors • The information retrieval system ( = contents + system) • The user of the retrieval system and the search strategy applied to the system Result of a search

5. Evaluations in information retrieval:why? (Part 1) • To study the differences in outcome/results when a component of a retrieval system is changed, such as • the user interface • the retrieval algorithm • addition by the database of uncontrolled, natural language keywords versus keywords selected from a more rigid, controlled vocabulary

6. Evaluations in information retrieval:why? (Part 2) • To study the differences in outcome/results when a search strategy is changed. • To study the differences in outcome/results when searches are performed by different groups of users, such as • children versus adults • inexperienced users versus more experienced, professional information intermediaries/professionals

7. Evaluations in information retrieval: the simple Boolean model Boolean model: # items in database = # items selected + # items not selected # Items selected = # relevant items + # irrelevant items Relevant Yes 1 In Irrelevant No 0 Out

8. Relevant items in a database: scheme Relevant items! (In most cases the small subset) Irrelevant / NOT relevant items (In most cases the large subset) Dependent on the aims, independent of the search strategy

9. Selecting relevant items by searching a database: scheme Dependent on the aims and dependent on the search strategy Selected and relevant! Selectedbut not relevant Not selected but relevant Not selectedand not relevant Dependent on the aims, independent of the search strategy

10. Recall: definition and meaning • Definition: # Of selected relevant items “Recall” = ------------------------------------------------- * 100% Total # of relevant items in database • Aim: high recall • Problem: in most practical cases, the total # of relevant items in a database cannot be measured.

11. Selecting relevant items: recall Selected and relevant! Selectedbut not relevant Not selected but relevant Not selectedand not relevant

12. !? Question !? Task !? Problem !? How can you change your search strategy to increase the recall?

13. Precision: definition and meaning • Definition: # Of selected relevant items“Precision” = --------------------------------------- * 100% Total # of selected items • Aim: high precision

14. Selecting relevant items: precision Selected and relevant! Selectedbut not relevant Not selected but relevant Not selectedand not relevant

15. !? Question !? Task !? Problem !? How can you change your search strategy to increase the precision?

16. !? Question !? Task !? Problem !? When you change your search strategy to increase the precision, which consequence do you expect for the recall, in most cases?

17. Relation between recall and precision of searches Ideal = Impossibleto reach in most systems 100% Recall 0 0 Precision 100% Search (results)

18. !? Question !? Task !? Problem !? Indicate on the figure that a user improves a search.

19. !? Question !? Task !? Problem !? Indicate on the figure that a database producer and / or the retrieval system improves the retrieval quality.

20. !? Question !? Task !? Problem !? Indicate the relation between the recall and precision in a classical information retrieval system in the form of a figure. Indicate in that figure a good and a bad search.

21. Recall and precision should be considered together Examples: • Increase in retrieved number of relevant items may be accompanied by an impractical decrease in precision. • Precision of a search close to 100% may NOT be ideal, because the recall of the search may be too low. Make search / query broader to increase recall ! • Poor (low) precision is more noticeable than bad (low) recall.

22. Evaluation in the case of systems offering relevance ranking • Many modern information retrieval systems offer output with relevance ranking. • This is more complicated than simple Boolean retrieval, and the simple concepts of recall and precision cannot be applied. • To compare retrieval systems or search strategies, decide to consider for comparison a particular number of items ranked highest in each output.This brings us to for instance: “first-20 precision”.

23. !? Question !? Task !? Problem !? Give examples of retrieval systems that offer relevance ranking.