REFEREE: An open framework for practical testing of recommender systems using ResearchIndex

1. REFEREE: An open framework for practical testing of recommender systems using ResearchIndex Dan Cosley, Steve Lawrence, David M Pennock Proceedings of the 28th VLDB Conference Hong Kong, China, 2002 Presenter: Raghav Karumur Date: 4/11/2011 Course: [CSCI 8735] Advanced Database Systems Department of Computer Science and Engineering University of Minnesota, Twin Cities Spring 2011

2. Background REcommenderFramework and Evaluator for REsEarchIndex An autonomous citation indexing system that indexes a large fraction of the computer science literature available on the web. • It locates papers, downloads them, and automatically extracts information such as abstracts, authors, and citations. • It provides citation links both to and from a document. • It computes similarity: “users who viewed x also viewed y”. • What is REFEREE? • What is ResearchIndex? Advanced Database Systems Raghav Karumur Spring 2011

3. Motivation Currently we have lot of automated recommender systems!! Advanced Database Systems Raghav Karumur Spring 2011

4. Motivation Problem: No standard, public, real-world test-bed for evaluating the recommendation algorithms. How this paper addresses this issue? This paper built one such test-bed REFEREE using ResearchIndex database so that anyone can develop , deploy or evaluate recommender systems relatively easily and quickly. Why did they choose ResearchIndex? Because it is : (author, title, abstract, fulltext), collaborative filtering information (user behaviors), as well as linkage info(citations). content rich Advanced Database Systems Raghav Karumur Spring 2011

5. Rest Of the Paper: Types of Recommenders A brief history of CF Hybrid systems Why use Research Index? REFEREE PD-Live: An example recommender Results and Discussion Conclusion References Advanced Database Systems Raghav Karumur Spring 2011

6. Types of Recommenders alone uses several recommender systems: • Non-personalized: Ex: “Users who bought Men Are From Mars, Women Are From Venus also bought Space Invaders From Pluto!”. • Users post comments about an item. Customers read the comments before they buy. • Personalized recommendations based on user’s ratings (explicit or implicit). (An example of a pure collaborative filtering (CF) system that computes recommendations based only on the ratings (explicit or implicit) given by users, ignoring product attributes.) As a result, CF systems are completely domain independent. Advanced Database Systems Raghav Karumur Spring 2011

7. Collaborative Filtering • Earlier collaborative filtering systems used to be manual. User had to create filters, know the product, etc. • Automated collaborative filtering addresses this problem, users assign ratings to items. System uses this rating to suggest users with similar interests. • Original algorithms use similarity metrics such as correlation, MS distance, vector similarity, etc or use ML algorithms such as clustering, neural networks, Bayesian networks, etc. Advanced Database Systems Raghav Karumur Spring 2011

8. Collaborative Filtering However, CF systems can fail when • Data is too sparse, • When recommending new items, cold start problem. • or when recommending to new users. More sophisticated recommender systems combine user ratings with domain-specific descriptive information about the items. Advanced Database Systems Raghav Karumur Spring 2011

9. Hybrid Systems • content-based filtering + collaborative filtering • Standard databases such as EachMovie dataset of movie ratings, exist for testing pure CF algorithms. • This is not the case with hybrid systems! • You have to populate it with content information from databases such as IMDB. • Ex: filterbot, P-Tango, PD-Live etc Advanced Database Systems Raghav Karumur Spring 2011

10. Hybrid Systems “Makes recommendations from all 1,000,000 documents in <200msec” • PD-Live is a hybrid recommender they have built in order to evaluate the performance of this testbed. • It uses content based filtering to select the documents and collaborative filtering to rank them. • They claim that PD-live performs reasonably well compared to other recommenders in ResearchIndex. Advanced Database Systems Raghav Karumur Spring 2011

11. Why use ResearchIndex as a testbed? The algorithm designers of this paper have also taken performance into consideration as >1000 users access this database of >1,000,000 papers per hour. Research papers have both objective ("this work is related to mine") and taste ("I think this work is interesting") components, thus can help for both content-based and collaborative filtering. Recommender system developers can install a local copy of ResearchIndex, and use a rich API to access the database. ResearchIndex also has a large, active system, which will allow recommender systems developers to ensure that their algorithms scale well to systems which demand quick response time while servicing thousands of requests per hour, over a database of hundreds of thousands of users and items. Advanced Database Systems Raghav Karumur Spring 2011

12. ResearchIndex: A peek inside ! The basic entities in ResearchIndex: users, documents, citations, and citation groups. Users : People who make queries to access documents and citations. ResearchIndex makes it easy to observe a user’s behavior, automatically generating & assigning user IDs and stores them in cookies. Documents: (research papers) each is assigned a document ID (DID). Citations: The actual text of citations extracted from the documents. Each citation receives a unique citation ID (CID). Citation groups: relate individual citations that cite the same document, with each citation group receiving a group ID (GID). User actions (interaction with a document or citation/issue a query) are called “events”. There are about 30 kinds of events; some of the events most likely to interest recommender systems developers are shown in Table 1. Advanced Database Systems Raghav Karumur Spring 2011

13. ResearchIndex: A peek inside ! Table1: A sample of user actions in ResearchIndex. Most actions involve a user interacting with a document or citation in the database. Advanced Database Systems Raghav Karumur Spring 2011

14. ResearchIndex: A peek inside ! REFEREE has no built-in notion of how important or interesting an event is. Recommender system developers have to infer what an event signifies about a user’s preferences. This is also known as using implicit ratings, as opposed to explicit ratings, where the user effectively says “I like item xthis much”. Implicit ratings are useful in ecommerce recommenders, as it is a burden on users to ask for explicit ratings. ResearchIndex does provide a way for users to give a rating on a 1 to 5 scale! Advanced Database Systems Raghav Karumur Spring 2011

15. ResearchIndex: Built in recommenders Sentence Overlap: recommends docs with significant text identical to current document. Cited By: documents that site current document. Active Bibliography: documents that make similar citations. Users who viewed: documents that userbase has seen. Co-citation: documents sited together with current document. On Same site: documents found on same webpage as current document. Advanced Database Systems Raghav Karumur Spring 2011

16. REFEREE click-through rates and download rates, purchase rates Purposes: • To help recommender systems communicate with ResearchIndex. • To minimize the amount of work recommender system developers must do in order to implement and test working systems. • REFEREE allows anyone to quickly implement, and evaluate recommenders for documents in ResearchIndex. • Want to evaluate recommenders using: user behavior • A Recommender Framework Daemon (RFD) runs on the ResearchIndex site and fetches user events to the recommenders. • It also requests recommendations of the form “user x is looking at document y: what three documents might x want to look at next?” • The system generally issues between one and five recommendation requests per second, and usually requires recommenders to respond within 200 milliseconds. • Figure 2 shows the REFEREE architecture. Advanced Database Systems Raghav Karumur Spring 2011

17. REFEREE: Architecture Logs recs Requests to join, recs upon request Engine RFD events RFD Client Broadcast events, request recs from specific engines recs Events, rec requests Research Index Figure 2: An overview of the REFEREE architecture. Recommender engine writers need only write the Engine portion, making use of the shaded components which REFEREE provides. Advanced Database Systems Raghav Karumur Spring 2011

18. The Process In order to develop a REFEREE-based recommender, a researcher first contacts ResearchIndex for access to the framework, codebase, etc. Then he develops a recommender & tests it locally against his local instance. Once the recommender is ready, the researcher changes the client to point at the actual ResearchIndex site, and it then goes live. The recommender can run on any machine, as long as the network latency plus computation time required to make recommendations is within the deadline which ResearchIndex imposes. Advanced Database Systems Raghav Karumur Spring 2011

19. Metrics and Accuracy How accurate are the metrics? MAE : user ratings Vs system ratings User ratings are often not precise: depend on user’s mood. Also often they are not explicit; system must infer from user behavior. So, do we need absolute prediction accuracy? - In most cases, we don’t! Metrics are there to measure whether user accepted a recommendation. This paper distinguishes between user looking at a document’s detail page and a user actually downloading it. Instead of Click-Through and Buy-Now, they have Click-Soon and Buy Soon. Advanced Database Systems Raghav Karumur Spring 2011

20. What should(can) a recommender system do? Neither of these important cases can be captured by evaluating recommenders using accuracy metrics on a static dataset!! Help the user make a decision about what item to choose. Actually change user behavior. By suggesting a set of items, a recommender provides a user with a set of choices which the user might not otherwise have made. influence which items users consume (and thus can rate) and might even influence the actual ratings users give (“the system recommended it, so it has to be good”).

21. Static Datasets: Of what use are they? Provide a fixed and unchanging world. Then what use???? . . . Valuable tool for evaluating recommender algorithms Helpful in removing unsuitable algorithms Increase effectiveness of online experiments. In any case, ResearchIndex captures raw data: it logs user events, recommendation requests, and documents recommended. Advanced Database Systems Raghav Karumur Spring 2011

22. PD-Live : An example hybrid recommender PD-Live : variant of Personality Diagnosis algorithm of Pennock. Maintains an in-memory sparse-matrix representation of users, items and ratings Remembers its state between runs Computes most recommendations in under 200 milliseconds The authors of this paper chose PD because they had a reasonably fast working offline version, and the algorithm has been shown to be more accurate than standard correlation methods on EachMovie and ResearchIndex data. Advanced Database Systems Raghav Karumur Spring 2011

23. Considering context in hybrid recommenders A user viewing recipes should not be suggested jokes. Most CF recommenders do not consider context explicitly, they draw recommendations from all the items they know about. They get some implicit context from the fact that people tend to like similar types of items (e.g., if Dan likes mostly science fiction movies, a recommender will tend to match him up with other sci-fi fans and recommend science fiction rather than romances). Issues with ResearchIndex: Scalability: Computing a recommendation from among 500,000 items Second, researchers typically have several areas of interest. Documents with the highest predicted scores may not be what the user is currently interested in. Advanced Database Systems Raghav Karumur Spring 2011

24. Two Phase Approach for determining context similarity Citation links and other similarity measures form a directed graph with documents as nodes and similarity relationships as edges. PD-Live does a BFS from the document a user is currently looking at, to select a candidate set of documents. It then uses the PD algorithm to generate predictions for these documents and conducts a lottery biased toward higher-scoring documents to select which ones to recommend. If the user is new, or no neighbors can be found who have rated a document, the system falls back to non-personalized average ratings for the documents. If the document is new and no similarity information is available, the system chooses the set of recommendable documents randomly. Advanced Database Systems Raghav Karumur Spring 2011

25. Two Phase Approach: Advantages • Easy and fast to access and use. • Using content to select a set of plausible documents and then collaborative information to rank them is also a reasonable model of how people often make decisions: eliminate obviously wrong alternatives and then choose the best among them. • This approach fits well with the notion of relevance adopted by the IR community. • Their recommender starts with the impersonal, binary notion of relevance, then uses CF to include personal notions of relevance. Advanced Database Systems Raghav Karumur Spring 2011

26. Two Phase Approach: Drawbacks • Consumes a lot of memory. • It also reduces the number of documents considered by only recommending documents that have at least five ratings, since documents rated by only a few users would seldom be recommended anyway. • User’s session is updated only every 15 minutes. • PD-Live also makes use of only some user actions: those where the user is interacting with a document already stored in the system. Advanced Database Systems Raghav Karumur Spring 2011

27. Results and Discussion Four metrics produce the same rank ordering of the recommenders. The best PD-Live-based recommender places in the middle of the pack, outperforming three of ResearchIndex’s built-in recommenders (On Same Site, Co-citation, and Users Who Viewed) while lagging behind three others (Sentence Overlap, Cited By, and Active Bibliography). The percentages for click-thru do not add up to 100% because there are many other actions that users can take besides following a recommendation link. The Sentence Overlap and Cited By recommenders do much better than the rest. These recommenders often produce documents related to the current document and that are more recent (and often, most detailed). Cited By helps researchers find newer/more papers related to area of interest. Advanced Database Systems Raghav Karumur Spring 2011

28. Results and Discussion • REFEREE uses metrics that evaluate how the recommender system affects user behavior. • PD-Live exploits both content-based (to limit the population of documents to recommend)and collaborative similarity measures (to order them). • PD-Live has great flexibility and corresponds well to the problem of recommending items to a user in context. • The prototype version makes reasonably good recommendations compared to several content-based recommenders, has good performance, and appears to have much potential for improvement. Advanced Database Systems Raghav Karumur Spring 2011

29. Future Research • REFEREE has the potential to stimulate research in at least three areas: • Recommender Systems that merge content and collaborative data • Systems that must make tradeoffs to sacrifice accuracy and elegance for speed and memory savings, and • Systems that recommend based on current user actions (context) Advanced Database Systems Raghav Karumur Spring 2011

30. Advanced Database Systems Raghav Karumur Spring 2011

31. Thank You! Questions? Advanced Database Systems Raghav Karumur Spring 2011