1 / 22

Search Engines

Search Engines. IST 516 Fall 2011 Dongwon Lee, Ph.D. Search Engine Overview. Search Engine typically consists of: Crawler : that crawls the web to identify interesting URLs to fetch Fetcher : fetches web documents from the stored URLs

phillipe
Télécharger la présentation

Search Engines

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Search Engines IST 516 Fall 2011 Dongwon Lee, Ph.D.

  2. Search Engine Overview • Search Engine typically consists of: • Crawler: that crawls the web to identify interesting URLs to fetch • Fetcher: fetches web documents from the stored URLs • Indexer: build local indexes (eg, inverted index) from the fetched web documents • Query Handler: processes users’ queries using indexes and prepare answers accordingly • Presenter: user interface component to present answers to users

  3. 1. Crawler (more later) • Called robot or spider too • View Web as a graph • Employee different graph search algorithm • Depth-first • Breadth-first • Frontier • Objectives: • Completeness • Freshness • Resource maximization

  4. 2. Fetcher • Crawler generates a stack of URLs to visit • Fetcher retrieves web documents for specific URL • Typically multi-threaded

  5. 3. Indexer • To handle large-scale data of the Web, needs to build index structure • Index is a small (memory-resident) data structure to help locate data fast (at the cost of extra space) • Trading space for time • In Search Engine or IR, popular form of index is called the Inverted Index

  6. Inverted Index • A list for every word (index term). • The list of term t holds the locations (documents + offsets) where t appeared. 6

  7. 4. Query Handler • Given a query keyword Q, which web documents are the right answers? • Eg, • Boolean-matching model • Return all documents that contain Q • Vector space model • Return a ranked list of documents that have the largest cosine() similarity to Q • PageRank model • Return a ranked list of documents that have the highest PageRank values, in addition to other matching model

  8. Boolean-Matching

  9. Vector Space Model • Cosine similarity: similarity of two same-length vectors measured by the cosine angle between them • Range: -1 -- +1 • Dot product: V1 . V2 • Magnitude: || V1 ||

  10. Vector Space Model • Given N web documents gathered, extract all significant token set (ie words), say T. • |T| becomes the dimension of the vector • Convert each web document w_i to |T|-length boolean vector, say V_w_i • Given a query string Q, convert it to |T|-length boolean vector, say V_q • Compute cosine similarity btw V_q and V_w_i • Sort similarity scores in descending order

  11. Vector Space Model Example • 3 documents • D1 = {penn, state, football} • D2 = {state, gov} • D3 = {psu, football} • Vector space representation • V = {football, gov, penn, psu, state} • D1 = [1, 0, 1, 0, 1] • D2 = [0, 1, 0, 0, 1] • D3 = [1, 0, 0, 1, 0] • Query Q = {state, football} • Q1 = [1, 0, 0, 0, 1] • Which doc is the closest to Q?

  12. Term-Weighting • Instead of Boolean vector-space model, each term in dimension carries an importance weight of the corresponding token • [1, 0, 0, 1, 0 ]  [0.9, 0.12, 0.14, 0.89, 0.13]

  13. Term-Weighting • Eg, In the tf-idf, the importance of a term t • Increases proportionally to # of times t appears in the document  tf • Is offset by # of times t appears in corpus  idf

  14. PageRank • Link graph of the web • Prioritizes the results of a search • d: damping factor (eg, 0.85) • C(Ti): # of outgoing links of the page Ti • High PageRank • Many pages pointing to it • Some pages pointing to it have high PageRank

  15. PageRank Eg, PR(A)=(1-d)+d(PR(T1)/C(T1)+…+(PR(Tn)/C(Tn)) = PR(A)=(1-0.85) +0.85(8/1) +0.85(4/2) +0.85(1/4) +0.85(2/3)= 9.3725 8 4 ? 1 2

  16. Effect of Link Structure (1) • Example for simple PR • Initial RPs of pages A, B, and C are all 0.15 Page A = 0.15 Page A = 1 Page A = 1.85 Page A = 1.4594 Page B = 0.2775 Page B = 1 Page B = 0.575 Page B = 1 Page C = 0.15 Page C = 1 Page C = 0.575 Page C = 0.575 Page A Page B Page A Page B Page A Page B Page A Page B Page C Page C Page C Page C

  17. Effect of Link Structure (2) • Example of practical PR • rank sink : no outbound links, PR0 Page A Page B Page A Page B Page A : 0.6277 Page B : 0.6836 Page C : 0.4405 Page D : 1.5735 Page E : 1.6747 Page A : 1.1922 Page B : 1.1634 Page C : 0.6444 Page D : 0.1500 Page E : 0.1500 Page C Page C Page D Page E Page D Page E Simple PR Practical PR Sink

  18. 5. Presenter • Different presentation models • Simple keyword vs. Advanced interface

  19. 5. Presenter • Different presentation models • Ranked list: Google, Bing vs. Clustered list: Yippy Clusters

  20. Evaluation of Results • The deciding factor for a search engine is it’s effectiveness. • Two factors: • Precision — The percentage of the relevant documents returned that are actually about the particular query. • Recall — The percentage of the relevant documents that were actually returned from the available set of documents for a particular query. 20

  21. Evaluation of Results (cont.) • T: True-Relevant documents • R: Retrieved documents • Precision = (T intersect R) / R • Recall = (T intersect R) / T • F-measure = 2 X Precision X Recall / (Precision + Recall) • P-R graph P R 21

  22. A Lot More Details Later • Through the second half of the semester, we will review • Each component of search engines and • Principles behind it in more details • We will use materials from the IIR textbook  • Contents are freely available at: • http://nlp.stanford.edu/IR-book/html/htmledition/irbook.html

More Related