search engine technology n.
Skip this Video
Loading SlideShow in 5 Seconds..
Search Engine Technology PowerPoint Presentation
Download Presentation
Search Engine Technology

Search Engine Technology

115 Views Download Presentation
Download Presentation

Search Engine Technology

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Search Engine Technology Slides are revised version of the ones taken from

  2. Search Engine Technology Two general paradigms for finding information on Web: • Browsing: From a starting point, navigate through hyperlinks to find desired documents. • Yahoo’s category hierarchy facilitates browsing. • Searching: Submit a query to a search engine to find desired documents. • Many well-known search engines on the Web: AltaVista, Excite, HotBot, Infoseek, Lycos, Google, Northern Light, etc.

  3. Browsing Versus Searching • Category hierarchy is built mostly manually and search engine databases can be created automatically. • Search engines can index much more documents than a category hierarchy. • Browsing is good for finding some desired documents and searching is better for finding a lot of desired documents. • Browsing is more accurate (less junk will be encountered) than searching.

  4. Search Engine A search engine is essentially a text retrieval system for web pages plus a Web interface. So what’s new???

  5. Start on 2/19/01

  6. Some Characteristics of the Web Standard content-based IR Methods may not work • Web pages are • very voluminous and diversified • widely distributed on many servers. • extremely dynamic/volatile. • Web pages have • more structures (extensively tagged). • are extensively linked. • may often have other associated metadata • Web users are • ordinary folks (“dolts”?) without special training • they tend to submit short queries. • There is a very large user community. Use the links and tags and Meta-data! Use the social structure of the web

  7. Overview Discuss how to take the special characteristics of the Web into consideration for building good search engines. Specific Subtopics: • The use of tag information • The use of link information • Robot/Crawling • Clustering/Collaborative Filtering

  8. Use of TAG information

  9. Use of Tag Information (1) • Web pages are mostly HTML documents (for now). • HTML tags allow the author of a web page to • Control the display of page contents on the Web. • Express their emphases on different parts of the page. • HTML tags provide additional information about the contents of a web page. • Can we make use of the tag information to improve the effectiveness of a search engine?

  10. Use of Tag Information (2) Two main ideas of using tags: • Associate different importance to term occurrences in different tags. • Use anchor text to index referenced documents. Page 2: Page 1 . . . . . . airplane ticket and hotel . . . . . .

  11. Use of Tag Information (3) Many search engines are using tags to improve retrieval effectiveness. • Associating different importance to term occurrences is used in Altavista, HotBot, Yahoo, Lycos, LASER, SIBRIS. • WWWW and Google use terms in anchor tags to index a referenced page. • Shortcomings • very few tags are considered • relative importance of tags not studied • lacks rigorous performance study

  12. Use of Tag Information (4) The Webor Method (Cutler 97, Cutler 99) • Partition HTML tags into six ordered classes: • title, header, list, strong, anchor, plain • Extend the term frequency value of a term in a document into a term frequency vector (TFV). Suppose term t appears in the ith class tfi times, i = 1..6. Then TFV = (tf1, tf2, tf3, tf4, tf5, tf6). Example: If for page p, term “binghamton” appears 1 time in the title, 2 times in the headers and 8 times in the anchors of hyperlinks pointing to p, then for this term in p: TFV = (1, 2, 0, 0, 8, 0).

  13. Use of Tag Information (5) The Webor Method (Continued) • Assign different importance values to term occurrences in different classes. Let civi be the importance value assigned to the ith class. We have CIV = (civ1, civ2, civ3, civ4, civ5, civ6) • Extend the tf term weighting scheme • tfw = TFV  CIV = tf1civ1 + … + tf6 civ6 When CIV = (1, 1, 1, 1, 0, 1), the new tfw becomes the tfw in traditional text retrieval. How to find Optimal CIV?

  14. Use of Tag Information (6) The Webor Method (Continued) Challenge: How to find the (optimal) CIV = (civ1, civ2, civ3, civ4, civ5, civ6) such that the retrieval performance can be improved the most? One Solution: Find the optimal CIV experimentally using a hill-climbing search in the space of CIV Details Skipped

  15. Use of LINK information

  16. Use of Link Information (1) Hyperlinks among web pages provide new document retrieval opportunities. Selected Examples: • Anchor texts can be used to index a referenced page (e.g., Webor, WWWW, Google). • The ranking score (similarity) of a page with a query can be spread to its neighboring pages. • Links can be used to compute the importance of web pages based on citation analysis. • Links can be combined with a regular query to find authoritative pages on a given topic.

  17. Connection to Citation Analysis • Mirror mirror on the wall, who is the biggest Computer Scientist of them all? • The guy who wrote the most papers • That are considered important by most people • By citing them in their own papers • “Science Citation Index” • Should I write survey papers or original papers? Infometrics; Bibliometrics

  18. What Citation Index says About Rao’s papers

  19. Desiderata for ranking • A page that is referenced by lot of important pages (has more back links) is more important • A page referenced by a single important page may be more important than that referenced by five unimportant pages • A page that references a lot of important pages is also important • “Importance” can be propagated • Your importance is the weighted sum of the importance conferred on you by the pages that refer to you • The importance you confer on a page may be proportional to how many other pages you refer to (cite) • (Also what you say about them when you cite them!) Different Notions of importance

  20. Authority and Hub Pages (1) The basic idea: • A page is a good authoritative page with respect to a given query if it is referenced (i.e., pointed to) by many (good hub) pages that are related to the query. • A page is a good hub page with respect to a given query if it points to many good authoritative pages with respect to the query. • Good authoritative pages (authorities) and good hub pages (hubs) reinforce each other.

  21. Authority and Hub Pages (2) • Authorities and hubs related to the same query tend to form a bipartite subgraph of the web graph. • A web page can be a good authority and a good hub. hubs authorities

  22. Authority and Hub Pages (3) Main steps of the algorithm for finding good authorities and hubs related to a query q. • Submit q to a regular similarity-based search engine. Let S be the set of top n pages returned by the search engine. (S is called the root set and n is often in the low hundreds). • Expand S into a large set T (base set): • Add pages that are pointed to by any page in S. • Add pages that point to any page in S. If a page has too many parent pages, only the first k parent pages will be used for some k.

  23. Authority and Hub Pages (4) 3. Find the subgraph SG of the web graph that is induced by T. T S

  24. Authority and Hub Pages (5) Steps 2 and 3 can be made easy by storing the link structure of the Web in advance. Link structure table parent_url child_url url1 url2 url1 url3

  25. B USER(41): aaa ;;an adjacency matrix #2A((0 0 1) (0 0 1) (1 0 0)) USER(42): x ;;an initial vector #2A((1) (2) (3)) USER(43): (apower-iteration aaa x 2) ;;authority computation—two iterations [1] USER(44): (apower-iterate aaa x 3) ;;after three iterations #2A((0.041630544) (0.0) (0.99913305)) [1] USER(45): (apower-iterate aaa x 15) ;;after 15 iterations #2A((1.0172524e-5) (0.0) (1.0)) [1] USER(46): (power-iterate aaa x 5) ;;hub computation 5 iterations #2A((0.70641726) (0.70641726) (0.04415108)) [1] USER(47): (power-iterate aaa x 15) ;;15 iterations #2A((0.7071068) (0.7071068) (4.3158376e-5)) [1] USER(48): Y ;; a new initial vector #2A((89) (25) (2)) [1] USER(49): (power-iterate aaa Y 15) ;;Magic… same answer after 15 iter #2A((0.7071068) (0.7071068) (7.571644e-7)) A C

  26. Start of 2/21 lecture

  27. Authority and Hub Pages (6) • Compute the authority score and hub score of each web page in T based on the subgraph SG(V, E). Given a page p, let a(p) be the authority score of p h(p) be the hub score of p (p, q) be a directed edge in E from p to q. Two basic operations: • Operation I: Update each a(p) as the sum of all the hub scores of web pages that point to p. • Operation O: Update each h(p) as the sum of all the authority scores of web pages pointed to by p.

  28. Authority and Hub Pages (7) q1 Operation I: for each page p: a(p) =  h(q) q: (q, p)E Operation O: for each page p: h(p) =  a(q) q: (p, q)E q2 p q3 q1 p q2 q3

  29. Authority and Hub Pages (8) Matrix representation of operations I and O. Let A be the adjacency matrix of SG: entry (p, q) is 1 if p has a link to q, else the entry is 0. Let AT be the transpose of A. Let hi be vector of hub scores after i iterations. Let ai be the vector of authority scores after i iterations. Operation I: ai = AT hi-1 Operation O: hi = A ai

  30. Authority and Hub Pages (9) After each iteration of applying Operations I and O, normalize all authority and hub scores. Repeat until the scores for each page converge (the convergence is guaranteed). 5. Sort pages in descending authority scores. 6. Display the top authority pages.

  31. Authority and Hub Pages (10) Algorithm (summary) submit q to a search engine to obtain the root set S; expand S into the base set T; obtain the induced subgraph SG(V, E) using T; initialize a(p) = h(p) = 1 for all p in V; for each p in V until the scores converge { apply Operation I; apply Operation O; normalize a(p) and h(p); } return pages with top authority scores;

  32. Authority and Hub Pages (11) q1 Example: Initialize all scores to 1. 1st Iteration: I operation: a(q1) = 1, a(q2) = a(q3) = 0, a(p1) = 3, a(p2) = 2 O operation: h(q1) = 5, h(q2) = 3, h(q3) = 5, h(p1) = 1, h(p2) = 0 Normalization: a(q1) = 0.267, a(q2) = a(q3) = 0, a(p1) = 0.802, a(p2) = 0.535, h(q1) = 0.645, h(q2) = 0.387, h(q3) = 0.645, h(p1) = 0.129, h(p2) = 0 p1 q2 p2 q3

  33. Authority and Hub Pages (12) After 2 Iterations: a(q1) = 0.061, a(q2) = a(q3) = 0, a(p1) = 0.791, a(p2) = 0.609, h(q1) = 0.656, h(q2) = 0.371, h(q3) = 0.656, h(p1) = 0.029, h(p2) = 0 After 5 Iterations: a(q1) = a(q2) = a(q3) = 0, a(p1) = 0.788, a(p2) = 0.615 h(q1) = 0.657, h(q2) = 0.369, h(q3) = 0.657, h(p1) = h(p2) = 0 q1 p1 q2 p2 q3

  34. x x2 xk (why) Does the procedure converge? As we multiply repeatedly with M, the component of x in the direction of principal eigen vector gets stretched wrt to other directions.. So we converge finally to the direction of principal eigenvector

  35. What about non-principal eigen vectors? • Principal eigen vector gives the authorities (and hubs) • What do the other ones do? • They may be able to show the clustering in the documents (see page 23 in Kleinberg paper) • The clusters are found by looking at the positive and negative ends of the secondary eigen vectors (ppl vector has only +ve end…)

  36. Authority and Hub Pages (13) Should all links be equally treated? Two considerations: • Some links may be more meaningful/important than other links. • Web site creators may trick the system to make their pages more authoritative by adding dummy pages pointing to their cover pages (spamming). Domain name: the first level of the URL of a page. Example: domain name for “” is “”.

  37. Authority and Hub Pages (14) • Transverse link: links between pages with different domain names. • Intrinsic link: links between pages with the same domain name. • Transverse links are more important than intrinsic links. Two ways to incorporate this: • Use only transverse links and discard intrinsic links. • Give lower weights to intrinsic links.

  38. Authority and Hub Pages (15) How to give lower weights to intrinsic links? In adjacency matrix A, entry (p, q) should be assigned as follows: • If p has a transverse link to q, the entry is 1. • If p has an intrinsic link to q, the entry is c, where 0 < c < 1. • If p has no link to q, the entry is 0.

  39. Authority and Hub Pages (16) For a given link (p, q), let V(p, q) be the vicinity (e.g.,  50 characters) of the link. • If V(p, q) contains terms in the user query (topic), then the link should be more useful for identifying authoritative pages. • To incorporate this: In adjacency matrix A, make the weight associated with link (p, q) to be 1+n(p, q), where n(p, q) is the number of terms in V(p, q) that appear in the query.

  40. Authority and Hub Pages (17) Sample experiments: • Rank based on large in-degree (or backlinks) query: game Rank in-degree URL 1 13 2 12 3 12 4 12 gamelink/gamelink.html 5 11 6 11 • Only pages 1, 2 and 4 are authoritative game pages.

  41. Authority and Hub Pages (18) Sample experiments (continued) • Rank based on large authority score. query: game Rank Authority URL 1 0.613 2 0.390 http://ad/doubleclick/net/jump/ 3 0.342 4 0.324 5 0.324 6 0.306 • All pages are authoritative game pages.

  42. Authority and Hub Pages (19) Sample experiments (continued) • Rank based on large authority score. query: free email Rank Authority URL 1 0.525 2 0.345 3 0.309 4 0.261 5 0.254 6 0.246 • All pages are authoritative free email pages.

  43. Cora thinks Rao is Authoritative on Planning Citeseer has him down at 90th position…  How come??? --Planning has two clusters --Planning & reinforcement learning --Deterministic planning --The first is a bigger cluster --Rao is big in the second cluster 

  44. Start of 2/26

  45. Announcements • Project task 2 given • Project task 1 progress report due on this Friday • Questions??

  46. ad page obscure web page Authority and Hub Pages (20) For a given query, the induced subgraph may have multiple dense bipartite communities due to: • multiple meanings of query terms • multiple web communities related to the query

  47. Authority and Hub Pages (21) Multiple Communities (continued) • If a page is not in a community, then it is unlikely to have a high authority score even when it has many backlinks. Example: Suppose initially all hub and authority scores are 1. q’s p q’s p’s G1: G2: 1st iteration for G1: a(q) = 0, a(p) = 5, h(q) = 5, h(p) = 0 1st iteration for G2: a(q) = 0, a(p) = 3, h(q) = 9, h(p) = 0

  48. Authority and Hub Pages (22) Example (continued): 1st normalization (suppose normalization factors H1 for hubs and A1 for authorities): for pages in G1: a(q) = 0, a(p) = 5/A1, h(q) = 5/H1, h(p) = 0 for pages in G2: a(q) = 0, a(p) = 3/A1, h(q) = 9/H1, a(p) = 0 After the nth iteration (suppose Hn and An are the normalization factors respectively): for pages in G1: a(p) = 5n / (H1…Hn-1An) ---- a for pages in G2: a(p) = 3*9n-1 /(H1…Hn-1An) ---- b Note that a/b approaches 0 when n is sufficiently large, that is, a is much much smaller than b.