1. Search Engines

2. Internet protocol (IP) • Two major functions: Addresses that identify hosts, locations and identify destination • Connectionless protocol • Reliability: • Data corruption : faulty way of breaking up messages • Lost data packets • Duplicate arrival • IP addresses for host: • A numerical label assigned to each device participating in a computer network that uses internet protocol for communication • Hostnames (Ex: cs.umb.edu) • We prefer meaningful names but behind the scenes hostnames are converted to IP addresses which are a series of four decimal numbers separated by dots. Ex: 205.39.155.18 (stored in 32 bits). What is a potential problem with 32 bit addresses? • Can be split into network address (type/size of network) and a host number (machine/device number on this network)

3. Domain name system cs.bu.edu ? ? • Domain names: The part of a hostname that specifies type of organization or group • Top-level domain (TLD): The last section of a domain name specifying the type of organization or its country of origin. • Domain name system is used to translate hostnames into numeric IP addresses so domain name servers (when you make a request) translates request into an IP address and then searches for the IP address

4. World Wide Web • Good News: • Millions of webpages available on a variety of topics • Bad News: • Millions of webpages available on a variety of topics • Haphazard labeling • Sitting on servers in various locations How do search for a specific topic?

5. Search engines! • A web search engine is designed to search for information on the world wide web and FTP servers • Search based on key words (Crawl) • Keep an index of useful pages (Index) • Presents users with information based on the index (Search) • Search engines operate algorithmically or a mix of algorithmic and human input

6. Search engines: History • Early summer of 1993: No commercial or large scale search engines existed • W3Catalog : • World’s first primitive search engine • By Oscar Nierstrasz at the University of Geneva • Wandex : • Web Robot • Mathew Gray at MIT • Measure the size of World Wide Web

7. Search engines: History • Aliweb : • Indexed by hand • Jump Station : • Web Robot • Crawl, Index and Search • 2000: Google rose to fame! • Algorithm called PageRank that ranks webpages based on the number and PageRank of links available on the website

8. Difference • Early search engines • Few hundred thousand pages • One or two thousand inquiries • Top search engines • Few hundred millions of pages • Billions of queries per day

9. Web Crawling • Also known as a spider • Special software agent that finds web pages, also follows links on web pages • Contents are analyzed • Words, titles, special fields called meta tags • Starting point? • Popular pages

10. Google: Web Crawling • At its peak: • Use multiple spiders • Each spider can keep ~300 connections to pages at a time • Generates 600K/s • Starting points: • Dedicated server that feeds URLs to spiders • Instead of relying on ISP for domain names they have their own DNS server • Google spider looks at two things: • Significant words within the page • Location of the words Why is location important?

11. Meta tags • Owner specific • Can be helpful • Problem? • Robot exclusion protocol

12. Indexing • Spiders get the data • Now what? • Content analysis • Method by which information is sorted and stored • One way: Storing the word and associated URL • No way to tell if the word is important or trivial • How many times was the word used?

13. Ranking • A relationship between items about their ordering • For more useful information: • Number of times word appears on page • Assign a weight to each word • Each search engine has a different formula for assigning weight to words in its index • Popular way of indexing : Hashing • Numerical value assigned to each word that can be retrieved using a formula

14. Building a search • Query: string of words or a single word • Complex queries requires the use of Boolean operators • AND : terms joined by operator, also ‘+’ • OR • NOT • FOLLOWED BY • NEAR • Quotation Marks

15. Building a search • Literal searches: based on Boolean operators • Concept-based: Statistical analysis on pages containing words or phrases you search for • Information stored about each page is greater • Search times may be longer • Natural language queries • Ask a question : AskJeeves.com • Parses keywords

16. Money money money.. • Beyond selling shares or private investment • Three main methods: • Online purchases • Web advertising • Keywords relating to product, service or business • Allowing users to integrate ads into their own websites • Fourth shady way: Selling user information

17. Google Company Culture • Sergey Brin and Larry Page began google with a few networked computers at Stanford • Multibillion dollar organization • >19000 employees globally • Market Capitalization >$145 billion • Googleplex: • Free food – gourmet café stations • Snack rooms • Exercise rooms • Game rooms • Grand piano