ENGS 4 - Lecture 4 Technology of Cyberspace Winter 2004 Thayer School of Engineering Dartmouth College

1. ENGS 4 - Lecture 4 Technology of Cyberspace Winter 2004Thayer School of EngineeringDartmouth College Instructor: George Cybenko, x6-3843 gvc@dartmouth.edu Assistant: Sharon Cooper (“Shay”), x6-3546 Course webpage: www.whoopis.com/engs4 ENGS4 2004 Lecture 4

2. Today’s Class • Discussion of Morgridge’s talk • Assignment (due Jan 20) • Web page and HTML status • Basics of search technologies • Break • Phillip’s presentation • Chad’s presentation • Rule-based and expert systems ENGS4 2004 Lecture 4

3. Cisco – Technology and Culture • What were your reactions? • What were his main points? • Was it an effective presentation? • How could it have been improved? • What are Cisco’s strengths? • What are Cisco’s weaknesses? • Would you invest in Cisco? • Would you want to work for Cisco? ENGS4 2004 Lecture 4

4. Homework 1 – Due Jan 20 • Estimate the number of bytes in the ORC (2003-2004 edition, printed) • How much time would downloading it require on a 56 kbps modem line? • How much time would downloading it require on a 10 mbps ethernet? • How much time would downloading it require on a 100 mbps ethernet? • What is the bandwidth and latency of the NASA Mars Rover to earth channel? • Create a web page with the answers to these questions on the webpage. ENGS4 2004 Lecture 4

5. Homework 1 – Due Jan 20 Create a web page with the answers to these questions on the webpage. ENGS4 2004 Lecture 4

6. Web pages and basic HTML • Questions? • Have you tested your web account? • Try something simple first and build up from that – simple page with “hello”, upload and test it, add text, add graphics, etc. • Read about more advanced HTML and try to use advanced constructs • Try to copy interesting/clever constructs you have seen on other pages ENGS4 2004 Lecture 4

7. Basic web search technology • Visit www.google.com www.excite.com • What are some differences? • How does the basic technology work? ENGS4 2004 Lecture 4

8. Vector Space Model in Information Retrieval • List all words in your “dictionary” • eg aardvark, aaron, able, act, advertise, bad, … • A “stop list” consists of words too common to be useful for retrieval – eg, the, is, a, up • Process a document to obtain a “vector” of word frequencies: • “Aaron the acting aardvark was able to join the Aardvark Society of Actors.” • becomes (2, 1, 1, 0, 0, 1, …) • this is a document word-frequency representation • no syntax, grammar, semantics…just words ENGS4 2004 Lecture 4

9. Comparing two word-frequency vectors aardvark, aaron, able, act, advertise, bad, … • Another document: • “An aardvark would be a bad actor.” • becomes (1, 0, 0, 0, 0, 1,...) • “Aaron the acting aardvark was able to join the Aardvark Society of Actors.” • was (2, 1, 1, 0, 0, 1, …) • The score between the two documents is obtained by multiplying coordinates and adding. ENGS4 2004 Lecture 4

10. Comparing two word-frequency vectors • “An aardvark would be a bad actor.” • is (1, 0, 0, 0, 0, 0) • “Aaron the acting aardvark was able to join the Aardvark Society of Actors.” • is (2, 1, 1, 0, 0, 1, …) • The score between the two documents is obtained by multiplying coordinates and adding. • 1*2 + 0*1 + 0*1 + … = 2 • Stemming: reduce words to roots (ie actor, actors, acting, etc have “act” as root. • Score becomes larger. IE, 4. ENGS4 2004 Lecture 4

11. First generation web search • Each web page is represented as a word-frequency vector after stemming and other normalizations • a user search is made into another word-frequency vector • the search vector is compared against web page vectors that have been indexed • pages with the highest scores are listed as results for that search ENGS4 2004 Lecture 4

12. Embellishments • Adding searches to retailers and others who pay for their pages to be ranked highly (Seems like Excite does that…how does Google handle this revenue opportunity?) • Taking the highest ranking pages and doing some more advanced processing to determine the “hubs” and “authorities” (Google does something like this) • Mini-lecture topic – economics of search engines, revenue models, etc ENGS4 2004 Lecture 4

13. Hubs and authorities page page page page page page page page page page page page page page page hub Google ranks hubs and authorities differently than other pages page authority ENGS4 2004 Lecture 4

14. Small world graphs • Social networks • Biological networks • Infrastructure networks • Kevin Bacon • Milliken’s experiment • Power-law distributions • Mini-lecture topic…volunteers? ENGS4 2004 Lecture 4

15. Break ENGS4 2004 Lecture 4

16. Phillip’s Mini-lecture ENGS4 2004 Lecture 4

17. Chad’s Mini-lecture ENGS4 2004 Lecture 4

18. Rule-based systems • Visit advanced search in Google • User constructs a “Boolean query” • EG. • must include – dartmouth, hockey • may include – women female • does not include – men • Boolean expression is: “dartmouth and hockey and (women or female) and (not men)” ENGS4 2004 Lecture 4

19. Aristotelian logic • predicates: A, B, C, etc. • Basic operators: • and : A and B true when both true • or: A or B true when either true • not: not A true when A false • Derived operators: if A then B • true providing B is true whenever A is true • only false when A is true but B is false • (if A then B) is equivalent to: not (A and not B) ENGS4 2004 Lecture 4

20. Aristotelian Logic at work • Set of “rules” • eg All humans are mortal. • logical form: “if (x is human) then (x is mortal).” • (x is human) is a predicate with variable x • Set of “axioms”: statements known to be true • eg (Aristotle is human). • Combine them to get: Aristotle is mortal. ENGS4 2004 Lecture 4

21. Such logic is the basis for “expert systems” or “rule-based systems” • Early automated medical diagnosis • Maintenance procedures for complicated machinery (cars, planes, etc) • It is the easiest and most prevalent way to implement some sort of “artificial intelligence” • What are the limitations? • inability to deal with uncertainty (ie probability) • large sets of rules developed by many people often become inconsistent, brittle, unmaintainable ENGS4 2004 Lecture 4

22. Next lecture • Classical uses of rule-based systems to “predict the future” with distributed information • Current uses of rule-based systems on the internet • Critique ENGS4 2004 Lecture 4

23. Mini-lecture topics • Technology behind recommender systems such as Amazon, Netflix, etc. ENGS4 2004 Lecture 4