Querying Infinite Databases Safety of Datalog Queries over infinite Databases (Sagiv and Vardi ’90) Queries and Computation on the Web (Abiteboul and Vianu ’97) Itay Maman 049011 Student Symposium, 5 July 2006
Simple Technion Queries… (Domain: The Technion’s students database) • Q1: Which courses did Gidi attend? • SELECT course FROM students WHERE name='Gidi' • Q2: Which students took 234218? • SELECT name FROM students WHERE course='234218'
Simple Web Queries… Q3: Which pages does my home page link to? SELECT target FROM links WHERE source='www.geocities.com/mysite' Q4: Which pages link to my home page? SELECT source FROM links WHERE target='www.geocities.com/mysite' Q4 is challenging: No matter how long my web-crawler works… … I can never find all incoming links of a page! This is an infinite query The more you crawl the more answers you get (In Q3 the size of the result set is bounded)
Leading questions • What does an infinite DB look like? • Can we evaluate a query over an infinite DB? • Can we determine the finiteness of a query? • But first, some Datalog…
Datalog • Why Datalog? • Supports recursion/transitive closure (unlike SQL) • Recursion is essential in large data-sets • Terminates if DB is finite • Very simple • program = A collection of rules • rule = A sequence of terms • In our program: • Three rules • Two queries (AKA: IDB): g(X), small(X,Y) • One Table (AKA: EDB): before(X,Y) • A goal predicate from which execution starts • We choose g(X) as the goal g(X) :- small(X,2). small(X,Y) :- before(X,Y). small(X,Y) :- small(X,Z), before(Z,Y).
Finiteness • A DB is finite If every table is a finite set • before(X,Y) { (0,1), (1,2), (2,3) } • Possible evaluation schemes: • Brute force • Bottom up • Optimizations • The Requirement: Finiteness of tables • The guarantee: Termination of the Datalog program
Infinity • Here is another definition for our table • before(X,Y) { (X,X+1) | X 0 } • We now have an infinite DB • The Problem: we cannot iterate over the tuples in the set • The solution: Top-down algorithm • Such tables are quite common • The internet links relation links(X,Y) { (X,Y) | page X links to page Y } • Java’s subclassing relation extends(X,Y) { (X,Y) | class X extends Y } Leading question: What does as infinite DB look like?
Example: Top-down evaluation g(W) :- small(W,2). small(A,B) :- before(A,B). small(X,Y) :- small(X,Z), before(Z,Y). before(X,Y) { (X,X+1) | X 0 } s(X,Y) = b(X,Y) s(X,Z) b(Z,Y) g(W) = s(W,2) = b(W,2) s(W,Z) b(Z,2) = {(1,2)}s(W,1) {(1,2)} = {(1,2)} [b(W,1)s(W,Z) b(Z,1)] {(1,2)} = {(1,2)} [{(0,1)}s(W,0){(0,1)}] {(1,2)} = {(1,2)} [{(0,1)} [b(W,0) s(W,Z) b(Z,0)] {(0,1)}] {(1,2)} = {(1,2)} [{(0,1)} [s(W,Z) ] {(0,1)}] {(1,2)} = {(1,2)} [{(0,1)} {(0,1)}] {(1,2)} = {(1,2)} {(0,1)} {(1,2)} = {(1,2)} {(0,2)} = {(1,2), (0,2)} • b : before • s : small • : Join
Top-down evaluation • The Top-down algorithm • Init: assign r body of the goal • Loop: • (Intelligently) Pick a term, t, from r • If t is a query term: • Replace it with the union of the rules indicated by t • If t is a table term: • Replace it with the set generated by the table • Replace s expressions (in r) with • Replace s expressions (in r) with s • Evaluate relational algebra expressions (if both sides are known) • Stop if no further replacements can be made Leading question: Can we evaluate a query over an infinite DB? Yes
Infinite Queries • Can the top-down algorithm run forever? • Yes • Case 1: An table that returns an infinite result • evenProduct(X,Y) { (X,Y) | X*Y mod 2 = 0 } • divides(X,Y) { (X,Y) | X mod Y = 0 } • links(X,Y) { (X,Y) | page X links to page Y } • weak-safety: all intermediate results are finite • Result #1 (Sagiv and Vardi ’90): • Weak-safety is decidable given F/C (finiteness constraints) of tables • F/C of evenProduct: None • F/C of divides: X => Y • F/C of links: X => Y • Algorithm: Tracking flow of values from assigned variables
Infinite Queries (cont.) • Can the top-down algorithm run forever? • Yes • Case 2: The algorithm’s recursion never stops • A query/table is used in its “unbounded” direction g(W) = s(2,W) = b(2,W) s(2,Z) b(Z,W) = {(2,3)}s(2,Z) b(Z,W) = {(2,3)} [b(2,Z) s(2,Z’) b(Z’,Z)] b(Z,W) … s(X,Y) = b(X,Y) s(X,Z) b(Z,Y) g(W) :- small(2,W). small(A,B) :- before(A,B). small(X,Y) :- small(X,Z), before(Z,Y). before(X,Y) { (X,X+1) | X 0 } • Results #2-3 (Sagiv and Vardi ’90): • Termination is undecidable in the general case • Termination is decideable if all queries are unary
Infinite Queries (summ.) Leading question: Can we determine the finiteness of a query? No • We can automatically determine weak-safety • We cannot (automatically) determine termination • But, one can analytically prove that a given query over a given DB is finite • E.g., our small(W,2) program
The Web as a DB • The web data model (WDM): • A scheme of a DB that can represent the web graph • Just three tables: urls = { u | u is a url of a web-page } links = { (u1,u2) | u1 links to u2; u1, u2 urls } Words = { (u,w) | w appears in page u; u urls } • Result #4 (Abiteboul and Vianu ’97): • If a Datalog program with no literals halts over an infinite DB, its result is • => A non-trivial query (over an infinite DB) must have a literal
Web - Machines • Browsing Machine • A weakly safe Datalog program (over WDM) • At least one URL literal • Searching/Browsing Machine • An unsafe Datalog program (over WDM) • Evaluates queries in parallel • Allowed literal types: URLs, Words • Claims #1-2 (Abiteboul and Vianu ’97): • Browsing machine: • Represent a user following static links from a page • Searching/Browsing machine: • Also allows the user to access search engine
Discussion: Finite approximation • Relational Database servers are very popular • Such DBs are finite • Also, computing a table on demand may be slow • Better performance at batch processing The challenge: Build a finite replacement for an infinite DB • Formally: • Given a finite query, q, over an infinite DB, • (Finiteness of q proved analytically) • Build a finite Database, , such that q over yield the same result as q over
Discussion: Finite approximation • Example: Our small(W,2) program • A finite, sound table: before(X,Y) { (0,1), (1,2) } • A finite, unsound table: before(X,Y) { (0,1) } • The process: • Compute the transitive closure of the before relation • Start from the literal ‘2’ at the right-hand side position • Condition: the table graph must end with a sink • In before the sink is the vertex ‘0’ • => We can build a finite DB • Sadly, In the web-graph no such sink exists
Discussion: Temporality • Crawling takes time • The subject may change while crawling • The DB is a snapshot which never happened • (Open Question): • Can we decide whether a result was really “true” at some point?
More issues • Relational algebra over large relations • BDD • Negation • Stratified Datalog
- Questions ? -
Datalog • Semantics: ??? • Straight forward mapping to Relational Algebra?? g(X) :- small(X,2). small(X,Y) :- before(X,Y). small(X,Y) :- small(X,Z), before(Z,Y).
Example: Bottom-up evaluation Initialization: Translate the EDBs into relations
Example: Bottom-up evaluation apply small(X,Y) :- before(X,Y).
Example: Bottom-up evaluation apply small(X,Y) :- small(X,Z), before(Z,Y). Join
Example: Bottom-up evaluation apply g(X) :- small(X,2).
Finiteness before(X,Y) { (0,1) (1,2) (2,3) } • The Bottom-up algorithm: • Init: • For each EDB, p, assign r(p) Relation of all tuples satisfying p • For each IDB, p, assign r(p) • Loop: • Choose a rule p(…) :- t1(…), t2(…), … tn(…) • t join of all r(ti), where 1 i n • r(p)r(p)t • Continue until a fix-point is reached • Requires: Finiteness of EDBs • Ensures: Termination
