Optimized Index Structures for Querying RDF from the Web

1. Andreas Harth, Stefan Decker andreas.harth@deri.org 3rd Latin American Web Congress Optimized Index Structures for Querying RDF from the Web

2. Contents Scenario Index Structure Query Processing Implementation Evaluation Summary

3. Scenario • Data collected from the Web • Much instance data, few ontology • Storage of large amounts of data with mostly unknown schemas, fast retrieval essential

4. Example RDF with Context

5. Notation 3 • #s #p #o . syntax for RDF • E.g. @prefix foaf: http://xmlns.com/foaf/0.1/ . http://decker.cn/stefan/ foaf:name “Stefan Decker” . • N3 is extension of RDF data model with quotation of graphs and universally quantified variables • Able to express queries with ql:select and ql:where predicates

6. Example Query • Get all triples where predicate is foaf:name and object is “Stefan Decker” @prefix foaf: <http://xmlns.com/foaf/0.1/> . @prefix yars: <http://sw.deri.org/2004/06/yars#> . @prefix ql: <http://www.w3.org/2004/12/ql#> . <> ql:where { ?s foaf:name “Stefan Decker” . } .

7. Contents • Scenario • Index Structure • Query Processing • Implementation • Evaluation • Summary

8. Indexes • Lexicon: store mappings from literal values and resources to object IDs and vice versa • Quad Index: store quads (s, p, o, c)

9. Object Identifiers • OIDs help to save space (only need to store 64 bit OID instead of whole resource/literal all the time)

10. Keyword Index • Simple search UI require keyword searches • Inverted index on literals

11. Quad Access Patterns • We want to be able to retrieve all combinations of (s,p, o, c) without performing a join • In total, 2*2*2*2 = 16 combinations

12. Recap: B+-Trees • Underlying storage technique based on B+-trees • One property of B+-trees: range lookups/prefix lookups • (key, value) pairs with fast retrieval on given (partial) key

13. Complete Index on Quads • Given prefix lookup capabilities, only 6 indexes are needed to cover all access patterns

14. Occurrence Counts • Queries for in-degree/out-degree of a node in a graph • Also: statistics for join reordering • Store occurrence counts for quad patterns directly in index

15. Contents • Scenario • Index Structure • Query Processing • Implementation • Evaluation • Summary

16. Physical Access Plans • Access pattern (?s, foaf:name, “Stefan Decker”, ?c) • Translate string values to OIDs (2, 11) • Determine index (POCS) • Construct key (2:11:*:*) • Perform prefix lookup on POCS with key 2:11:*:* • Translate result back to string values

17. Contents • Scenario • Index Structure • Query Processing • Implementation • Evaluation • Summary

18. Prototype Implementation • Java, JDBM, Apache Tomcat • HTTP interface: GET/POST for querying, PUT for adding data, DELETE for removing data

19. Contents • Scenario • Index Structure • Query Processing • Implementation • Evaluation • Summary

20. Index Construction Performance - Lehigh Univ(20)

21. Query Performance Evaluation Queries: 1: ?x rdf:type univ:UndergradStudent 2: ?x ?p "UndergraduateStudent0" 3: <http://www.Univ965.edu> ?p ?o 4: ?x univ:worksFor ?y

22. Contents • Scenario • Index Structure • Query Processing • Implementation • Evaluation • Summary

23. Summary • Complete index on RDF quads to minimize joins • Keyword-based searches • Extensive statistical information • High-performance metadata repository • Core storage backend and query-processing technology for SWSE (Semantic Web Search Engine) • Used in projects at e.g. University of Karlsruhe (RDFReactor, RDF2Go)