Using Provenance for Quality Assessment and Repair in Linked Open Data

1. Using Provenance for Quality Assessment and Repair in Linked Open Data Giorgos Flouris, Yannis Roussakis, Maria Poveda-Villalon, Pablo N. Mendes, Irini Fundulaki Publication at EvoDyn-12

2. Setting and General Idea • Linked Open Data cloud • Uncontrolled • Vast • Unstructured • Dynamic • Datasets interrelated, fused etc • Quality problems emerge • Assessment (measure quality) • Repair (improve quality)

3. ES FR EN GE PT Motivating Example • User seeks information on Brazilian cities • Fuses Wikipedia dumps from different languages • Guarantees maximal coverage, but may lead to conflicts • E.g., city with two different population counts

4. Main Tasks • Assess the quality of the resulting dataset • Framework for associating data with its quality • Repair the resulting dataset • By removing one of the conflicting values (i.e., one of the conflicting population counts) • How to determine which value to keep? • Solution: use heuristics • Here, we evaluate the use of provenance-related heuristics • Prefer most recent information • Prefer most trustworthy information

5. Contributions • Emphasis on provenance • Assessment metrics (done) • Heuristics for repair (done, but does not support metadata information) • Contributions: • Extend repair algorithm to support heuristics on metadata • Define 5 different metrics based on provenance • Used for both assessment and repair • Evaluate them in a real setting

6. Quality Assessment • Quality = “fitness for use” • Multi-dimensional, multi-faceted, context-dependent • Methodology for quality assessment • Dimensions • Aspects of quality • Accuracy, completeness, timeliness, … • Indicators • Metadata values for measuring dimensions • Last modification date (related to timeliness) • Scoring Functions • Functions to quantify quality indicators • Days since last modification date • Metrics • Measures of dimensions (result of scoring function) • Can be combined • We use this framework to define our metrics

7. Quality Repair (Setting) • Focus on validity (quality dimension) • Encodes context- or application-specific requirements • Applications may be useless over invalid data • Binary concept (valid/invalid) • Generic

8. Quality Repair (Rules) • Rules determine validity • Expressive • Disjunctive Embedded Dependencies (DEDs) • Cause interdependencies • Resolution causes/resolves other violations • Difficult to foresee ramifications of repairing choices • User cannot make the selection alone

9. Quality Repair (Preferences) • Selection is done automatically, according to a set of user-defined specifications • Which repairing option is best? • Ontology engineer determines that via preferences • Specified by ontology engineer beforehand • High-level “specifications” for the ideal repair • Serve as “instructions” to determine the preferred solution for repair • Highly expressive

10. Quality Repair (Extensions) • Existing work on repair is limited • Provenance cannot be considered for preferences • Assessment metrics based on provenance cannot be exploited • Extensions are needed (and provided) • Metadata (including provenance) can be used in preferences • Preferences can apply on both repairs and repairing options • Formal details omitted (see paper)

11. Experiments (Setting) • Setting taken from the motivating example • Fused 5 Wikipedias: EN, PT, SP, GE, FR • Distilled information about Brazilian cities • Properties considered: • populationTotal • areaTotal • foundingDate • Validity rules: properties must be functional • Repaired invalidities (using our metrics) • Checked quality of result • Dimensions: consistency, validity, conciseness, completeness and accuracy

12. Metrics for Experiments (1/2) • PREFER_PT: select conflicting information based on its source (PT>EN>SP>GE>FR) • PREFER_RECENT: select conflicting information based on its recency (most recent is preferred) • PLAUSIBLE_PT: ignore “irrational” data (population<500, area<300km2, founding date<1500AD) otherwise use PREFER_PT

13. Metrics for Experiments (2/2) • WEIGHTED_RECENT: select based on recency, but in cases where the records are almost equally recent, use source reputation (if less than 3 months apart, use PREFER_PT, else use PREFER_RECENT) • CONDITIONAL_PT: define source trustworthiness depending on data values (prefer PT for small cities with population<500.000, prefer EN for the rest)

14. Consistency, Validity • Consistency • Lack of conflicting triples • Guaranteed to be perfect (by the repairing algorithm), regardless of preference • Validity • Lack of rule violations • Coincides with consistency for this example • Guaranteed to be perfect (by the repairing algorithm), regardless of preference

15. Conciseness, Completeness • Conciseness • No duplicates in the final result • Guaranteed to be perfect (by the fuse process), regardless of preference • Completeness • Coverage of information • Improved by fusion • Unaffected by our algorithm • Input completeness = output completeness, regardless of preference • Measured to be at 77,02%

16. Accuracy • Most important metric for this experiment • Accuracy • Closeness to the “actual state of affairs” • Affected by the repairing choices • Compared repair with the Gold Standard • Taken from an official and independent data source (IBGE)

17. Accuracy Evaluation … fr.dbpedia en.dbpedia pt.dbpedia Instituto Brasileiro de Geografia e Estatística(IBGE) Fuse/Repair dbpedia:areaTotal dbpedia:populationTotal dbpedia:foundingDate dbpedia:areaTotal dbpedia:populationTotal dbpedia:foundingDate Gold Standard integrated data Compare Accuracy

18. Accuracy Examples • City of Aracati • Population: 69159/69616 (conflicting) • Record in Gold Standard: 69159 • Good choice: 69159 • Bad choice: 69616 • City of Oiapoque • Population: 20226/20426 (conflicting) • Record in Gold Standard: 20509 • Optimal approximation choice: 20426 • Sub-optimal approximation choice: 20226

19. Accuracy Results

20. Accuracy of Input and Output

21. Conclusion • Quality assessment and repair of LOD • Evaluated a set of sophisticated, provenance-inspired metrics for: • Assessing quality • Repairing conflicts • Used in a specific experimental setting • Results are necessarily application-specific • THANK YOU!