1 / 16

Efficient Detection & Correction of Business Data Inconsistencies

Learn about algorithms for identifying and resolving obvious errors in business survey data, improving data accuracy and editing processes for better results.

luke-jensen
Télécharger la présentation

Efficient Detection & Correction of Business Data Inconsistencies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Algorithms for Detecting and Resolving Obvious Inconsistencies in Business Survey Data Sander Scholtus Statistics Netherlands UN ECE Work Session on Statistical Data Editing April 22, 2008

  2. Overview • Selective editing at Statistics Netherlands • Sign errors • Rounding errors

  3. Editing SBS at SN • Selective editing • SLICE • Automatic correction of systematic errors (obvious inconsistencies)

  4. Systematic errors • Currently implemented: • ‘Thousand-error’ • Wrongfully negative value • Empty total with non-empty component items

  5. Obvious inconsistencies • Correcting (obvious) systematic errors in a separate step… • makes the editing process more efficient • makes SLICE more effective

  6. Results block (SBS ≤ 2005) operating returns + operating results + operating costs – financial revenues + operating surplus + financial expenditure – provisions rescinded + balance of provisions + provisions addes – + exceptional income exceptional result + – exceptional expenses pre-tax results

  7. Results block (example 1) 2,100 + 150 + 1,950 – 0 + 10 + 10 – 20 + 15 + 5 – + 50 40 + – 10 195

  8. Results block (example 1) 2,100 + 150 + 1,950 – 0 + 10 + 10 – 20 + 15 + 5 – + 50 40 + – 10 195

  9. Results block (example 1) 2,100 + 150 + 1,950 – 0 + –10 + 10 – 20 + 15 + 5 – + 50 40 + – 10 195

  10. Results block (example 2) 5,100 + 450 + 4,650 – 0 + 130 + 130 – 20 + 20 + 0 – + 15 10 + – 25 610

  11. Results block (example 2) 5,100 + 450 + 4,650 – 0 + 130 + 130 – 20 + 20 + 0 – + 15 10 + – 25 610

  12. Results block (example 2) 5,100 + 450 + 4,650 – 130 + 130 + 0 – 20 + 20 + 0 – + 25 10 + – 15 610

  13. Definition • Sign error / interchanged returns and costs: • at least one edit rule is violated • it is possible to obtain a consistent results block by only changing the signs of balance variables and/or interchanging the values of returns items and costs items

  14. Mathematical formulation • Edit rules: • Find the solution to: • if then change the sign of • if then interchange and

  15. Rounding errors • Linear equalities • Smallest possible violation of edit rule • Virtually no influence on publication figures • SLICE • Separate step

  16. Thank you for your attention!

More Related