Quality Reporting with JD+

1. Quality Reporting with JD+ Where are we? Where are we going? SAEG – June 7th, 2016

2. The Current JD+ Quality Report: Principle • n numerical quality indicators. • Each of them is translated into a qualitative indicator. • This set of qualitative indicators is then summarized in a qualitative global indicator. • Use of colors: green, orange, red, severe Palate Jean (2014), JD+ Quality Indicators, NBB

3. Current JD+ Quality Report

4. JD+ QR: Selected Indicators • Diagnostics on: • The coherence of the decomposition ("Basic checks" group), • Visual spectral inspection ("Visual spectral analysis"), • The residuals of the RegArima pre-processing ("RegArima residuals" group), • The residual seasonality ("Residual seasonality" group) • The decomposition ("Seats" group for Tramo-Seats, M-Statistics group for X12). • Most of them use parameters (usually thresholds) that can be modified • Each group of diagnostics might be disabled, when it is considered as meaningless.

5. Pros and Cons Pros Very simple to read Very useful for the producer Cons Could be misunderstood (Warnings vs. Quality) Tramo/Seats “biased” Too many indicators (7 different seasonality tests) But output csv matrix incomplete SACE 5

6. The “Quality Report Plug-In Project” 2 very different ideas/objectives Technical side: Disseminating the knowledge on JD+ to make people able to contribute Statistical side: Developing a customized Quality Report The challenges Getting basic programming skills What is a Quality Report? Selecting and weighting statistical indicators SACE 6

7. Quality Report: A Quite Fuzzy Concept • “Quality report” are words widely used in the SA context but they cover various meanings and concepts. • When is it used, by whom, and for which purposes? • Producing, Disseminating, Using (the series)

8. Producing the series • To define the process • choosing the best strategy/model • To run the process • using the old/previous parameters • To check/improve the process • Annual campaign, production time • To prepare the series for publication & storage

9. Disseminating and storing the series • Various publications • Informing on overall quality and revisions etc. • Various databases, various metadata • Producer databases • Institute databases • Other databases • Often a need for other quality diagnostics (timeliness for example)

10. Using the series • “Basic” user • Direct use of the series (indexing contract) • Use of the series to comment the evolution of the sector/Economy • Usually “trust the series”; basic QR • Researcher/Econometrician • Focus on very specific criteria: timeliness, revisions, noise, dynamics of the series (model) etc. • Use the QR ….. or not: redo its own adjustment

11. Various objectives → Various QR? • Yes • Metadata ≠ Publication ≠ Producer ≠ Researcher • …. and No • Almost every information we can think about can be output by JD+. • Metadata and Publication: “just a reporting problem” • Some problems: • JD+ cannot contain all necessary information (timeliness, real time revisions, classifications etc.) • Quality of indirect aggregates (chain-linked or not)

12. What can do JD+? • Lots of things ….. • The output matrix could be extended / customized • The signal alerts (red lights) can be adapted/tuned • Reports can be automatically created using standard formats (SDMX) • But not everything • It might consume resources (computing time, space) if too many indicators are required. • JD+ does not store all the information useful for a QR • JD+ does not handle “direct vs. indirect” aggregates.

13. A Generic Structure • A set of descriptive data (ex: method, model, parameters, length etc.) • Weight of the descriptive data • A set of quality statistics (ex: M1-M11 statistics in X-12) • Weight of the statistics and quality statistics • Weight of the series

14. The “descriptive data” • Typically information necessary for the metadata • “Clarity” and “transparency” aspects of quality • Information on the series (name, length etc.), on the process (method, regressors etc.) and on the result of the process (model, # of outliers, TD effect etc.) • Could be also used to rerun the process • Weight • Typically, each item is present (1) or not present (0). • Allows summarizing the overall quality (presence) of the “metadata”

15. The “quality statistics” • Typically information necessary to check the efficiency of the process for a series • “accuracy” aspect of quality • No residual seasonality, TD effect, M1-M11 statistics, JD+ set of quality statistics • Also used to improve the seasonal adjustment • Weight • To derive a global quality index: Q-statistics, JD+ alerts.

16. Weight of the series • Important in production time • Examples: weight of the sector in the NACE, contribution of the sector to the growth of the aggregate, weight of the country in the UE etc. • Permits a selective editing: if something is going wrong during production, you focus on the most important series

17. The real problems to solve • The choice of the information to output • Revisions: the last value of the series can be kept • Some metadata cannot be included (see example) • Too many indicators (7 seasonality tests) • The choice of the weights • Simple (?) for metadata (1/0) • More complex for quality statistics • How to use the QR in production time?

18. A First Plug-In for Eurostat Unit F3 To report on the SA of Job Vacancy Survey series First specifications during a Hackathon session organized by the SAUG (08/12/2015) Several iterations to agree on the quality indicators A “final product” delivered end of 02/2016. Used in production by Member States SACE 18

19. The “command” SACE 19

20. The “final” version SACE 20

21. JD+ Quality Report • Joint work with Istat and NBB • First steps: • Learning to develop simple plug-ins • Output a very complete csv matrix for both TS & X12 • First statistical studies using this basic plug-in to select relevant quality indicators • Checking the probable redundancy of seasonality tests SACE 21