1 / 39

SDMX Standards

SDMX Standards. Relationships to ISO/IEC 11179/CMR Arofan Gregory Chris Nelson Joint UNECE/Eurostat/OECD workshop on statistical metadata (METIS): Geneva 3-5 April 2006. Major Points. Activity and Results Scope Areas of Commonality Applying ISO 11179 to SDMX CMR and SDMX Registries

mblaine
Télécharger la présentation

SDMX Standards

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SDMX Standards Relationships to ISO/IEC 11179/CMR Arofan Gregory Chris Nelson Joint UNECE/Eurostat/OECD workshop on statistical metadata (METIS): Geneva 3-5 April 2006

  2. Major Points • Activity and Results • Scope • Areas of Commonality • Applying ISO 11179 to SDMX • CMR and SDMX Registries • Use in Applications: A Case Study

  3. Activity • We were asked to describe the relationship between ISO 11179/CMR and SDMX • There is not complete overlap • We focused on points of contact between them • This was an interesting project, which we will continue working on

  4. Results • ISO 11179/CMR and SDMX: • ISO 11179 provides a useful pivotal model for mapping between CMR (or others) and SDMX constructs • ISO 11179 does not model the semantics of a metamodel – it is useful for the semantics of the models (specific metadata structure) or even instances of the model (specific data set) • The utility of this mapping depends on the application • The SDMX and ISO 11179 registries are complementary in function (semantics + access)

  5. Characterization • ISO 11179 models the semantics of data elements • CMR extends this to cover survey lifecycle metadata – adds structure • SDMX models the structure of metadata and data for aggregate statistical data – you supply your own semantics (concepts) • This is not a comparison of like things: it is a connecting of complementary ones

  6. Scope – ISO 11179 • ISO 11179 is a metadata content standard focused primarily on the semantics of data. Secondarily, it provides the rules and structures for registering descriptions of data. • The standard exists in 6 Parts, and the 2nd edition is now available. • Part 1: Framework for the specification and standardization of data elements • Part 2: Classification for data elements • Part 3: Registry metamodel and basic attributes • Part 4: Rules and guidelines for the formulation of data definitions • Part 5: Naming and identification principles for data elements • Part 6: Registration of data elements • This presentation is concerned mainly with Parts 3 and 5

  7. SCOPE: ISO 11179 Part 3High Level View - Data Elements and Concepts Country Identifier Countries of the World ISO 3166: Country Code List Country Name Country Code

  8. SCOPE – CMR • The CMR model is designed to support • metadata necessary to describe the survey life cycle • linkages between similar designs and processes used across surveys • use of metadata to drive systems in support of the survey life cycle • The CMR model extends the ISO 11179 (Part 3) metamodel to support CMR specific artefacts

  9. SCOPE - CMR ISO 11179 CMR Links to appropriate ISO 11179 artefact to supply semantics and management

  10. SCOPE - SDMX • SDMX standards comprise • Information Model • Syntax implementations based on the Information model (XML schemas and UN/EDIFACT) • Content-oriented guidelines • Statistical subject matter domain scheme • Cross Domain metadata concepts • Metadata vocabulary (MCV – Metadata Common Vocabulary)

  11. Design Meta Schema for aggregated data and metadata Create data/metadata structure instance (data/metadata structure definition) Generate structure specific schema Report/publish data/metadata Use cross domain metadata concepts and code lists Register SDMX Registry Information Meta ModelAggregated data, metadata, structural metadata

  12. Areas of Commonality ISO 11179 and SDMX • Both standards support the definition of • Concepts • Use of concepts in structures • Representations/Value domains • Both standards support the concept of a registry • ISO 11179 specifies a registry metamodel • SDMX does not specify a registry metamodel • ISO 11179 does not specify registry interfaces based on the registry model • SDMX specifies registry interfaces based on the SDMX model • Implementation of a registry and mapping to a registry model is left to the implementor • Can use ISO 11179 registry implementation, ebXML registry implementation, or bespoke registry implementation

  13. Concepts – ISO 11179

  14. Concepts - SDMX Data Element Concept Conceptual Domain (for SDMX this defines the core/default representation: implicit here is whether it is enumerated or non enumerated) [not given explicit name in SDMX] ISO-11179 equivalent

  15. Data Element: ISO 11179

  16. Concept Usage - SDMX Concept Usage = Data Element [can be given explicit name in SDMX (e.g. ISO 11179 equivalent name), but this is not required and is not used for identification purposes] Value Domain

  17. Concept Usage - SDMX • There is no explicit “Data Element” name for usage of a Concept (e.g. FREQUENCY Concept used for a Dimension) • Concept names are used within the context of the role of the Concept (Dimension, Attribute, Measure) – these are structure components • The unique identifier for a structure component comprises: <data structure definition.structure component list.concept>e.g. BOP.KEYDESCRIPTOR.FREQUENCY [actual identifier includes maintenance agencies] • This is useful if one wishes to reference the component in, say, a registry scenario, but it is not normally retained in processing systems • For instance in the data structure definition the identifier of the Concept is used, but the full identifier can be constructed from its context in the structure • However the SDMX-ML specification does allow the full identifier [URN] to be specified as well • ISO 11179 equivalent name could be BOP_KEYDESCRIPTOR.FREQUENCY.CODE

  18. SDMX: Part of Key Family

  19. CMR and ISO 11179 • CMR is a model that supports the survey lifecycle • Element and Concept semantics are provided by an association to ISO 11179 e. g • Question is associated to Data Element Concept • Response Domain is associated to Value Domain • Data Set component (element) is associated to Data Element • Data set contains the output from a survey • All data elements are explicitly named • ISO 11179 is the pivot between CMR and SDMX • We need to apply ISO 11179 to SDMX

  20. Applying ISO 11179 to SDMX • ISO 11179 gives you: [Object_Class].[Property].[Representation] • An ISO 11179 instance would have: [Object_ID].[Property_Term].[Representation_Term] • We have these constructs in SDMX data and metadata sets

  21. SDMX Objects • For data, we have a limited set of object classes: • Data Sets • Groups • Series • Observations • For reference metadata, we have a larger set of object classes: • Data Providers • Category Schemes • Data and Metadata Flows • Etc. • Note that SDMX instance IDs are compound • For data, a “key” • For metadata, a “target identifier”

  22. SDMX Properties and Representations • In SDMX, properties are taken from Concepts • The names and definitions are supplied by the user for most properties • Some are required by the model • Representations are equivalent to those in ISO 11179 (code, text, etc.)

  23. Representation Term (derived from the Representation of the Data Attribute or Measure in the Data Structure Definition) Property (from the Concept that is the Data Attribute or Measure in the Data Structure Definition) Object (from the SDMX model) code [Observation_ID]. Confidentiality. Availability. [Series_ID]. code Title. text [Group_ID]. Value. number [Observation_ID]. text Title. [Data Set_ID].

  24. SDMX Object IDs • Compound object IDs in SDMX are either: • Data Keys (for Group, Series, or Observation), which depend on the data structure definition • Metadata Target Identifiers (for data provider, metadata flow, categorization scheme, etc.), which depend on the metadata structure definition • Data Examples • Observation Key: “Annual – Total Population – Kenya – 1994” • Series Key: “Annual – Total Population – Kenya” • Group Key: “Annual – Total Population” (this is the group of all countries)

  25. Examples, cont. • For the concept of “Availability” for a series, in ISO 11179 we would have: Annual – Total Population – Kenya. Availability. Code • Note that the SDMX distinction between “dimensions” and “attributes” is not important to ISO 11179 – the semantic models use the same basic approach for both • You only model the semantics of discrete, value-containing constructs: attributes and observation values

  26. Reference Metadata Object IDs in SDMX • All of the objects in the SDMX Information Model can be Object Classes for ISO 11179 • They can be combined to identify new object classes • The IDs for instances of objects are composite “target identifiers”

  27. Example of Some SDMX Object Classes CategoryScheme Structure Definition comprises subject or reporting categories uses specific data/metadata structure can be linked to categories in multiple category schemes Data Set or Metadata Set Data or Metadata Flow Category conforms to business rules of the data/metadata flow publishes/reports data sets or metadata sets can have child categories can get data/metadata from multiple data/metadata providers can provide data/metadata for many data/metadata flows using agreed data/metadata structure Provision Agreement Data Provider

  28. Reference Metadata Example • A data provider is identified with an ID which references the organization scheme (and agency) the data provider ID code comes from: [OrgSchemeAgencyID]_[OrgSchemeID]_[Data_ProviderID] • Example: sdmx_sdmxProviderScheme1_IMF shows the ID for a data provider from agency scheme 1 maintained by SDMX, identifying the IMF as a data provider

  29. Reference Metadata Example, cont. • To model a “Contact_Name” concept in an SDMX metadata report for a specific data provider, we have in ISO 11179 sdmx_sdmxProviderScheme1_IMF.Contact_Name.Text A more useful approach is possible: Data_Provider.Contact_Name.Text This is not possible for data objects in SDMX.

  30. Registry Model: ISO 11179 • But no API specification for implementing the model

  31. Registry Metamodel: SDMX • SDMX does not have an abstract registry metamodel • SDMX specifies registry interfaces based on the SDMX Information Model • Compliance with SDMX registry standards is based on implementation of the SDMX registry APIs • The actual registry implementation can be whatever the registry service provider chooses • The JEDH pilot project is implemented using an ebXML registry (this has a model which is very similar to the ISO 11179 model) • An SDMX compliant registry could use an ISO 11179 registry (if such an implementation exists)

  32. Functionality • An ISO 11179 registry is used to provide a semantic registry of specific data elements • The focus is on understanding the meaning of data and metadata • An SDMX registry provides mechanistic services to facilitate the exchange of data and metadata sets • The focus is on access to data and metadata • These are complementary functions

  33. Using the Mapping • CMR is one example of a model which could be mapped against SDMX • SDMX is about aggregate data exchange • CMR is about survey lifecycle metadata • These are potentially related, but are not the same thing • To be useful in an application, we care about the semantic equivalencies of data and metadata • These equivalencies allow us to re-use and map data between models and systems • ISO 11179 acts as a pivot

  34. ISO 11179 SDMX Semantics Semantics Structures and rules are proprietary to individual organisations Structures and rules prescribed by SDMX standards Describe SDMX semantics based on SDMX structures Organisations map between their data and metadata structures and and SDMX data and metadata structures using ISO 11179 semantics Data and Metadata Structures Data and Metadata Structures SDMX defines a conceptual model for data and metadata structures for the aggregated data and reference metadata and provides a canonical syntax representation CMR defines a conceptual model for data and metadata structures for the survey lifecycle, using the ISO 11179 model for structuring semantics

  35. Another Case Study • One use case – CMR – is not enough to validate our work • At the ISO TC-154 meeting in Vancouver we met with the designers of ISO-15000 part 5, the “Core Components Technical Specification” (CCTS) • CCTS is a modelling methodology for e-commerce, standardized in ISO TC 154 • It is based on ISO-11179 • We were asked: “Could SDMX data be modelled according to CCTS?”

  36. Why Bother? • Statistics and e-commerce do have connection points: • Customs and international trade • Payments transactions and business reporting • Some systems today map between business transactions and statistical reports • Thus, mapping between CCTS (e-commerce transactions) and SDMX (statistics) is potentially a real-world use case.

  37. The Answer is “Yes” • Based on our SDMX-to-ISO 11179 mapping: • We could express our semantics • Compare them to the e-commerce semantics • Determine the relationship between specific data elements • This was a quick, straightforward validation of our work

  38. Conclusions • ISO 11179 is very useful as a pivotal semantic model for working with other models (CMR, CCTS, etc.) • SDMX is mapped at the model level, not the metamodel level • Semantics are introduced at this level • The SDMX and ISO 11179 registries are complementary in function (semantics + access) • The structural mapping of any model depends on the ISO 11179 implementation • ISO 11179 only does semantics, not structure • Applications need both structural and semantic mapping

More Related