Presentation Transcript

1. NIEM and XML for Architects and Developers

   Delivering NIEM-aligned data services

   David Webber, Information Architect, Oracle Public Sector
2. The following is intended to outline Oracle general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. DisclaimerNotice
3. Agenda Background - What are the business drivers and needs? NIEM today – Technical Introduction The Delivery Life Cycle – Lessons Learned Software Delivery - Anatomy of a NIEM “IEPD” Tooling to the Rescue! Key Pillar of Dictionaries Summary
4. Business Drivers and Needs Background
5. XML is everywhere… PMIX – Prescription Drug information sharing Elections Results Reporting Pharmacy and Vaccines locating COBie Buildings Information sharing PESC – Student Post-Secondary Education City 311 Services Vehicle spare parts inventory Public Safety information sharing Healthcare services delivery coordination eDevice control templates
6. Public Sector Challenges Cost of building information exchanges Reduction of development sunk costs through minimizing coding tasks Allowing rapid prototyping and proof of concepts (agile development) Flexibility - dynamic adaptable templates instead of rigid fixed code Reuse – templates for common systems / solutions easily shared and adapted Government transparency and public information sharing Structured content from data stores – e.g. election results reporting, monthly cost reports, scheduled road maintenance; obvious metadata and semantics Simple search requests on data – e.g. part numbers, codes, licenses lookup Not intended for unstructured and textual content document delivery Practical real world data sharing anywhere Direct simple tools with short learning curve and plug and play deployment
7. Technology Capability Requirements Delivering on the Open Data Digital Strategy Vision for government Open Standards and Open Source based Plug and play with code-free templates and rapid development Supports NIEM and Open XML exchanges Leverages deployed technology today Workflow integration Secure messaging delivery and partner management Delivered across-platform and device End-to-end security & governance
8. Technical Introduction NIEM today
9. The NIEM Framework NIEM connects communities of people who share a common need to exchange information in order to advance their missions, and provides a foundation for seamless information exchange between federal, state, local, and tribal agencies. Much more than a data model, NIEM offers an active user community as well as a technical and support framework. Community Technical Framework Support Framework Formal Governance Processes Data Model Tools for Development and Discovery Established Training Program Online Repositories XML Design Rules Mission-Oriented Domains Implementation Support Development Methodology Predefined Deliverables (IEPD) Self-Managing Domain Stewards Help Desk & Knowledge Center
10. The NIEM Data Model NIEM’s data model is a set of common, controlled,and approved XML data structures and definitions vetted through the Federal, State, Local, Tribal and Private Sectors. Data elements are organized into core and domain-specific components Core components are used by multiple domains and can be described by structure, semantics, and definition universally Domain-specific components are continually updated by subject matter experts that are actual NIEM participants and industry experts for their particular domain NIEM Naming and Design Rules (NDR) specify how each of these components are defined and utilized
11. NIEM’s Domains View NEWEST DOMAINSNIEM Government ResourceManagement Domain NIEM Health Domain NIEM Human Services Domain DOMAINS IN PLANNINGNIEM Agriculture NIEM Education NIEM Transportation 11
12. Available XML Dictionaries NIEM 2.1 dictionaries CBRN dictionary Emergency dictionary Family dictionary Immigration dictionary Infrastructure dictionary Intelligence dictionary Justice dictionary Maritime dictionary Screening dictionary Trade dictionary Immigration blueprint NIEM core dictionary Note: Those marked in bold are model style dictionaries with recursive components. Available from download site direct link: http://sourceforge.net/projects/camprocessor/files + includes spreadsheets and sample models XML XML XML XML XML XML
13. XML Schema Industry Domain schemas *Code list xsd Code list xsd xsd subset Exchange xsd LEGEND: * Optional *Extension xsd *Constraint xsd *Restriction xsd Namespace NIEM uses W3C Schema Collection XSD Collection NIEM Exchange Schema
14. NIEM Past and Present Challenges Vertical domain vocabulary for DHS and DOJ One dozen domain dictionaries Technology limited - built using XSD schema Core components highly contextual to DHS/DOJ Surprisingly – no actual logical models of information! Gap with semantic technologies integration Limited data content rules and code lists Mapping automation support missing Multi-year development life cycles
15. NIEM is still improving… Collections of complex XML Schema Verbose components Embedded context in names Currently facing significant scaling challenges Inconsistencies; too much manual management; slow lifecycles Dictionary technology incubating Enhanced code lists mechanism incubating UML profile is evolving initiative with OMG Attempt to marry modelling techniques and XSD Schema syntax Policy and Security mechanisms incubating Need to embrace enabling tools for rapid expansion of domains and community collaboration
16. How to build NIEM solutions Lessons learned Foundation Pillars Delivery LIFECYCLE
17. The 8 “D”s and NIEM NIEM IEPD Process Design Develop Deploy Document Dictionaries Discovery Differentiate Diagnose Repeatable, Reusable Process (Exchange Specification Lifecycle) *IEPD - Information Exchange Package Documentation
18. NIEM IEPD / Exchange Delivery Lifecycle Dictionaries Discovery Requirements Updates Design Drag and Drop Visual Designer Diagnose Develop XML Samples Exchange Templates Differentiate 1 Production Results XSD Schema Deploy Document 2 NIEM IEPD Reports 3 Validated Templates / Schema XMI / UML Models 4 Required IEPD artifacts Documentation 5
19. Integrating into your existing processes and control Critical to manage and steer use of NIEM Ensure business outcomes match requirements Lessons learned with NIEM The earlier in the software development process that NIEM is introduced the better the overall outcomes Each step has deliverables – business plan, delivery architecture, data model, alignment reports, schema, test results that can be monitored in your SDLC* Monitoring products reinforces reuse as best practice Tools essential in review processes Collaboration tools can coordinate artifacts across projects and participants Test bed tools and open APIs reduce integration alignment costs *SDLC – Software Development Life Cycle
20. Anatomy of a NIEM “IEPD” It is not just all about the XSD schema Software Delivery
21. IEPD Components & Requirements IEPD MPD Main Document IEPD IEM <Exchange_Schema/> Catalog In order to be NIEM-conformant, the IEPD must adhere to: NIEM Conformance Document NIEM Naming and Design Rules (NDR) v1.3 NIEM Model Package Description (MPD) Specification v1.0 <Extension_Schema/> Change Log <Subset_Schema/> Domain Schema(s) NIEM Core Schema(s) Sample XML Instance
22. Formal NIEM IEPD Packaging Details Generated using CAM Never Used Word Document Manual Entry
23. XML Technology Components NIEM: need to manage complete aspects of exchange delivery, not just the structure and content data model
24. Swim Lane View This is what NIEM exchanges are delivered with on-the-wire! UML Technologies W3C Schema XML Technologies Web UML XML HTML XSD Namespace XQuery XMI Patterns XPath XSLT JavaScript Enumerations Associations Elements JSON Cardinality Stereotypes Attributes Facets REST Inheritance Content Complex Types SQL SOAP Objects Audience: Web application developers Audience: Data Modellers / Designers Audience: Software Developers and Middleware Engineers Audience: General content delivery and applications; content engineers; rules representation systems; multimedia delivery; geospatial systems Open-XDX 24
25. Accelerating and Improving the NIEM process Tooling to the Rescue
26. IEPD Delivery Tasks / Tools Matrix In order to be NIEM-conformant, the IEPD must adhere to: NIEM Conformance Document NIEM Naming and Design Rules (NDR) v1.3 NIEM Model Package Description (MPD) Specification v1.0
27. CAM Toolkit for NIEM Concepts behind the Content Assembly Mechanism approach Understanding the steps involved in top down / bottom up exchange design, testing and delivery Applicability to NIEM and IEPD / MPD delivery * NIEM – National Information Exchange Model * IEPD – Information Exchange Package Documentation *MPD – Model Package Description
28. CAM Template CAM Templates – WYSIWYG Exchange Structure Rules Documentation http://www.oasis-open.org/committees/cam
29. XML Schema Industry Domain schemas Code list xsd *Code list xsd xsd subset Exchange xsd LEGEND: * Optional CAM Template *Extension xsd *Constraint xsd *Restriction xsd Namespace CAM and W3C Schema Wizards Exchange Schema Structure XSLT tools process entire XSD collection XSD Collection Rules INGEST EXPORT Documentation
30. CAM Editor – Main Screens http://www.cameditor.org
31. CAM Toolkit Features Dictionary “Drag and Drop” of components to build exchanges Visual editor and structure editing Cross-platform deploy - Windows, Ubuntu/LINUX, Mac; 32/64bit Mind map models – integration with Freemind tool UML/XMI export – import to UML tools / ArgoUML Code list table lookups – rule based and selectable This allows people to fully integrate their exchange development from dictionaries through to working deployed exchange schema.
32. XML Exchange Package Artifacts Template Evaluation Report Naming and Design Rules assessment Interoperability factors review Spell checking Comparison to Dictionary Reuse report, cross-reference spreadsheet, subset “wantlist” Generate Schema Bundle with layout styles (e.g. NIEM) Realistic test samples generation Testing with runtime validation engine (CAMV) Business information exchange documentation Models – XMI and mind maps
33. CAM Template Evaluation Report The purpose of the CAM template evaluation report is to provide a suite of checks against a template that are difficult, tedious or otherwise to determine by manual inspection of a template structure and its associated rules and annotation details The report covers the following areas: alerts users to potential issues with their template and provides suggestions for improving the functional details addresses logical and conceptual issues that simple syntax checking alone does not flag can be used as part of a formal review process to ensure a baseline for quality control
34. CAM Toolkit Capabilities
35. Online Tutorials – www.niemtrainingvideos.org
36. Introduction and Concepts KEY Pillar of Dictionaries
37. The Semantic Framework and Vision XML Technologies UML Technologies XML/NIEM Components in Neutral Dictionary Representation Rules Technologies W3C Schema Tools HTML 5 rendering tools Semantic Representations Security solutions Middleware solutions
38. Dictionaries: Key Pillar of Information Sharing Components Canonical Dictionary Collections Canonical XML Components Dictionary 1 XML User Inserts Navigation and Query Tools Relationship Lookups Component Associations and Couplings 2 Domains Reuse Library Exchange Templates and Rules 3 Artefact relationships Catalog Template Presentation Automatic Rendering Tools W3C Schema and Model Representations 4 Collaboration Services XML Schema Components Models Delivery Control, Messaging, Security Deployment Environments and Middleware 5 Testing Workbench, Rules Engine, Data Samples, Integration ETL Implementation Artifacts and Examples 6
39. Dictionaries Semantics Approach Abstract dictionary representations with strong semantics, rules and code lists support Schema, models, documentation are generated from dictionary definitions automatically Contextual mechanisms allow linear number of true core components – not exponential growth Easier for practitioners to leverage with more predictable and repeatable results Enable domain use horizontally across government ETL mapping automation support Continuous collaboration environment
40. Developing domain dictionaries (EIEM) Allows domains to manage their components libraries Provides consistency for project development teams Sets of NIEM consistent XML exchange components Aligned to enterprise data stores Optimized for reuse and interoperability Save time and effort across the enterprise Perennial question for developers – when should I use NIEM components, and when our own local ones? Are there components already available for that purpose? Provide formal mechanisms and procedures to share components and collaborate across SDLC process Provide external parties consistent data views *EIEM – Enterprise Information Exchange Model
41. EIEM/BIEC Business Information Exchange Components (BIEC) NIEM-conforming XML schema data component definition that meets a particular recurring business requirement for an enterprise Enterprise Information Exchange Model (EIEM) NIEM-conforming schemas that define data components to be reused in IEPDsdeveloped by an enterprise; collection of enterprise BIECs organized into a subset and one or more extension schemas
42. Managed Dictionaries – Authoritative Sources Centralized SFIS Values Library Service, NIEM, Metadata Registry schema Distributed schema ALLOWED VALUES Publish schema SQL schema Managed Dictionaries
43. Next Steps Summary
44. Why CAM and NIEM? “Can you give us a brief overview of how you expect IT professionals and line of business managers to use CAM*?” – Van Ristau, CTO DLT Inc. *CAM editor toolkit – http://www.cameditor.org and OASIS Content Assembly Mechanism standard – http://docs.oasis-open.org/cam
45. Response Line of business managers can use CAM to drive better alignment to NIEM and faster more consistent and predictable delivery of NIEM information exchanges CAM provides tools that allow business analysts to work collaboratively with software developers to build exchanges Teams use CAM to perform the 8 “D’s” tasks when implementing NIEM exchanges CAM automates the delivery of key XML artifacts for NIEM exchange services XML validation can be deployed using the CAMV engine Open-XDX provides rapid flexible open data web services
46. Public Sector Open Data Services DeliveryOpen Platform Solutions Open Data Test Bed Services Dictionaries Security Open-XDX / CAM Editor SQL DB XML Repository Open XDX Dictionary Templates XACML SQL XML CAMV Web Visual Editor SQL Browser Dictionaries Drag and Drop Query/Response Generate Data Validate Data Send Data JAX-WS soapUI CAM Editor Templates Policy Rules XACML CAM Editor Content Models OER Integration Web Forms UI
47. Technology Components and Features Providing open data sharing Open-XDX component accelerates deployment of interfaces Aligning meaning and definitions Collaboration using XML component dictionaries and NIEM Formal information verification and content rules CAMV rules engine provides runtime handling Industry domain test bed facilities Rules documentation and sharing Business process integration Oracle BPM and Forms combined with CAMV and Open-XDX In progress – Security profiles and policy templates Upcoming release of CAM Editor and XACML integration
48. Cloud Services Deployment Web site and collaboration tools Registry for Dictionaries Open-XDX web services Security Profile Templates CAMV test bed for industries CAMV interactive testing Security solutions Semantic Representations XML Technologies Rules Technologies W3C Schema Tools Policy Rules Middleware solutions
49. CAMeditor.ORG Project Statistics SNAPSHOT OF PROJECT ACTIVITIES135,000+ CAMeditor.org page visits to site 165+ countries have downloaded tools; 30% of visitors are from U.S.; 800+ downloads weekly 2000+ student views of online video training resources 8 languages now available www.cameditor.org www.niemtrainingvideos.org