NIST BIG DATA WG Reference Architecture Subgroup Intermediate Report

1. NIST BIG DATA WGReference Architecture SubgroupIntermediate Report Co-chairs: Orit Levin (Microsoft) James Ketner (AT&T) Don Krapohl (Augmented Intelligence) August 21, 2013

2. Starting Point From the Previous Joint Call • Transformationincludes • Processing functions • Analytic functions • Visualization functions • Data Infrastructureincludes • Data stores • In-memory DBs • Analytic DBs Sources Transformation Data Infrastructure Security Cloud Computing Management Network Usage NIST Big Data WG / Ref Arch Subgroup

3. The Latest RA Diagram NIST Big Data WG / Ref Arch Subgroup

4. RA Current View • The current RA view is technological, meaning that • It doesn’t show topology or deployment models • It doesn’t show business stakeholders or business relationships • By using different sets of taxonomies, the same baseline diagram can be modified to show topologies or business relationships • These different views would need to be shown separately (e.g., technologies vs. deployment models vs. business stakeholders) • A proposal: Discuss these aspects in the context of the RoadMap scope and direction and sync on the direction and responsibilities NIST Big Data WG / Ref Arch Subgroup

5. Deliverables • Delivery #1: White Paper describing different RA approaches • The proposed Outline is in M0150 • Call for review • Call for volunteers to produce content for different sections • Delivery #2: Common RA Draft • The proposed Outline is in M123 • Call for additional volunteers to produce content for different sections • A number of participants volunteered already, more are welcome NIST Big Data WG / Ref Arch Subgroup

6. Deliverable #2:Dependencies on Other Subgroups 2 Big Data Reference Architecture: Use Cases Requirements • This section will be prepared by the NIST BDWG Requirements SG and will contain high level requirements relevant to the design of the Reference Architecture. 6 Big Data Reference Architecture: Security and Privacy • This section will be prepared by the NIST BDWG Security and Privacy SG and will contain high level security and privacy considerations relevant to the design of the Reference Architecture. 7 Big Data Taxonomy • This section will be prepared by the NIST BDWG Def&Tax SG and will contain high level taxonomy relevant to the design of the Reference Architecture. NIST Big Data WG / Ref Arch Subgroup

7. Backup Slides NIST Big Data WG / Ref Arch Subgroup

8. Reference Architecture Objectives(No Change) • Addresses a broad range of stakeholders (e.g., data owners, industries, academia, policy makers) • Wide scope: • Encompasses the whole data life cycle or in the ecosystem • Can be applied to different use cases (including various verticals) • Represents different system architectures (e.g., an enterprise data warehouse, distributed cloud-based system using multiple service providers) • Focus • Potentially with initial focus on the Big Data analytics and tools • Assists in identifying security and privacy issues • Agnostic to any specific technologies NIST Big Data WG / Ref Arch Subgroup

9. Input for Discussion on the BD RANext Level of Details • The next slides are based on various input documents to NIST BD subgroups • They are included here to provide insight on the evolution of the NIST BD RA work NIST Big Data WG / Ref Arch Subgroup

10. Proposal I: Highlighting the Different DB Approaches Sources Data Infrastructure VOLUME VARIETY VELOCITY Collection Streaming Transformation Aggregation Curation Integration Buffer Manager Data Manager Relational DB Pre-analytics Transfer Security Cloud Computing Management Network Search Storage (disk, memory, etc.) Specialized Abstractions Analytics Statistics RT Analytics Data Mining NoSQL DB File System Map Reduce Interactive An. Visualization Batch Analytics Export Usage NIST Big Data WG / Ref Arch Subgroup

11. Proposal II: Based on Submitted Requirements NIST Big Data WG / Ref Arch Subgroup