CHAPTER 7: ARCHITECTURAL COMPONENTS

1. CHAPTER 7: ARCHITECTURAL COMPONENTS

2. CHAPTER OBJECTIVES • Understand data warehouse architecture • Examine how the architectural framework supports the flow of data • Study the functions and services of the architectural components • Revisit the five major architectural types

3. UNDERSTANDING DATA WAREHOUSE ARCHITECTURE • Architecture: Definitions • Architecture in Three Major Areas • ARCHITECTURAL FRAMEWORK • Architecture Supporting Flow of Data • The Management and Control Module • TECHNICAL ARCHITECTURE • Data Acquisition • Data Storage • Information Delivery • ARCHITECTURAL TYPES • Centralized Corporate Data Warehouse • Independent Data Marts • Federated • Hub-and-Spoke

4. UNDERSTANDING DATA WAREHOUSE ARCHITECTURE We were introduced to the building blocks of the data warehouse. At that stage, we quickly looked at the list of components and reviewed each very briefly. In this chapter, we want to review the data warehouse architecture from different perspectives. You will study the architectural components in the order in which they enable the flow of data from the sources as business intelligence to the end-users. Then you will be able to look at each area of the architecture and examine the functions, procedures, and features in that area.

5. Architecture: Definitions • The structure that brings all the components of a data warehouse together is known as the architecture. • In your data warehouse, • The architecture includes the integrated data that is the centerpiece. • The architecture includes everything that is needed to prepare the data and store it. • The architecture includes all the means for delivering information from your data warehouse. • The architecture is further composed of the rules, procedures, and functions that enable your data warehouse to work and fulfill the business requirements. • The architecture is made up of the technology that empowers your data warehouse. • What is the general purpose of the data warehouse architecture? • The architecture provides the overall framework for developing and deploying your data warehouse; it is a comprehensive blueprint. • The architecture defines the standards, measurements, general design, and support techniques.

6. Architecture in Three Major Areas • As you already know, the three major areas in the data warehouse are: • Data acquisition • Data storage • Information delivery Figure 7-1 groups these major architectural components into the three areas.

7. ARCHITECTURAL FRAMEWORK Architecture Supporting Flow of Data This collection of data from the various sources moves to the staging area. What happens next? The extracted data goes through a detailed preparation process in the staging area before it is sent forward to the data warehouse to be properly stored. From the data warehouse storage, data transformed into useful information is retrieved by the users or delivered to the user desktops as required. Figure 7-2 shows the flow of data from beginning to end and also highlights the architectural components enabling the flow of data as the data moves along.

8. Figure 7-2 Architectural framework supporting the flow of data.

9. The Management and Control Module • This architectural component is an overall module managing and controlling the entire data warehouse environment. It is an umbrella component working at various levels and covering all the operations. • Major functions: • Monitor all the ongoing operations • Recover from problems when things go wrong. • Manages and controls the data acquisition functions, ensuring that extracts and transformations are carried out correctly and in a timely fashion. • Manages backing up significant parts of the data warehouse and recovering from failures. • Monitoring the growth and periodically archiving data from the data warehouse. • Governs data security and provides authorized access to the data warehouse. • Interfaces with the end-user information delivery component to ensure that information delivery is carried out properly. • Figure 7-3 shows how the management component relates to and manages all of the data warehouse operations.

10. Figure 7-3 The management and control component.

11. TECHNICAL ARCHITECTURE • Let us now consider the technical architecture in each of the three major areas of the data warehouse , Data Acquisition, Data Storage, Information Delivery. • Data Acquisition • This area covers the entire process of extracting data from the data sources, moving all the extracted data to the staging area, and preparing the data for loading into the data warehouse repository. Figure 7-4 Data acquisition: technical architecture.

12. 1) Data Flow Flow In the data acquisition area, the data flow begins at the data sources and pauses at the staging area. After transformation and integration, the data is ready for loading into the data warehouse repository. Data Sources Usually, these systems are supported by relational DBMSs. Here you may use an SQL-based language for extracting data. A fairly large number of companies have adopted ERP (enterprise resource planning) systems. ERP data sources provide an advantage in that the data from these sources is already consolidated and integrated. For including data from outside sources, you will have to create temporary files to hold the data received from the outside sources. After reformatting and rearranging the data elements, you will have to move the data to the staging area.

13. Intermediary Data Stores As data gets extracted from the data sources, it moves through temporary files. Sometimes, extracts of homogeneous data from several source applications are pulled into separate temporary files and then merged into another temporary file before moving it to the staging area. Staging Area This is the place where all the extracted data is put together and prepared for loading into the data warehouse. The staging area is like an assembly plant or a construction area. In this area, you examine each extracted file, review the business rules, perform the various data transformation functions, sort and merge data, resolve inconsistencies, and cleanse the data. When the data is finally prepared either for an enterprise-wide data warehouse or one of the conformed data marts, the data temporarily resides in the staging area repository waiting to be loaded into the data warehouse repository.

14. 2) Functions and Services • The list of functions and services in this section relates to the data acquisition area and is broken down into three groups. This is a general list. It does not indicate the extent or complexity of each function or service. For the technical architecture of your data warehouse, you have to determine the content and complexity of each function or service. • List of Functions and Services • Data Extraction • Generate automatic extract files from operational systems using replication and other techniques. • Create intermediary files to store selected data to be merged later. • Transport extracted files from multiple platforms. • Reformat input from outside sources. • Reformat input from departmental data files, databases, and spreadsheets. • Generate common application codes for data extraction.

15. Data Transformation • Map input data to data for data warehouse repository. • Clean data, de-duplicate, and merge/purge. • Convert data types. • Calculate and derive attribute values. • Aggregate data as needed. • Resolve missing values. • Consolidate and integrate data. • Data Staging • Provide backup and recovery for staging area repositories. • Sort and merge files. • Create files as input to make changes to dimension tables. • If data staging storage is a relational database, create and populate database. • Resolve and create primary and foreign keys for load tables. • If staging area storage is a relational database, extract load files.

16. Data Storage This covers the process of loading the data from the staging area into the data warehouse repository. All functions for transforming and integrating the data are completed in the data staging area. Figure 7-5 shows a summarized view of the technical architecture for data storage. Figure 7-5 Data storage: technical architecture.

17. 1) Data Flow Flow For data storage, the data flow begins at the data staging area. The transformed and integrated data is moved from the staging area to the data warehouse repository. If the data warehouse is an enterprise-wide data warehouse being built in a top-down fashion, then there could be movements of data from the enterprise-wide data warehouse repository to the repositories of the dependent data marts. • Data Groups • Prepared data waiting in the data staging area fall into two groups. • The first group is the set of files or tables containing data for a full refresh. This group of data is usually meant for the initial loading of the data warehouse. Occasionally, some data warehouse tables may be refreshed fully. • The other group of data is the set of files or tables containing ongoing incremental loads. • The Data Repository • Almost all of today’s data warehouse databases are relational databases. All the power, flexibility, and ease of use capabilities of the RDBMS become available for the processing of data.

18. 2) Functions and Services • The general list of functions and services given in this section is for your guidance. The list relates to the data storage area and covers the broad functions and services. This is a general list. It does not indicate the extent or complexity of each function or service. For the technical architecture of data warehouse, you have to determine the content and complexity of each function or service. • List of Functions and Services • Load data for full refreshes of data warehouse tables. • Support loading into multiple tables at the detailed and summarized levels. • Optimize the loading process. • Provide automated job control services for loading the data warehouse. • Provide backup and recovery for the data warehouse database. • Provide security. • Monitor and fine-tune the database. • Periodically archive data from the database.

19. Information Delivery This area spans a broad spectrum of methods for making information available to users. For your users, the information delivery component is the data warehouse. They do not come into contact with the other components directly. For the users, the strength of your data warehouse architecture is mainly concentrated in the flexibility of the information delivery component. The information delivery component makes it easy for the users to access the information either directly from the enterprise-wide data warehouse, from the dependent data marts, or from the set of conformed data marts. Almost all modern data warehouses provide for online analytical processing (OLAP). In this case, the primary data warehouse feeds data to proprietary multidimensional databases (MDDBs) where summarized data is kept as multidimensional cubes of information. The users perform complex multidimensional analysis using the information cubes in the MDDBs. Refer to Figure 7-6 for a summarized view of the technical architecture for information delivery.

20. Figure 7-6 Information delivery: technical architecture.

21. 1) Data Flow Flow For information delivery, the data flow begins at the enterprise-wide data warehouse and the dependent data marts when the design is based on the top down technique. When the design follows the bottom-up method, the data flow starts at the set of conformed data marts. Generally, data transformed into information flows to user desktops during query sessions. Sometimes, the result sets from individual queries or reports are held in proprietary data stores of the query or reporting tool vendors. Recently progressive organizations implement dashboards and scorecards as part of information delivery. Dashboards are real time or near real time information display devices.

22. Service Locations • In your information delivery component, you may provide query services from the user desktop, from an application server, or from the database itself. This will be one of the critical decisions for your architecture design. • Data Stores • For information delivery, you may consider the following intermediary data • stores: • Proprietary temporary stores to hold results of individual queries and reports for repeated use • Data stores for standard reporting • Proprietary multidimensional databases

23. 2) Functions and Services • Review the general list of functions and services given below and use it as a guide to establish the information delivery component of your data warehouse architecture. The list relates to information delivery and covers the broad functions and services. Again, this is a general list. It does not indicate the extent or complexity of each function or service. For the technical architecture of your data warehouse, you have to determine the content and complexity of each function or service. • Provide security to control information access. • Monitor user access to improve service and for future enhancements. • Allow users to browse data warehouse content. • Simplify access by hiding internal complexities of data storage from users. • Automatically reformat queries for optimal execution. • Govern queries and control runaway queries. • Provide self-service report generation for users, consisting of a variety of flexible options to create, schedule, and run reports. • Store result sets of queries and reports for future use. • Make provision for the users to perform complex analysis through online analytical processing (OLAP).

24. ARCHITECTURAL TYPES • 1) Centralized Corporate Data Warehouse In this architecture type, a centralized enterprise data warehouse is present. There are no data marts, whether dependent or independent. Therefore all information delivery is from the centralized data warehouse. Figure 7-7 Overview of the components of a centralized data warehouse.

25. 2) Independent Data Marts In this architecture type, the data warehouse is a collection of unconnected, disparate data marts, each serving a specific department or purpose. Each data mart delivers information to its own group of users. Figure 7-8 Overview of the components of independent data marts.

26. 3) Federated In the federated architectural type, common data elements in the various data marts and even data warehouses that compose the federation are integrated physically or logically. The goal is to strive for a single version of truth for the organization; a centralized enterprise data warehouse is present. There are no data marts, whether dependent or independent. Therefore all information delivery is from the centralized data warehouse. Figure 7-9 Overview of the components of a federated data warehouse.

27. 4) Hub-and-Spoke In this architecture type, a centralized enterprise data warehouse is present. In addition, there are data marts that depend on the enterprise data warehouse for data feed. Information delivery can, therefore, be both from the centralized data warehouse and the dependent data marts. Figure 7-10 Overview of the components of a hub-and-spoke type of data warehouse.

28. CHAPTER SUMMARY • Architecture is the structure that brings all the components together. • Data warehouse architecture consists of distinct components with the read-only data repository as the centerpiece. • A few typical data warehouse architectural types are in use at various organizations. Broadly these types reflect how data is stored and made available—centrally as the single enterprise data warehouse database or as a collection cohesive data marts. • The architectural components support the functioning of the data warehouse in the three major areas of data acquisition, data storage, and information delivery. • Data warehouse architecture is wide, complex, expansive, and has several distinguishing characteristics. • The architectural framework enables the flow of data from the data sources at one end to the user’s desktop at the other. • The technical architecture of a data warehouse is the complete set of functions and services provided within its components. It includes the procedures and rules needed to perform the functions and to provide the services. • The flow of data from the source systems to end-users as business intelligence depends on the architectural type.