1 / 45

Release Management in the UBS Data Warehouse Program

Release Management in the UBS Data Warehouse Program. Friedrich Lehn for UBS AG, Switzerland friedrich.lehn@fhlConsult.com. Agenda. Project Overview Project Infrastructure Change Management Release Process Summary. Project Overview. UBS

norina
Télécharger la présentation

Release Management in the UBS Data Warehouse Program

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Release Managementin the UBS Data Warehouse Program Friedrich Lehn for UBS AG, Switzerland friedrich.lehn@fhlConsult.com

  2. Agenda • Project Overview • Project Infrastructure • Change Management • Release Process • Summary

  3. Project Overview UBS • Global, integrated investment services firmand leading bank in Switzerland • World’s leading provider of private banking services • Total client assets US$ 1.41 trillion in 2001 • Acquired in 2000

  4. Project Overview Data Warehouse Program (DWP) • Establish common infrastructure for analytical data processing • Provide a business oriented set of data warehouseand data mart services • Standardized business data model • Align the bank’s data mart portfolio • Improve flexibility, time-to-market and data quality

  5. Project Overview Data Flow System of Records extract, condition & load (inbound) Common Data / Business Warehouse data mart sourcing (outbound) Data Marts visualization business user

  6. Project Overview Delivery Streams • Main organizational element:team working on a subject data area / data mart • Three letter acronym as base for naming standards • Standardized infrastructure:UNIX directories, access group, meta data area, ... • Team size typically between 1 and 5 • Delivery streams release independently from each other

  7. Project Overview System of Records DSF <STR> <STR> <FDS> <FDS> ... ... DWP Sourcing Framework ... <MAR> <MAR> Application Structure RCL MDR DSF RCL release control tools MDR meta data repository DSF DWP sourcing framework <FDS> data feed definition <STR> delivery stream <MAR> outbound stream / data mart sourcing

  8. Project Infrastructure System Environments Development Test Production • IBM SP2 cluster, AIX 4.3 • DB2 UDB EEE • PowerCenter V5.1 • DWP Sourcing Framework • Cognos / Business Objects • Netscape Enterprise Webserver V3.63 • ClearCase V4.2

  9. Project Infrastructure S emergency (SOS) releases D A P delivery stream development framework test / delivery stream migration to new framework releases X F V T framework development Logical Environments and Release Structure Development Test Production mandatoryoptional

  10. Project Infrastructure Release Cycles • delivery stream development: D  A  P • framework development: T  X  F [  V ]after sign-off: X  D  A  P • migration to new frameworkreleases: {DAP}  X  F • emergency releases: P  E  P

  11. Change Management Design Principles • Support different, clearly separated environments with different responsibilities • All environments have identical structure(products, databases, server configurations) • All program changes are done on the development system • All changes on test and production systems go through the release process and are clearly tracked • “just-in-time production”

  12. Change Management Challenges • currently ~60 delivery streams • and ~500 feeds • between 10 and 7103 objects per release unit • >3500 single releases since May 2000 • >5500 deliver requests • ~1000 active installations

  13. Change Management • Two areas:/dwp_root release area, version controlled/dwp_data dynamic data, archival on demand • Additional directory level in order to support more than one logical environment on one system • /dwp_root is organized by delivery streams, e. g.:/dwp_root/d/streams/rcl/bin Directory Structure

  14. Change Management Directory Structure (continued) • /dwp_data is organized by logical processing steps, e. g.:/dwp_data/p/data/landing (landing area)/dwp_data/p/data/tgtfiles (target files)/dwp_data/p/logs/system (framework log area) • Tool support for generation of directories in source environments (delta processing) • Automatic creation of missing directories in target environments by release procedures

  15. Change Management ClearCase Set-up • One single VOB /vobs/dwp • Same directory structure as below /dwp_root, directories are automatically created • In general: only linear version trees (important: synchronization with database change management) • Branch support for emergency releases

  16. Change Management ClearCase Set-up • In general: fully automated access layer(freeze and deliver routines) • On demand (larger teams): direct ClearCase access via team specific views:/dwp_root/d/streams/rcl -> /view/rcl_team/vobs/dwp/streams/rcl • Allows use of DWP framework • Common labeling strategy (freeze -nocheckin)

  17. Change Management Release Naming • <delivery stream>_<major>.<minor>.<patch>e. g.: RCL_1.1.0 • <major> major release number (high level “wave” planning) • <minor> minor release number (delivery stream development plan) • <patch> patch level (bug fixes) • Emergency releases: RCL_1.1.0_SOS_<#>

  18. Change Management Release Attributes • Description (standard comment) • INSTALLED (list of environments where release is installed) • TRANSITIONS (describes the deployment path of a release) • RESPONSIBLE (due to centralized freeze routine) • PCRVERSION (PowerCenter meta data version) • CR (change request link) • PSO (production sign off)

  19. Change Management /main/1 RCL_1.0.0 RCL_1.0.0_sos_1 /main/sos/1 /main/2 RCL_1.0.1 RCL_1.0.0_sos_1 /main/sos/1 /main/3 RCL_1.0.2 /main/4 RCL_2.0.0 /main/5 Versioning Example common.pm

  20. Change Management Change Control Board • Responsible for high level planning and impact analysis • Defines release scope and release numbers on baseof delivery streams • Assigns responsibilities (delivery stream manager, data modeler, database administrator, business responsible) • Result is documented in “wave plan” document

  21. Release Process Roles & Responsibilities Role Responsibility developer development, unit and integration testing delivery stream manager manager, release planning database administrator database change control release manager deployment, tracking, configuration control, administration

  22. Release Process Freeze Receive Deliver Process Overview Development Test Production ClearCase Deployment Package

  23. Release Process Release Objects • PowerCenter mappings • Job dependency data • scripts / SQLs / programs and source code in general not included: • Database objects (in concept phase) • documentation (intranet database)

  24. Release Process Release Procedure Responsible 1. Submit database change request *) delivery stream manager 2. Implement database changes *) database administrator 3. Prepare release area delivery stream manager (UNIX, PowerCenter, job dependencies, Uniserv) 4. Submit release request delivery stream manager 5. Prepare release area (DDLs) *) database administrator 6. Create new release (Freeze) release manager 7. Create deployment package (Deliver) release manager 8. Apply database changes to target system *) database administrator 9. Install release in daily deployment window release manager(Receive) (IT integration / IT operation) *) in case of database changes only

  25. Release Process Freeze Process freeze <release unit> [ -patch | -minor | -major | -sos <release> ] 1. Retrieve previous release (“element * RCL_1.0.0”) 2. Compare with release area (check for new, changed, deleted files) 3. Display results and ask for confirmation 4. Apply changes in ClearCase 5. Create release label and set release attributes 6. Attach release label

  26. Release Process Deliver Process deliver <release unit> -t <target env.> [ -r <release> ] [ -a ] 1. Retrieve specified / latest release in ClearCase 2. Retrieve current release in target environment and create delta 3. Use -a(ll) for initialization / synchronization 4. Create deployment package (tar file + control file) 5. Update release attributes (predecessor and current release!) 6. Lock release label

  27. Release Process Receive Process receive 1. Check hand-over area for pending releases 2. Remote copy deployment package to target system 3. Install it 4. Standardized post-installation steps: e. g. access permissions 5. Delivery stream defined post-installation steps (PostInstall.ksh file):e. g. for non-standard path names, generators, setuid bits

  28. Release Process Delta Deployment • Smaller packages (“top 1” feed: > 7000 files) • clear change log • both ways: creation and removal of files

  29. Release Process Release Cache Question: Which is the predecessor release in a specific target? Installation information is stored with each release label.  cleartool lstype -kind lbtype cleartool desc lbtype:RCL_1.0.0 Problem: takes ages with 3500 release labels!

  30. Release Process Release Cache Solution 1:Manage installation information on element version level + delta processing easy – label twice (release and installation label)– missing overview, need log files for tracking– error prone (e. g. interruption of freeze process)

  31. Release Process Release Cache Solution 2:Release cache (including all release attributes,master information remains in ClearCase) – additional software layer required (consistency!) + clear information structure+ efficient delta processing+ added value: release database

  32. Release Process Release Database

  33. Release Process Release Database (Details)

  34. Release Process Release Request (1)

  35. Release Process Release Request (2)

  36. Release Process Release Request (3)

  37. Release Process Release Tracker (Overview)

  38. Release Process Release Tracker (Details)

  39. Release Process Releasing Meta Data • PowerCenter, job dependency and feed configurationmeta data are stored in the database • Unload utilities to export the corresponding dataand store it in the delivery stream´s release area (tar file) • Meta data is frozen and delivered together with all otherfile system objects • Load utility for loading objects in target environment • e. g.: meta_pcr <delivery stream> [ unload | load ]

  40. Release Process Database Change Management (Concept Phase) • PATROL DB-Change Manager by bmcsoftware • Scope filter: assign database objects to delivery stream( view in ClearCase) • Apply database changes to development database • Create release baseline:freeze all database object versions for a delivery stream( label in ClearCase, name equal to release label) • Export DDL to release area(for documentation & change detection)

  41. Release Process Database Change Management (target system) • Target baseline: target database version(delivery stream plus timestamp as name) • create delta DDL depending on release baselineand latest target baseline (PATROL) • apply delta DDL • create new target baseline

  42. Release Process Database Retrofitting Process • Rationale: certain database changes have to be applied and tested directly on the Production system (load performance optimization: indexes, summary tables, ...) • In order to include target system changes into next regular release, changes are promoted back to Development using the Retrofitting Process • In principle: new, database administrator driven release in ClearCase and PATROL that is not delivered

  43. Summary Experiences • Over 3500 releases since May 2000 • Effort for creation and installation of new release:5 - 60 minutes depending on amount of meta data(without database changes) • Main effort necessary for handling of meta data • No ClearCase problem encountered so far

  44. Friedrich H. Lehnfriedrich.lehn@fhlConsult.comwww.fhlConsult.com Thank You! ©2002 Friedrich Lehn - All rights reserved

  45. Questions? ©2002 Friedrich Lehn - All rights reserved

More Related