Restoring SAP to a different Oracle System Identifier

1. Restoring SAP to a different Oracle System Identifier David Pavicic Insurance Corporation of British Columbia

2. Agenda • About ICBC • Backup Infrastructure • NetBackup & SAP • SAP Database • SAP Backup Performance • Plans for the Future

3. ICBC • ICBC is a provincial Crown Corporation established in 1973 to provide universal auto insurance to BC motorists • By 1974 every motorist in British Columbia was insured by ICBC • ICBC is recognized as an industry leader in injury rehabilitation and in promoting road safety

4. ICBC • Six lines of Business • provide driver licensing service • establish standards for obtaining a driver’s license • anything related to vehicle licensing & insurance • regulate commercial vehicle traffic • roadside safety inspections & auditing

5. ICBC • Six lines of Business (con’t) • promotes safe driving through a series of public education efforts • Drinking & Driving Counter Attack initiatives • Road infrastructure investments • as seen on NBC’s Dateline • State Farm Ins. in the US and Ont, follow this program • recovery of money owed to the corporation • traffic violation fines and photo radar

6. ICBC • Six lines of Business (con’t) • represents the recovery side of ICBC’s business • helping our customers to recover from crashes and auto crime • automobile safety research • Head restraints • Child restraints • Mission Statement: “Helping British Columbians take the risk out of road transportation”

7. ICBC • Head Office: North Vancouver, BC, Canada • 1 Central Accident Reporting Call Center • 46 Accident Claim Centers • 70 Driver Service Centers • 34 Truck Weigh Scales • 900 Independent Insurance Broker Offices • Customer base of 2.5 million motorists • Gross revenue of $2.4 billion • About 5200 employees throughout BC

8. Unix Technical Support • Our IS environment has evolved from a purely IBM mainframe shop to include a rapidly growing distributed infrastructure • Implementation of our distributed infrastructure is primarily for the SAP, Oracle & MS Exchange • Unix Technical Support is responsible for consolidating the use of Unix for the future

9. Unix Technical Support • SAP Implementation • Finance • Project Management, Budget • Human Resources • Accounts Receivable / Accounts Payable • SAP helped consolidate the use of two mainframe systems used by ICBC & the Provincial Government • Asset Management • Futures ??? - New E-Commerce Solutions New Claims System New Insurance System • SAP makes up about half of our Unix environment

10. Unix Technical Support • How many servers? • 2 Sun Enterprise 10000 (8 Domains) • 5 Sun Enterprise 4500 • 3 Sun Enterprise 3000/4000 • 25 Sun Enterprise 450 • 15 Sun Ultra 1/2/5/10 • Running: • Sun Solaris • Veritas File System • Veritas Volume Manager • Veritas Raid Manager • Veritas NetBackup with SAP, Oracle & MS Exchange Extensions

11. Backup Infrastructure • Master Server • Sun E4000 • 2 GB of memory • 2 100 Mb Ethernet Interfaces • 2 Gb Ethernet Interfaces • Tape Library • Sun L3500 • 7 DTL7000 Drives • Manage 22 TB of data in 7 TB Library • 180 GB/hour transfer rate(compressed)

12. Backup Infrastructure • Applications & OSes we backup • SAP • Oracle • Data Warehouse • NIS & File Server • NT File Servers • MS Exchange • MS SQL • Novell • Solaris, AIX, DEC and NT • Backup about 3 Terabytes per week

13. Backup Infrastructure • Veritas Global Data Manager • Manage autonomous NetBackup Masters

14. Backup Infrastructure

15. NetBackup and SAP

16. NetBackup and SAP

17. NetBackup and SAP • Advantages • Use of Media not supported by SAP • Optical disk • Storage Libraries • Incorporate SAP in existing Backup procedures • File systems, Databases & Offsiting, BPVault • Use the Network for backups, no need for locally attached devices • SAP passes the file list to NetBackup • Disadvantages • Number of redo logs grows for duration of Backup • Dependant on how many processes are going on

18. NetBackup and SAP • Components • Backint • Sends information from SAP Tools to NetBackup • Standard Class vs SAP Class • SAP Class setup - shifts database administration from SAPdba to NetBackup • Standard Class setup - SAPdba retains database administration • We use Standard Class definitions for our SAP servers with a user backup schedule • SAP must be configured to use the parameters in the initSID.utl file for backup • See your SAP Basis team for configuration

19. NetBackup and SAP • Components(con’t) • SAP Extension Utility file - initSID.utl # backint parameters # # drives: set number of simultaneous tape drives to kickoff backup. # be sure to setup Netbackup Server to support multiple simultaneous # client backups as well as number of storage units/tape drives # for this backup class # drives 5 drives 3 # # backup class: make sure has manual backup schedule. class sapprd # # client: this machine client sappr1 # # server: name of backup server server bckpr # # schedule: name of schedule to use schedule userbkps # # schedule2: name of schedule to use for secondary backup files # this is optional if you wish to copy to different tape or server. schedule2 userbkps

20. NetBackup and SAP • SAP Extension Utility file - initSID.utl (con’t) # backint working directory backint_dir /oracle/(SID)/sapscripts/backint_work # # optional sleep time for displaying bpbackup or bprestore logs sleep 10 # # params for file_online switch_list /oracle/(SID)/sapbackup/.switch.lis switch_sem /oracle/(SID)/sapbackup/.switch.sem switch_log /oracle/(SID)/sapbackup/.switch.log # # check_directory_flag: Netbackup doesn't backup directories the # same as SAP. Set this flag to reject directory backup requests. check_directory_flag 1 # # print_log_flag - allows logs to be monitored, but output not printed print_log_flag 1 # # standard_class_flag - allows use of a standard class type for SAP clients standard_class_flag 1 # # sort_backup_type - sort_backup_type drive # # sort_restore_type - sort_restore_type drive

21. The SAP Database Problem • SAP Infrastructure - Early 1998

22. The SAP Database Problem • SAP Infrastructure - Today

23. The SAP Database Problem • Growing number of database servers • Test environment • Training environment • Development environment • Pre-Production environment • Sandbox • New project demands • Requirements to replicate database(s) for test • Y2K testing • Database issues when rolling back the date

24. Solution • Build new database servers from existing images • Populate new Oracle SID on the fly • Outcome should be two identical databases with independent SAP and Oracle attributes

25. How do you do it • Load & configure the OS, SAP, Oracle & the System Identifier for the new system • Perform a cold backup of SAP system you wish to emulate • Shut down SAP and Database • brbackup -c -u user/password • -c - confirm, runs in unattended mode • -u user/password- user performing backup

26. How do you do it • 3 areas to configure • /ORACLE_HOME/sapbackup/ • directory that contains log files for backups & restores • /ORACLE_HOME/sapscripts/ • directory that contains the SAP extension utility file & the backint working directory • /usr/openv/netbackup/bp.conf • NetBackup configuration file

27. How do you do it • Capture SAP Detailed Backup Log file • b*******.xyf • b - backup • ******* - code timestamp • x - a (all), p (partial) • y - n (online), f (offline) • f - utility file backup • Ex bdazohmg.aff - full, offline backup on Sept 27 at 0511 located in /ORACLE_HOME/sapbackup • Capture SAP Summary Backup Log file • backOSID.log, in /ORACLE_HOME/sapbackup • Transfer these files to new system • rename backOSID.log to backNSID.log

28. How do you do it • Edit /ORACLE_HOME/sapscripts/initNSID.utl on new server to replicate original server # backup class: make sure has manual backup schedule. #class sapnew class saporig # # client: this machine #client sapnew client saporig # # backint working directory backint_dir /oracle/NSID/sapscripts/backint_work # # params for file_online switch_list /oracle/NSID/sapbackup/.switch.lis switch_sem /oracle/NSID/sapbackup/.switch.sem switch_log /oracle/NSID/sapbackup/.switch.log # # standard_class_flag - allows use of a standard class type for SAP clients standard_class_flag 1 # # sort_restore_type - sort_restore_type drive

29. How do you do it • Edit /usr/openv/netbackup/bp.conf on new server to replicate original server server = bckpr #CLIENT_NAME = sapnew CLIENT_NAME = saporig VERBOSE • Change /usr/openv/netbackup/bp.conf back to original state after restore is complete server = bckpr CLIENT_NAME = sapnew #CLIENT_NAME = saporig VERBOSE

30. How do you do it • Issue the SAP restore command as an SAP user on the new server • brrestore -b b*******.aff -m full • brrestore - SAP restore utility • -b - from backup • b*******.aff - log file from backup of server emulating • -m - mode full • SAP Restore log file BR454W Values of oracle_home are different: current '/oracle/NSID', backupbdazohmg.aff '/oracle/OSID'. BR455W Value '/oracle/NSID' of oracle_home will be used for restore. BR427W All files will be restored to the new Oracle home directory! BR280I Time stamp 1999-08-27 11.10.48 BR256I Please enter 'cont' to continue, 'stop' to cancel the program: BR257I Your reply: 'cont’

31. Result Cloned SAP database server with new Oracle System Identifier properties

32. What to look out for • Use same version levels for both the SAP Extension and NetBackup • Confusion with multiple network interfaces on server, hostnames • Ensure proper permissions are granted when setting up SAP and Oracle • No formal documentation, lots of trial & error

33. SAP Backup Performance Problem • Size of the production SAP database • 190GB DB, 8 hour backup with NetBackup 3.1 • Exceeds backup window • Issues with NetBackup 3.0/3.1 and E10K • Significantly longer backup times with NetBackup 3.1 and E10K • NetBackup 3.0 is not certified Y2K compliant

34. The Solution • Splitting up the backup across multiple tape drives - reverse multi-plexing • Installing Gigabit Ethernet from end to end

35. How do you do it • Required Configuration Changes • SAP Extension Utility File, initSID.utl • Drive Parameter • drives = drives dedicated to class X MPX in schedule • 3 = 3 X 1 • NetBackup Global parameter for Maximum jobs per client • Should not be less than Drives Parameter in .utl file • Class attribute for Maximum jobs per class • Should not be less than Drives Parameter in .utl file • Install & configure Gigabit interfaces and configure switch

36. Result Size of the production SAP database today 260GB database, 3:15 hour backup with NetBackup 3.1 across 3 simultaneous streams using Gigabit Ethernet from end to end

37. What to look out for • Use same levels for both the SAP Extension and NetBackup • Job monitor displays multi-streamed jobs as multiple individual jobs • Problems when one stream fails • RESTORES • If issuing restore from the SAP system • Make sure all your partitions have enough free space • Minimum of 200 megabytes each partition • No problems when issuing the restore from NetBackup ( xbp or bp)

38. What to look out for • Sun Gigabit Cards • Auto-negotiation between Sun and Cisco • Gigabit driver is not a multi-threaded process • Sun Rule of Thumb for throughput • 1 Mhz of CPU = 1 mbps of Data • expect about 400 mbps MAX with a 400 Mhz processor

39. Plans for the Future • Roll out NetBackup 3.2 to Head Office • Install and configure new StorageTek 9740 Tape Library & a second master server • SuperDLT - 100 GB compressed (Q300) • Extend use of NetBackup to legacy NT servers • Block Level Incremental Backup • Gigabit Trunking to speed up backups for large Oracle Servers • Multi-streaming is supported for all classes with NetBackup 3.2

40. Plans for the Future • Storage Area Networks (SAN) • Fibre Channel to break the SCSI distance limitation • Server-less backups • communication from disk direct to tape without any master server intervention • Network-less backups • all storage traffic is performed on the fibre channel network, not the conventional data network • Researching the consolidation of Mainframe & Distributed System’s storage • Target NetBackup reporting tool

41. Multi-stream your Backups for large database servers Performance improvement Shrink your backup window Better utilize your backup time SAP Server rebuild strategy Fulfill Database server demands Fulfill project demands Y2K database testing Summary

42. Final Words of Wisdom... Don’t let your NetBackup partition fill up Document as much as possible Understand your environment Test your restores often