B1: After Imaging

1. B1: After Imaging New features. Greater flexibility. Richard Banville Fellow

2. Agenda Basics New features Online implementation Archival and Maintenance

3. Agenda Basics What is after imaging? Why use after imaging? How does it work? What is the AI archiver?

4. What is after imaging? A journal of transaction data that can replay changes to a database Referred to as a redo log Granularity of recovery To last completed transaction To a point in time To a specific transaction number. Space is not reused without interaction (as it is in the before image journal files)

5. Why use after imaging? Protection Removes single point of failure Media loss Human “error” Table drop File deletion No data loss High availability & Business Continuity For replication Manual or OpenEdge Replication

6. How does it work? • Transaction data synchronously written to an after image (AI) journal • AI journal “replayed” in recovery scenarios • Not automatically enabled • Automation • Some automatic maintenance features • Recovery not automated

7. How the AI File Works FOR EACH customer: UPDATE customer. END. Transaction Begin BI Note Written AI Note Written BI and AI Transaction End Notes Written *** Data updated “sometime” later

8. Forward Processing - Physical View Shared Memory Database Before Image After Image After Image After Image 3 + t 1 2 Full AI files must be managed

9. Forward Processing - Physical View Shared Memory Database Before Image After Image After Image After Image 3 + t 1 2

10. The Recovery Process Backup Database Before Image After Image After Image After Image Remote copy 1a 1b 3 thru n roll forward 2 Redo

11. Agenda New Features • Online AI functionality • Automatic AI file management • AI archiver

12. New after imaging features (10.1A thru 10.1C) Online AI functionality Add extents Start/Stop (enable/disable) Automatic archiving Archive extents Change archive directory Change archive interval Manage AI buffers Guarded recovery process Online features are a huge advance in AI’s flexibility

13. What is the AI archiver? Daemon (AIMD) to manage/move “full” AI files Timer mode (every <n> seconds) On-demand mode (when full) Archives & marks AI extents for reuse Enablement Offline via rfutil Online via the probkup utility Additional online features Switch archive locations Change archive interval No support for archiving to a second host No support for single user activity Introduced in 10.1A

14. AI Archiver Shared Memory BI DB AI Archive Daemon Knows when and where to archive Archive 1 A1 Busy Empty Full Archive 2 Busy Empty A2 Archive 3

15. Agenda Creating AI files Enabling AI Starting and managing the AI archiver Implementing After Imaging Online

16. Creating AI files New Syntax Notes: Ensure structure file correct before adding Defaults to 8K AI block size Ensure default user permissions are correct prior to adding extents Adding extents online prostrct addonline<db><st-file-name> prostrct addonline<db><st-file-name> -validate

17. Creating AI files • Q: How many AI files to create? • A: More than one • A2: Everyone has a different opinion • 10 is a good starting point • Extent type: variable, fixed or vixed? • Depends on archive mechanism • Variable – “easier” to maintain • Fixed for “ultimate” performance • Makes some sense with fixed amount of work

18. After Image File and Archive Locations Location, location, location • Need to have: • AI files must reside on a different disk/LUN than any other aspect of the database (db, dn, bn) • Nice to have: • AI archived to yet another disk/physical locale • not always possible/plausible • Avoid I/O bottlenecks • Add more spindles if need be

19. Enabling After Imaging Online • New option to the probkup utility • Notes: • Make sure your AI files are in place • Common uses: • Oops, “forgot” to enable after imaging • AI off  Recoverable maintenance  AI on probkup online <db> <backup-file>enableai

20. Managing AI buffers online (10.1C) • New option • Notes: • For performance, set aibufs >= bi buf pool size • Online default same as -bibufs (8K AI buffers) • Permissions for new shared memory proutil <db> increase -aibufs <n>

21. Enabling the AI archiver • Offline: • AI must already be enabled ;( • Online: • AI can be enabled in conjunction ;) rfutil <db> -C aiarchiver enable probkup online <db> <backup> [ enableai ] enableaiarchiver -aiarcdir <dir> -aiarcinterval <n-seconds> [ -aiarcdircreate ]

22. New Syntax: Can optionally create directories on the fly Again, watch permissions Changing the Archive Directory Online rfutil <db> -C aiarchiver setarcdir<dir-list> rfutil <db> -C aiarchiver setarcdir<dir-list>/ -aiarcdircreate

23. Changing Archive Interval Online Forcing an extent switch automatically • New Syntax: • Valid range 120 to 86,400 (in seconds) • 2 minutes to 1 day • Suggest 15 to 60 minutes based on activity • Common uses • Correct mistake of minutes vs seconds • Change in business constraints (lost data) • Archiver awakes every 5 seconds rfutil <db> -C aiarchiver setinterval <#>

24. Changing Archive Mode Online On-demand vs timed archival rfutil <db> -C aiarchiver setinterval <#> • On-demand Mode • No interval or interval 0 • Use with fixed length extents or manual switch • Timed Mode • Interval set 120 to 86,400 (in seconds) • Archives regardless of activity • Notes: • Change time and/or on-demand during off hours • Timed must be set at each invocation • Archiver still awakes every 5 seconds

25. Agenda Monitoring your AI files Archive log naming Backing up AI archive files Cleaning up AI archive files Archiving and Maintenance

26. Manual AI Archival No archiver? No problem. Manually archive AI files while db is online • Check for full extents • Force extent switch • Find next full extent to archive • Rinse, lather, repeat • Example code: http://wss.com/products/downloads.html rfutil <db> -C aimage list rfutil <db> -C aimage new rfutil <db> -C aimage full

27. Not user definable (it should be) Naming convention Recovery easily managed Files relate to a particular backup Date and time Extent sequence number in the name Directory separator “/” replaced with “~” Example: Archive Log Naming • <db> . date . time . backup-seq . ai-seq . ext-name usr1~richb~x.20080806.151638.00003.00000001.x.a1

28. Backup of Archives NOT included in the OpenEdge database backup • Increase types of recoverable disasters • How often • Throughout the day • Where to • Separate media for media failure protection • Secondary machine as quickly as possible • Best if archival system is “very” far away

29. Cleaning Up Archives Archiver retains archives forever! • Eventual disk space issue • Retention dependant on business needs • Clean up • Create an automated process to clean up archives after a period of time (days) • Keep at least 7 days of archives online • How often do you backup? • Archives without a backup is useless • Where are they stored? • Maintaining a hot standby?

30. Cleaning Up Archives - Unix/Linux The find command is very useful for this task • Beware of changes in $ARCDIR • Consider e-mail rather than /dev/null • Incorporate remote copy • Add automatic validation # Archive Cleaner # Should be run from cron daily # Test yourself with echo before implementing ARCDIR=<your-archive-directory> KEEPDAYS=+30 find $ARCDIR -mtime $KEEPDAYS -exec \rm {} \ 2>&1 > /dev/null

31. Cleaning Up Archives - Windows Uses file from the free windows resource kit File name: %WINDOWS%\system32\forfiles.exe REM ** Archive cleanup batch file REM ** Have this run as a scheduled task REM ** Test yourself with echo before implementing Set ARCDIR=your-archive-directory Set KEEP=-30 forfiles /P %ARCDIR% /m *.a* /D %KEEP% /C “cmd /C del /q @FILE” REM OR forfiles -p%ARCDIR% -m*.a* -d%KEEP% -c“cmd /c del /q @FILE" //technet.microsoft.com/en-us/library/cc755872.aspx

32. Disaster Scenarios Bad stuff happens • Fire • Floods

33. Using auto AI archives to recover It is quite easy • In the AI archiver log “<db>.archival.log” • Find most recent backup to restore • Look for “BACKUP_SET” • Identify AI extent to start roll forward with • Match “aiseq” value to archived AI extent • Ignore AI extent’s backup sequence value • Roll forward it and all later archives • Log file layout documented in DB Admin Guide • Example: …

34. AI Example # OpenEdge RDBMS After-image Extent Manager Archival Log File # Do not edit this file with an editor as it is automatically # maintained by the OpenEdge RDBMS after-image extent manager. 0032,/usr1/db/x,20080807,100553,1,2,20080807,100551,1,1,x.bak,BACKUP_SET_20080807_100551 # # After-image Extent Manager started on : Thursday August 07, 2008 at 10:05:56 # 0255,20080807,100556,100100 0001,/usr1/db/x,20080807,100853,20080807,100552,2,1,/usr1/db/x.a1,/usr2/arcdir1,/usr2arcdir1/usr1~db~x.20080807.100552.00002.00000001.x.a1 0001,/usr1/dbx,20080807,100908,20080807,100552,2,2,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00002.00000002.x.a2 0032,/usr1/db/x,20080807,100947,1,3,20080807,100946,4,1,../x.bak2,BACKUP_SET_20080807_100946 0001,/usr1/db/x,20080807,100949,20080807,100552,3,3,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000003.x.a3 0001,/usr1/db/x,20080807,101029,20080807,100552,3,4,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000004.x.a4 0001,/usr1/db/x,20080807,101355,20080807,100552,3,5,/usr1/db/x.a1,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000005.x.a1 0001,/usr1/db/x,20080807,101441,20080807,100552,3,6,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000006.x.a2 0001,/usr1/db/x,20080807,101831,20080807,100552,3,7,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000007.x.a3 0001,/usr1/db/x,20080807,102037,20080807,100552,3,8,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000008.x.a4

35. A Eye Exam(ple) # OpenEdge RDBMS After-image Extent Manager Archival Log File # Do not edit this file with an editor as it is automatically # maintained by the OpenEdge RDBMS after-image extent manager. 0032,/usr1/db/x,20080807,100553,1,2,20080807,100551,1,1,x.bak,BACKUP_SET_20080807_100551 # # After-image Extent Manager started on : Thursday August 07, 2008 at 10:05:56 # 0255,20080807,100556,100100 0001,/usr1/db/x,20080807,100853,20080807,100552,2,1,/usr1/db/x.a1,/usr2/arcdir1,/usr2arcdir1/usr1~db~x.20080807.100552.00002.00000001.x.a1 0001,/usr1/dbx,20080807,100908,20080807,100552,2,2,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00002.00000002.x.a2 0032,/usr1/db/x,20080807,100947,1,3,20080807,100946,4,1,../x.bak2,BACKUP_SET_20080807_100946 0001,/usr1/db/x,20080807,100949,20080807,100552,3,3,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000003.x.a3 0001,/usr1/db/x,20080807,101029,20080807,100552,3,4,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000004.x.a4 0001,/usr1/db/x,20080807,101355,20080807,100552,3,5,/usr1/db/x.a1,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000005.x.a1 0001,/usr1/db/x,20080807,101441,20080807,100552,3,6,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000006.x.a2 0001,/usr1/db/x,20080807,101831,20080807,100552,3,7,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000007.x.a3 0001,/usr1/db/x,20080807,102037,20080807,100552,3,8,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000008.x.a4

36. A Eye Exam(ple) # OpenEdge RDBMS After-image Extent Manager Archival Log File # Do not edit this file with an editor as it is automatically # maintained by the OpenEdge RDBMS after-image extent manager. 0032,/usr1/db/x,20080807,100553,1,2,20080807,100551,1,1,x.bak,BACKUP_SET_20080807_100551 # # After-image Extent Manager started on : Thursday August 07, 2008 at 10:05:56 # 0255,20080807,100556,100100 0001,/usr1/db/x,20080807,100853,20080807,100552,2,1,/usr1/db/x.a1,/usr2/arcdir1,/usr2arcdir1/usr1~db~x.20080807.100552.00002.00000001.x.a1 0001,/usr1/dbx,20080807,100908,20080807,100552,2,2,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00002.00000002.x.a2 0032,/usr1/db/x,20080807,100947,1,3,20080807,100946,4,1,../x.bak2,BACKUP_SET_20080807_100946 0001,/usr1/db/x,20080807,100949,20080807,100552,3,3,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000003.x.a3 0001,/usr1/db/x,20080807,101029,20080807,100552,3,4,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000004.x.a4 0001,/usr1/db/x,20080807,101355,20080807,100552,3,5,/usr1/db/x.a1,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000005.x.a1 0001,/usr1/db/x,20080807,101441,20080807,100552,3,6,/usr1/db/x.a2,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000006.x.a2 0001,/usr1/db/x,20080807,101831,20080807,100552,3,7,/usr1/db/x.a3,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000007.x.a3 0001,/usr1/db/x,20080807,102037,20080807,100552,3,8,/usr1/db/x.a4,/usr2/arcdir1,/usr2/arcdir1/usr1~db~x.20080807.100552.00003.00000008.x.a4 Backup file-name, BACKUP_SET AI Seq #4 AI Seq #4 AI file-name to start rollforward AI Seq #4

37. Using auto AI archives to recover Some notes on the AI archive log • Documentation issue: buseq  aiseq • Wrong AI file? No problem. • Error message reported • Retry with the correct one • Roll forward last active/not archived AI extent • Always use AI sequence value • Archive location within .log file is “relatively” meaningless w/respect to its BACKUP_SET • WHY? …

38. Why AI Log is NOT Sequential AI Archive Daemon Extent Switch BI DB …with respect to the BACKUP_SET Archive Log DB.bak No sync on write order Backup A1 Archive 1 Full Busy Empty Archive 2 Archive 3 A2 Users Busy Empty

39. Recovery Process Overview Get everyone off the system (crons too!) Backup what you have (if anything) Database (even just parts) Recovery files (AI, BI,TL) AI Archives Delete broken database Restore most recent backup Apply after image archives rfutil <db> -C roll forward oplock –a <aifile> (10.1c)

40. Extending Archival Usefulness Log-based replication

41. Log-Based Replication Extending archival usefulness • Hot standby database • Good to use for backups • Eliminates backup I/O on production • Use -norecover option with probkup • Less recovery downtime • Cannot be updated (subsequent rollf. will fail) • Hot standby copy (#2) • Provide for reporting (semi-stale) • “Midnight” copy of hot standby #1

42. Log-Based Replication Extending archival usefulness • Auto archived files • Deal w/awkward naming convention • Create sub-directory under archive directory • Process looks for new archives • Applies new archives to hot-standby database • Moves applied archives • Best if on a remote host • Less contention w/production • Improved disaster recovery strategy

43. Summary All OpenEdge systems should use AI Many new online AI capabilities AI management tool provided Use archives to maintain a hot-standby Improved high availability

44. ? Questions

45. Outstanding Questions Answered • Does ai archiver use ai extract or simply copy? • It uses Ai Extract. • Is ai archiver available for Workgroup licenses? • Yes.  It is functionality and not performance related. • You can specify several archive directories.  Is there a way to tell which is the current archive directory? • There is a message in the log when a dir switch is performed. There is no way to programmatically determine this. I will add an enhancement request. • When ai is locked due to replication, is the file archived and later marked empty or does the archive process stall waiting for the locked extent to become unlocked? • The extent is archived.  Once it is “unlocked” it is emptied by the watch dog.  The aimgt deamon also attempts to empty locked extents. • Is there any documentation on using ai archiver with replication? • Yes, there is a callout in the ai mgt doc when using it with replication.  The doc explains what happens to an extent with ai mgt and repl running. • Is it safe to change permissions of a file after add online completes? • No. Although add online has completed, all users connected to the database have not opened the new file. They open it the next time they access it. Changing its permissions while the database is still running may cause the database to shutdown when a user attempts to open a file it does not have permission to.

46. Thank You