STREAMS: Downstream Capture

1. STREAMS:Downstream Capture PSS-DES meeting 11 April @ CERN Eva Dafonte Pérez

2. Agenda • STREAMS overview • STREAMS on the TESTBED • Configuration PROBLEMS • STREAMS log mining configuration survey • Downstream capture • Split & Merge solution Streams: Downstream Capture

3. STREAMS Overview • Flexible feature for sharing information • Basic elements: • Capture • Staging • Consumption • Replicate data from one database to one or more databases • Databases can be non identical copies Streams: Downstream Capture

4. capture apply propagation CAPTURE PROCESS APPLY PROCESS STREAMS Architecture LCRs SOURCE QUEUE capture changes propagate events LCRs log changes apply changes DESTINATION QUEUE REDO LOG TARGET DATABASE (replica) SOURCE DATABASE user changes Streams: Downstream Capture

5. capture apply apply capture propagation propagation CERN CERN LCR LCR LCR LCR LCR LCR LCR CNAF Sinica IN2P3 BNL RAL STREAMS on the TESTBED insert into employees values ( 03, “Joan”, ….); Streams: Downstream Capture

6. apply capture capture apply propagation propagation spill over performance CERN CERN LCR LCR LCR LCR LCR LCR LCR LCR LCR LCR CNAF Sinica IN2P3 RAL BNL Configuration PROBLEMS insert into employees values ( 03, “Joan”, ….); Streams: Downstream Capture

7. STREAMS log mining survey • Objectives • Source database (CERN) isolation against network or database problems at the replicas • Downstream Capture • Replica sites isolation against each other • split & merge solution Streams: Downstream Capture

8. CERN RAC RAL CERN CNAF downstream capture propagation jobs COPY redo log files SOURCE DATABASE DOWNSTREAM DATABASE DESTINATION SITES DOWNSTREAM Capture • Capture process runs on a different database • Redo log files are copied from source to downstream • Use of fewer resources and data loss protection • Definite latency in the replication process https://twiki.cern.ch/twiki/bin/view/PSSGroup/DownstreamDatabaseConfiguration Streams: Downstream Capture

9. DOWNSTREAM Capture • Real-Time Downstream capture • Redo transport services use the LGWR • records data in the online redo log at the source db • The redo data is stored in the standby redo log at the downstream db • Capture process captures changes from • standby redo log -> whenever possible • archived redo log files -> whenever necessary • Reduces the amount of time required to capture changes made at the source database Streams: Downstream Capture

10. Split & Merge solution • Split the capture process • (original) Real-Time capture for sites “in good shape” • (new) normal capture for site/s unstable/s • new capture queue and propagation job • original propagation job is dropped • spilled LCRs are dropped from the original capture queue • Merge the capture processes • Real-Time capture is used for all the sites • Resynchronization Streams: Downstream Capture

11. propagation jobs propagation jobs propagation jobs CERN RAL RAL RAL RAL Real-Time Downstream Capture CERN CERN … LCR LCR RAL CNAF … … CNAF CNAF start scn minimum start scn normal Capture x x propagation job propagation job RAL RAL Split & Merge solution x start scn: valid scn from which the capture process should start capturing changes with the help of Volker Kuhr (Oracle consulting) Streams: Downstream Capture

12. Questions & Answers Streams: Downstream Capture

13. Streams: Downstream Capture