Investigating Hiccups in RMAN Implementation for Production Database

My RMAN Implementation is stuck at the point of near to "Implemented".

1. I have configured the Production Database in Archivelog Mode.
2. I have created a Recovery Catalog.
3. I have registered the Production Database.
4. I have set all the configuration required for the Disk Backup to a shared SAN location.

But, RMAN backup to the shared SAN location is "killing" slow. It takes around 8-13 hours to have a full RMAN backup (full backup size-18GB).

So, I raised a Service Request with Oracle Support and have been following up with them since last week. The IT Admins claim, network is not a bottleneck as they push 70 GB of backup in 4 hours. And with the series of findings that I submitted to Oracle Support, RMAN seems to be doing its job perfectly. On the shared location on Test Machine, the same backup completes in 35-40 minutes.

Today, I had a work with Oracle Support and ran a series of Test to see how much time it takes to backup the production instance in 3 different location.

I connected RMAN and logged into to the Production Instance using target control file on the Production Server and carried out the test.

For the test, I used the following elements:

• Tablespace: ORION_TEST (1 datafile)
• Size: 1.36GB
• Local Backup Location: D:\oracle\backup
• Shared Backup Location: \\testdb\orcl
• Shared SAN Location: \\blade5\mis\backup\RMAN

Below is the spooled output of the test cases:

Spooling started in log file: \\Testdb\orcl\orcl_TEST_bkp.log
Recovery Manager: Release 9.2.0.8.0 - Production

RMAN>

List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Tag
------- -- -- - ----------- --------------- ------- ------- ---
71 B F A DISK 02-MAR-09 9 1 TAG20090302T114822
72 B A A DISK 02-MAR-09 1 1 TAG20090302T122345
73 B F A DISK 02-MAR-09 1 1
74 B F A DISK 03-MAR-09 9 1 TAG20090303T123712
75 B A A DISK 03-MAR-09 1 1 TAG20090303T131419
76 B F A DISK 03-MAR-09 1 1

RMAN>
RMAN>
Starting backup at 03-MAR-09
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=65 devtype=DISK
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00016 name=D:\ORACLE\ORADATA\ORCL\ORION_TEST.DBF
channel ORA_DISK_1: starting piece 1 at 03-MAR-09
channel ORA_DISK_1: finished piece 1 at 03-MAR-09
piece handle=D:\ORACLE\BACKUP\ORIONTEST_37K90DHC_1_1_20090303 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:04:55
Finished backup at 03-MAR-09

Starting Control File and SPFILE Autobackup at 03-MAR-09
piece handle=D:\ORACLE\ORA92\DATABASE\C-1032853409-20090303-02 comment=NONE
Finished Control File and SPFILE Autobackup at 03-MAR-09

RMAN>
RMAN>
Starting backup at 03-MAR-09
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00016 name=D:\ORACLE\ORADATA\ORCL\ORION_TEST.DBF
channel ORA_DISK_1: starting piece 1 at 03-MAR-09
channel ORA_DISK_1: finished piece 1 at 03-MAR-09
piece handle=\\TESTDB\ORCL\ORIONTEST_39K90DTF_1_1_20090303 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:02:25
Finished backup at 03-MAR-09

Starting Control File and SPFILE Autobackup at 03-MAR-09
piece handle=D:\ORACLE\ORA92\DATABASE\C-1032853409-20090303-03 comment=NONE
Finished Control File and SPFILE Autobackup at 03-MAR-09

RMAN>
RMAN>
Starting backup at 03-MAR-09
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00016 name=D:\ORACLE\ORADATA\ORCL\ORION_TEST.DBF
channel ORA_DISK_1: starting piece 1 at 03-MAR-09
channel ORA_DISK_1: finished piece 1 at 03-MAR-09
piece handle=\\BLADE5\MIS\BACKUP\RMAN\ORIONTEST_3BK90E8G_1_1_20090303 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:26:25
Finished backup at 03-MAR-09

Starting Control File and SPFILE Autobackup at 03-MAR-09
piece handle=D:\ORACLE\ORA92\DATABASE\C-1032853409-20090303-04 comment=NONE
Finished Control File and SPFILE Autobackup at 03-MAR-09

RMAN>
List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Tag
------- -- -- - ----------- --------------- ------- ------- ---
71 B F A DISK 02-MAR-09 9 1 TAG20090302T114822
72 B A A DISK 02-MAR-09 1 1 TAG20090302T122345
73 B F A DISK 02-MAR-09 1 1
74 B F A DISK 03-MAR-09 9 1 TAG20090303T123712
75 B A A DISK 03-MAR-09 1 1 TAG20090303T131419
76 B F A DISK 03-MAR-09 1 1
79 B F A DISK 03-MAR-09 1 1 TAG20090303T144812
80 B F A DISK 03-MAR-09 1 1
81 B F A DISK 03-MAR-09 1 1 TAG20090303T145439
82 B F A DISK 03-MAR-09 1 1
83 B F A DISK 03-MAR-09 1 1 TAG20090303T150031
84 B F A DISK 03-MAR-09 1 1
RMAN>


Here is the summary of the Test :

The backup tablespace size is 1.36 GB with one datafile.
The backup piece size in all 3 test cases is 1.1 GB each.
RMAN Backup Time at 3 location:
Local Backup Duration (D:\oracle\backup): 04:55 minutes
Shared Backup Duration (\\testdb\orcl): 02:25 minutes
Shared SAN Duration (\\blade5\mis\backup\RMAN): 26:25 minutes

With the results, it clearly indicates that backup of a 1.36 GB tablespace to SAN location is 5-6 times slower. This could be because of either Network Issue, High CPU Utilization, or Lots of I/Os. I have submitted the results to Oracle Support. Let's see what they have to say.

Until the issue is resolve, I am taking RMAN backups to shared location (\\testdb\orcl).

I will keep you posted on the further upcomings.