Complete Recovery of Production Instance

Yesterday at 01:12 pm, the production instance crashed. We were reported of "Shared Memory Realm Does not Exist" error by the users.

I noticed in the Alertlog, that the Control File (control05.ctl) was having some problem.

Wed Apr 29 13:04:37 2009
Errors in file d:\oracle\admin\orcl0\bdump\orcl_arc2_4852.trc:
ORA-00202: controlfile: 'C:\ORACLE\ORADATA\ORCL\CONTROL05.CTL'
ORA-27091: skgfqio: unable to queue I/O
ORA-27070: skgfdisp: async read/write failed
OSD-04006: ReadFile() failure, unable to read from file
O/S-Error: (OS 2) The system cannot find the file specified.

I brought down the instance, as it was in nomount state, and then renamed the corrupted Control File (control05.ctl) and by copying one of the other control files, I created a new Control File (control05.ctl).

I fired the startup to see the control file issue was resolved, but also got a couple of new error messages related to datafile 12.

The alert log revealed the following:

Wed Apr 29 13:17:29 2009
Errors in file d:\oracle\admin\orcl0\udump\orcl_ora_6512.trc:
ORA-00600: internal error code, arguments: [krdsod_1], [198804811], [0], [198804867], [0], [], [], []
ORA-01115: IO error reading block from file 12 (block # 334316)
ORA-01110: data file 12: 'D:\ORACLE\ORADATA\ORCL\ORION_ISCO01.DBF'
ORA-27070: skgfdisp: async read/write failed
OSD-04016: Error queuing an asynchronous I/O request.
O/S-Error: (OS 2) The system cannot find the file specified.

This file was part of the main production tablespace, kind of the heart of the Functional ERP System. I fired a recover on the tablespace and it still gave the same error.

SQL> alter database recover datafile 12;
alter database recover datafile 12
*
ERROR at line 1:
ORA-00283: recovery session canceled due to errors
ORA-01115: IO error reading block from file 12 (block # 334316)
ORA-01110: data file 12: 'D:\ORACLE\ORADATA\ORCL\ORION_ISCO01.DBF'
ORA-27070: skgfdisp: async read/write failed
OSD-04016: Error queuing an asynchronous I/O request.
O/S-Error: (OS 2) The system cannot find the file specified.

I was wondering how many files have been affected in this manner, to decide between full database restore or individual datafile/tablespace restore and recover.

Meanwhile, from the IT Admins, I came to know that one of the disks from the 4 RAID5 disks had a problem (defunc) at almost the same time when we noticed a Instance Crash, and that the disk needed to be fixed.

The database is in Archivelog mode and we have daily full database backups and also Archivelog backups in every 4 hours taken. Then I tried to restore the whole tablespace (datafiles 11, 12, 18, 19), and it again failed at datafile 12. Then, I renamed the existing datafile (ORION_ISCO01.DBF) and then restored the datafile 12 (ORION_ISCO01.DBF) from the backup, and eventually restored all the other 3 datafiles. And then I sucessfully recovered the tablespace (SR 7473105.994).

Now, I was able to successfully open the database.

SQL> alter database open;
Database altered.

SQL>

I checked the Systems and they were all running fine. And, then I took a full database RMAN backup with retention policy of redundacy 2. I raised a Service Request (SR 7559528.992) with Oracle Support to look into what the error messages mean and is it directly related to the RAID5 disk failure.
I would require to answer a few questions to the Management regarding this Incident, or should I say, Disaster Recovery Situation. It's high time, we need to change the database server.

Today, the users are using the system as if nothing happened. Thanks to the implementation of RMAN, I was able to bring the database back online in 3:30 hours. I can't imagine what I would have done with the full export dumps, or the ealier weekly Cold Backups (NOARCHIVELOG).

I will keep you all posted on my further findings.

Oracle Service and Oracle Instance

There was a simple question on OTN Forum today, yet the answer to it required a re-think on how to put the words right.

"Hi,
anybody can expain what is the difference between starting the Oracle Services(using SERVICES.MSC) and starting the oracle instance (using Startup commant) ?. whats happening in oracle while doing these things?"

Although most of us know the concepts behind the question, at times we do run out of words in explaining it or atleast expressing it others. I came across this blog [http://ewhalen.blogspot.com/], which had the following to say on "Oracle Services and the Windows Registry":

"A Microsoft Windows service allows you to create long-running executable applications that run in their own Windows sessions. These services can be automatically started when the computer boots, can be manually started, stopped and paused. Because they run in their own session, they do not contain a user interface, but they can take parameters or use registry parameters. Services are used for long-running functionality that does not interact with other users, thus is good for server applications such as the Oracle Database Server and Oracle Listener. In addition, services can be invoked under the user context of a domain user, local user or the system account. Thus the Oracle service can be started by a user other than the Oracle database owner, but will run under the context of the Oracle user."

So, baasically, a service is the way windows recognizes an executable and allocates and keeps reserved an amount of memory, until the service is stopped or forcefully terminated. Using this Service our Oracle Instance hangs in there for us to use the Oracle Database.

10G New Features: Data Pump Basics

Data Pump Components:

There are 3 major components of Data Pump Technology:

1. DBMS_DATAPUMP package
   In this package are the procedures used to carryout the Data Pump export and import activities.
   
2. DBMS_METADATA package
   This package is used to extract the metadata of the data dictionary objects.
   
3. expdp, impdp client utilities
   Using "expdp", you can extract data objects from the source database. And, using "impdp", you can load data objects into the target database.

Directory Objects:

The Data Pump job creates the dump files on the server. Using Directory Objects, Data Pump can write to/read from the O/S directory on the server.

You can use the DIRECTORY parameter of expdp/impdp utilities to relate to the directory.
You can also set the Environment Vairable "DATA_PUMP_DIR" and furnish on the dumpfile or logfile name.
Or, in the DUMPFILE and LOGFILE parameters, prefix the dumpfile name or the logfile name with the Directory Name followed by a ':', i.e. DIRECTORY:FILE notation.
Let's see how it works:

SQL> conn / as sysdba
Connected.

SQL> create directory dp_dir01 as 'c:\oracle\backup\dp';

Directory created.

SQL> grant read, write on directory dp_dir01 to system;

Grant succeeded.

SQL> host expdp system/db10gee full=y dumpfile=dp_dir01:exp01.dmp logfile=dp_dir01:exp01.log

;;;
Export: Release 10.2.0.1.0 - Production on Saturday, 11 April, 2009 14:46:36

Copyright (c) 2003, 2005, Oracle.  All rights reserved.
;;;
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
Starting "SYSTEM"."SYS_EXPORT_FULL_01":  system/******** full=y dumpfile=dp_dir01:exp01.dmp logfile=dp_dir01:exp01.log
Estimate in progress using BLOCKS method...
Total estimation using BLOCKS method: 120.8 MB
Processing object type DATABASE_EXPORT/TABLESPACE
................
................
................
Master table "SYSTEM"."SYS_EXPORT_FULL_01" successfully loaded/unloaded
******************************************************************************
Dump file set for SYSTEM.SYS_EXPORT_FULL_01 is:
  C:\ORACLE\BACKUP\DP\EXP01.DMP
Job "SYSTEM"."SYS_EXPORT_FULL_01" successfully completed at 14:51:10

Order of Precedence for file locations will be as follows:

1. DIRECTORY:FILE Notation
2. DIRECTORY Parameter
3. DATA_PUMP_DIR

10G New Features: Statistics Collection

I am reading through "OCP Oracle Database 10G: New Features for Administrators Exam Guide" by Sam R. Alapati. A must read for 9i DBAs looking to upgrade skills to Oracle 10G.

In 10G, there is a new background process called MMON [Manageability Monitor Process]. MMON keeps track of the High Water Mark Statistics and the Database Usage statistics, recording it in AWR [Automatic Workload Repository].

Database Usage Statistics:

SQL> desc dba_feature_usage_statistics
Name                                      Null?    Type
----------------------------------------- -------- ----------------------------
DBID                                      NOT NULL NUMBER
NAME                                      NOT NULL VARCHAR2(64)
VERSION                                   NOT NULL VARCHAR2(17)
DETECTED_USAGES                           NOT NULL NUMBER
TOTAL_SAMPLES                             NOT NULL NUMBER CURRENTLY_USED                                     VARCHAR2(5)
FIRST_USAGE_DATE                                   DATE
LAST_USAGE_DATE                                    DATE
AUX_COUNT                                          NUMBER
FEATURE_INFO                                       CLOB
LAST_SAMPLE_DATE                                   DATE
LAST_SAMPLE_PERIOD                                 NUMBER
SAMPLE_INTERVAL                                    NUMBER
DESCRIPTION                                        VARCHAR2(128)

HWM Statistics:

SQL>
SQL> desc dba_high_water_mark_statistics
Name                                      Null?    Type
----------------------------------------- -------- ----------------------------
DBID                                      NOT NULL NUMBER
NAME                                      NOT NULL VARCHAR2(64)
VERSION                                   NOT NULL VARCHAR2(17)
HIGHWATER                                          NUMBER
LAST_VALUE                                         NUMBER
DESCRIPTION                                        VARCHAR2(128)

SQL>
SQL> select name, highwater, description
2  from dba_high_water_mark_statistics
3  order by 3;

NAME                                                             HIGHWATER                                                          
---------------------------------------------------------------- ----------
DESCRIPTION
--------------------------------------------------------------------------------
CPU_COUNT                                                                 2
Maximum Number of CPUs
SESSIONS                                                                 18
Maximum Number of Concurrent Sessions seen in the database
DATAFILES                                                                12
Maximum Number of Datafiles

PART_INDEXES                                                              0
Maximum Number of Partitions belonging to an User Index
PART_TABLES                                                               6
Maximum Number of Partitions belonging to an User Table
SQL_NCHAR_COLUMNS                                                        11
Maximum Number of SQL NCHAR Columns

SERVICES                                                                  4
Maximum Number of Services
TABLESPACES                                                              13
Maximum Number of Tablespaces
QUERY_LENGTH                                                            139
Maximum Query Length

DB_SIZE                                                          1704001536
Maximum Size of the Database (Bytes)
USER_INDEXES                                                           2001
Number of User Indexes
USER_TABLES                                                             998
Number of User Tables

SEGMENT_SIZE                                                      183500800
Size of Largest Segment (Bytes)

13 rows selected.

SQL>

Pretty Cool.... Didn't have that in 9i.

Template Change

I really liked the template "Minima Dark", but had to change the template to "Minima Stretch", for better readability of texts, scripts and codes.

Cloning Database using RMAN

I mentioned in one of my RMAN posts earlier, that there was one feature that I really liked about RMAN. Well, today is your lucky day!!! I am going to talk about Database Cloning using RMAN.

I am sure most of you would be very well aware of RMAN''s Duplicate Command. We will see the command in action.

Before you start cloning you should fulfill the following pre-requisites:

1. Target Database must be in Archivelog mode
2. Target Database should be registered in the Recovery Catalog Database
3. A Valid Full Database Backup of Target Database should be in hand
4. A Test Machine that has Oracle Home Version same as that of Target Database

Example Considerations:

Test Machine O/S: Windows XP SP2
Oracle Version: Oracle 9iR2 (9.2.0.8) Standard Edition
Target Database: ORCL
Recover Catalog Database: RCVCAT
Auxiliary Database: DBPIMPL

On the Test Machine, create an initialization parameter file same as that of Target Database. Here I will refer my new Database to be cloned as Auxiliary Database having SID as "DBPIMPL".

1. Change the db_name and instance_name parameters to Auxiliary Database Name, i.e. "DBPIMPL".
2. Change the Target Database OFA path/location to the Auxiliary Database OFA path/location.
3. Modify the SGA size as per the availability of Resources on Test Machine.
4. Block Size of Auxiliary Database must be same as that of Target Database.

Please ensure that the following parameters are added to the PFILE:

*.lock_name_space=ORCL

This parameter ensures we don't accidently restore and recover the Target Database

*.db_file_name_convert=('D:\oracle\oradata\ORCL\', 'C:\oracle\oradata\DBPIMPL\')
*.log_file_name_convert=('D:\oracle\oradata\ORCL\', 'C:\oracle\oradata\DBPIMPL\')

Pointing to the new locaion, so that RMAN, using Target Database Backup Files, will restore at new location.

Your Auxilary Database's PFILE (initdbpimpl.ora) should look like this:

*.background_dump_dest='C:\oracle\admin\DBPIMPL\bdump'
*.compatible='9.2.0.8'
*.control_files='C:\oracle\oradata\DBPIMPL\CONTROL01.CTL','C:\oracle\oradata\DBPIMPL\CONTROL02.CTL','C:\oracle\oradata\DBPIMPL\CONTROL03.CTL'
*.core_dump_dest='C:\oracle\admin\DBPIMPL\cdump'
*.db_cache_size=33554432
*.db_block_size=4096
*.db_name='DBPIMPL'
*.instance_name='DBPIMPL'
*.java_pool_size=58720256
*.large_pool_size=10485760
*.open_cursors=300
*.pga_aggregate_target=25165824
*.processes=150
*.shared_pool_size=50331648
*.undo_management='AUTO'
*.undo_tablespace='UNDOTBS'
*.user_dump_dest='C:\oracle\admin\DBPIMPL\udump'
*.db_file_name_convert=('D:\oracle\oradata\ORCL\', 'C:\oracle\oradata\DBPIMPL\')
*.log_file_name_convert=('D:\oracle\oradata\ORCL\', 'C:\oracle\oradata\DBPIMPL\'
*.lock_name_space=ORCL

Next, create a service for Auxiliary Database using ORADIM:

oradim -new -sid dbpimpl -intpwd dbpimpl -pfile C:\oracle\product\9.2.0\db_1\database\initdbpimpl.ora -startmode m

In the Test Machine's TNSNAMES.ora, add a TNS Entry for the new database, i.e. Auxiliary Database. Ensure that all the 3 TNS entries are dedicated for RMAN to function smoothly.

In the Test Machine's LISTENER.ora, add the new database SID to the Listener's SID List.

Reload, or Stop and Start the to make the changes effective.

Ensure all the folders are created for the Auxiliary Database (DBPIMPL), as per Optimal Flexible Architecture. In my case, as follow:

.../admin/bdump
.../admin/cdump
.../admin/udump
.../oradata/DBPIMPL
.../oradata/DBPIMPL/archive

Open a new command window, and put the Auxiliary Database in NOMOUNT mode.

C:\>set oracle_sid=dbpimpl
C:\>sqlplus /nolog
SQL*Plus: Release 9.2.0.8.0 - Production on Tue Apr 7 09:43:42 2009
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

SQL> connect / as sysdba
Connected to an idle instance.

SQL> host lsnrctl
LSNRCTL for 32-bit Windows: Version 9.2.0.8.0 - Production on 07-APR-2009 10:07:23
Copyright (c) 1991, 2006, Oracle Corporation. All rights reserved.
Welcome to LSNRCTL, type "help" for information.

LSNRCTL> reload
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=zaffark)(PORT=1521)))
The command completed successfully

LSNRCTL> exit

SQL>
SQL> startup nomount;
ORACLE instance started.
Total System Global Area 177282424 bytes
Fixed Size 454008 bytes
Variable Size 142606336 bytes
Database Buffers 33554432 bytes
Redo Buffers 667648 bytes

SQL>
SQL> show parameter db_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_name string DBPIMPL

SQL>
SQL> disconn

SQL> exit

Now, we will require to connect to RMAN on Test Machine and start the cloning process:

C:\>set oracle_sid=dbpimpl

C:\>rman
Recovery Manager: Release 9.2.0.8.0 - Production
Copyright (c) 1995, 2002, Oracle Corporation. All rights reserved.

RMAN> spool log to c:\dbpimpl_dup_070409.txt

RMAN> connect catalog rman@rcvcat

recovery catalog database Password:

RMAN> connect target sys@orcl

target database Password:

RMAN> connect auxiliary /

RMAN>
RMAN> duplicate target database to DBPIMPL
2> LOGFILE
3> GROUP 1 ('C:\ORACLE\ORADATA\dbpimpl\REDO01A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO01B.RDO') SIZE 10M REUSE,
4> GROUP 2 ('C:\ORACLE\ORADATA\dbpimpl\REDO02A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO02B.RDO') SIZE 10M REUSE,
5> GROUP 3 ('C:\ORACLE\ORADATA\dbpimpl\REDO03A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO03B.RDO') SIZE 10M REUSE,
6> GROUP 4 ('C:\ORACLE\ORADATA\dbpimpl\REDO04A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO04B.RDO') SIZE 10M REUSE;

RMAN>

My Target Database Has Redo Multiplexed on 2 drives, where as my Auxiliary Database doesn't have the luxury of 2 drives. So, I will create Redo Log Group Members same as the Target Database, but point them to Test Machine's C: Drive.

I need to check the RMAN spool file for any errors. If none, my cloning is complete. So simple!!! That's why it is a beautiful feature of RMAN.

Here is what happens in the background when you fire the RMAN Duplicate command.

Spooling started in log file: c:\dbpimpl_dup_070409.txt
Recovery Manager: Release 9.2.0.8.0 - Production

RMAN> connected to recovery catalog database

RMAN> connected to target database: ORCL (DBID=1032853409)

RMAN> connected to auxiliary database: DBPIMPL (not mounted)

RMAN>
RMAN> 2> 3> 4> 5> 6>

Starting Duplicate Db at 07-APR-09
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=10 devtype=DISK

printing stored script:
Memory Script{
set until scn 195634017;
set newname for datafile 1 to "C:\ORACLE\ORADATA\DBPIMPL\SYSTEM01.DBF";
set newname for datafile 2 to "C:\ORACLE\ORADATA\DBPIMPL\UNDOTBS01.DBF";
set newname for datafile 3 to "C:\ORACLE\ORADATA\DBPIMPL\CWMLITE01.DBF";
set newname for datafile 4 to "C:\ORACLE\ORADATA\DBPIMPL\DRSYS01.DBF";
set newname for datafile 5 to "C:\ORACLE\ORADATA\DBPIMPL\INDX01.DBF";
set newname for datafile 6 to "C:\ORACLE\ORADATA\DBPIMPL\TOOLS01.DBF";
.....
set newname for datafile 24 to "C:\ORACLE\ORADATA\DBPIMPL\USERS01.DBF";
restore
check readonly
clone database ;
}

executing script: Memory Script
executing command: SET until clause
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME

Starting restore at 07-APR-09
using channel ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to C:\ORACLE\ORADATA\DBPIMPL\SYSTEM01.DBF
restoring datafile 00002 to C:\ORACLE\ORADATA\DBPIMPL\UNDOTBS01.DBF
restoring datafile 00003 to C:\ORACLE\ORADATA\DBPIMPL\CWMLITE01.DBF
restoring datafile 00004 to C:\ORACLE\ORADATA\DBPIMPL\DRSYS01.DBF
restoring datafile 00005 to C:\ORACLE\ORADATA\DBPIMPL\INDX01.DBF
restoring datafile 00006 to C:\ORACLE\ORADATA\DBPIMPL\TOOLS01.DBF
......
restoring datafile 00024 to C:\ORACLE\ORADATA\DBPIMPL\USERS01.DBF

channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=\\TESTDB\ORCL\ORCL_ARKBQBQB_1_1_20090406 tag=FULL_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restored backup piece 2
piece handle=\\TESTDB\ORCL\ORCL_ARKBQBQB_2_1_20090406 tag=FULL_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restored backup piece 3
piece handle=\\TESTDB\ORCL\ORCL_ARKBQBQB_3_1_20090406 tag=FULL_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restored backup piece 4
piece handle=\\TESTDB\ORCL\ORCL_ARKBQBQB_4_1_20090406 tag=FULL_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restored backup piece 5
piece handle=\\TESTDB\ORCL\ORCL_ARKBQBQB_5_1_20090406 tag=FULL_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restored backup piece 6
piece handle=\\TESTDB\ORCL\ORCL_ARKBQBQB_6_1_20090406 tag=FULL_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restored backup piece 7
piece handle=\\TESTDB\ORCL\ORCL_ARKBQBQB_7_1_20090406 tag=FULL_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restored backup piece 8
piece handle=\\TESTDB\ORCL\ORCL_ARKBQBQB_8_1_20090406 tag=FULL_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restored backup piece 9
piece handle=\\TESTDB\ORCL\ORCL_ARKBQBQB_9_1_20090406 tag=FULL_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restore complete

Finished restore at 07-APR-09

sql statement: CREATE CONTROLFILE REUSE SET DATABASE "DBPIMPL" RESETLOGS ARCHIVELOG
MAXLOGFILES 50
MAXLOGMEMBERS 5
MAXDATAFILES 100
MAXINSTANCES 1
MAXLOGHISTORY 226
LOGFILE
GROUP 1 ( 'C:\ORACLE\ORADATA\dbpimpl\REDO01A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO01B.RDO' ) SIZE 10485760 REUSE,
GROUP 2 ( 'C:\ORACLE\ORADATA\dbpimpl\REDO02A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO02B.RDO' ) SIZE 10485760 REUSE,
GROUP 3 ( 'C:\ORACLE\ORADATA\dbpimpl\REDO03A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO03B.RDO' ) SIZE 10485760 REUSE,
GROUP 4 ( 'C:\ORACLE\ORADATA\dbpimpl\REDO04A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO04B.RDO' ) SIZE 10485760 REUSE
DATAFILE
'C:\ORACLE\ORADATA\DBPIMPL\SYSTEM01.DBF'
CHARACTER SET WE8MSWIN1252

printing stored script: Memory Script
{ switch clone datafile all; }

executing script: Memory Script

datafile 2 switched to datafile copy
input datafilecopy recid=1 stamp=683556745 filename=C:\ORACLE\ORADATA\DBPIMPL\UNDOTBS01.DBF
datafile 3 switched to datafile copy
input datafilecopy recid=2 stamp=683556745 filename=C:\ORACLE\ORADATA\DBPIMPL\CWMLITE01.DBF
datafile 4 switched to datafile copy
input datafilecopy recid=3 stamp=683556745 filename=C:\ORACLE\ORADATA\DBPIMPL\DRSYS01.DBF
datafile 5 switched to datafile copy
input datafilecopy recid=4 stamp=683556745 filename=C:\ORACLE\ORADATA\DBPIMPL\INDX01.DBF
datafile 6 switched to datafile copy
input datafilecopy recid=5 stamp=683556745
filename=C:\ORACLE\ORADATA\DBPIMPL\TOOLS01.DBF
......
datafile 24 switched to datafile copy
input datafilecopy recid=23 stamp=683556745 filename=C:\ORACLE\ORADATA\DBPIMPL\USERS01.DBF

printing stored script: Memory Script

{
set until scn 195634017;
recover
clone database
delete archivelog ;
}

executing script: Memory Script

executing command: SET until clause

Starting recover at 07-APR-09
using channel ORA_AUX_DISK_1

starting media recovery

channel ORA_AUX_DISK_1: starting archive log restore to default destination
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=8168
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=\\TESTDB\ORCL\ORCL_ATKBQDRG_1_1_20090406 tag=ARCH_20090406_1730 params=NULL
channel ORA_AUX_DISK_1: restore complete
archive log filename=C:\ORACLE\PRODUCT\9.2.0\DB_1\RDBMS\ARC08168.001 thread=1 sequence=8168
channel clone_default: deleting archive log(s)
archive log filename=C:\ORACLE\PRODUCT\9.2.0\DB_1\RDBMS\ARC08168.001 recid=1 stamp=683556761
channel ORA_AUX_DISK_1: starting archive log restore to default destination
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=8169
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=8170
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=8171
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=\\TESTDB\ORCL\ORCL_B1KBS5S5_1_1_20090407 tag=ARCH_20090407_1000 params=NULL
channel ORA_AUX_DISK_1: restore complete
archive log filename=C:\ORACLE\PRODUCT\9.2.0\DB_1\RDBMS\ARC08169.001 thread=1 sequence=8169
channel clone_default: deleting archive log(s)
archive log filename=C:\ORACLE\PRODUCT\9.2.0\DB_1\RDBMS\ARC08169.001 recid=4 stamp=683556869
archive log filename=C:\ORACLE\PRODUCT\9.2.0\DB_1\RDBMS\ARC08170.001 thread=1 sequence=8170
channel clone_default: deleting archive log(s)
archive log filename=C:\ORACLE\PRODUCT\9.2.0\DB_1\RDBMS\ARC08170.001 recid=3 stamp=683556844
archive log filename=C:\ORACLE\PRODUCT\9.2.0\DB_1\RDBMS\ARC08171.001 thread=1 sequence=8171
channel clone_default: deleting archive log(s)
archive log filename=C:\ORACLE\PRODUCT\9.2.0\DB_1\RDBMS\ARC08171.001 recid=2 stamp=683556776
media recovery complete

Finished recover at 07-APR-09

printing stored script: Memory Script

{
shutdown clone;
startup clone nomount ;
}

executing script: Memory Script

database dismounted
Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area 177282424 bytes

Fixed Size 454008 bytes
Variable Size 142606336 bytes
Database Buffers 33554432 bytes
Redo Buffers 667648 bytes

sql statement: CREATE CONTROLFILE REUSE SET DATABASE "DBPIMPL" RESETLOGS ARCHIVELOG
MAXLOGFILES 50
MAXLOGMEMBERS 5
MAXDATAFILES 100
MAXINSTANCES 1
MAXLOGHISTORY 226
LOGFILE
GROUP 1 ( 'C:\ORACLE\ORADATA\dbpimpl\REDO01A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO01B.RDO' ) SIZE 10485760 REUSE,
GROUP 2 ( 'C:\ORACLE\ORADATA\dbpimpl\REDO02A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO02B.RDO' ) SIZE 10485760 REUSE,
GROUP 3 ( 'C:\ORACLE\ORADATA\dbpimpl\REDO03A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO03B.RDO' ) SIZE 10485760 REUSE,
GROUP 4 ( 'C:\ORACLE\ORADATA\dbpimpl\REDO04A.RDO', 'C:\ORACLE\ORADATA\dbpimpl\REDO04B.RDO' ) SIZE 10485760 REUSE
DATAFILE 'C:\ORACLE\ORADATA\DBPIMPL\SYSTEM01.DBF'
CHARACTER SET WE8MSWIN1252

printing stored script: Memory Script

{
catalog clone datafilecopy "C:\ORACLE\ORADATA\DBPIMPL\UNDOTBS01.DBF";
catalog clone datafilecopy "C:\ORACLE\ORADATA\DBPIMPL\CWMLITE01.DBF";
catalog clone datafilecopy "C:\ORACLE\ORADATA\DBPIMPL\DRSYS01.DBF";
catalog clone datafilecopy "C:\ORACLE\ORADATA\DBPIMPL\INDX01.DBF";
catalog clone datafilecopy "C:\ORACLE\ORADATA\DBPIMPL\TOOLS01.DBF";
......
catalog clone datafilecopy "C:\ORACLE\ORADATA\DBPIMPL\USERS01.DBF";
switch clone datafile all;
}


executing script: Memory Script

cataloged datafile copy
datafile copy filename=C:\ORACLE\ORADATA\DBPIMPL\UNDOTBS01.DBF recid=1 stamp=683557959
cataloged datafile copy
datafile copy filename=C:\ORACLE\ORADATA\DBPIMPL\CWMLITE01.DBF recid=2 stamp=683557959
cataloged datafile copy
datafile copy filename=C:\ORACLE\ORADATA\DBPIMPL\DRSYS01.DBF recid=3 stamp=683557959
cataloged datafile copy
datafile copy filename=C:\ORACLE\ORADATA\DBPIMPL\INDX01.DBF recid=4 stamp=683557960
cataloged datafile copy
datafile copy filename=C:\ORACLE\ORADATA\DBPIMPL\TOOLS01.DBF recid=5 stamp=683557960
......
cataloged datafile copy
datafile copy filename=C:\ORACLE\ORADATA\DBPIMPL\USERS01.DBF recid=23 stamp=683557961
datafile 2 switched to datafile copy
input datafilecopy recid=1 stamp=683557959 filename=C:\ORACLE\ORADATA\DBPIMPL\UNDOTBS01.DBF
datafile 3 switched to datafile copy
input datafilecopy recid=2 stamp=683557959 filename=C:\ORACLE\ORADATA\DBPIMPL\CWMLITE01.DBF
datafile 4 switched to datafile copy
input datafilecopy recid=3 stamp=683557959 filename=C:\ORACLE\ORADATA\DBPIMPL\DRSYS01.DBF
datafile 5 switched to datafile copy
input datafilecopy recid=4 stamp=683557960 filename=C:\ORACLE\ORADATA\DBPIMPL\INDX01.DBF
datafile 6 switched to datafile copy
filename=C:\ORACLE\ORADATA\DBPIMPL\TOOLS01.DBF
input datafilecopy recid=5 stamp=683557960
......
datafile 24 switched to datafile copy
input datafilecopy recid=23 stamp=683557964 filename=C:\ORACLE\ORADATA\DBPIMPL\USERS01.DBF

printing stored script: Memory Script

{
Alter clone database open resetlogs;
}

executing script: Memory Script
database opened

Finished Duplicate Db at 07-APR-09

RMAN>

So, the logs says my cloniong is successfully completed. Now there are few post-cloning activities that need to be taken care of.

Firstly, you need to add tempfiles to all the temporary tablespaces that you might be having in the Auxiliary Database. Then create your spfile from pfile, and restart the database.

SQL> ALTER TABLESPACE TEMP ADD TEMPFILE 'C:\oracle\oradata\DBPIMPL\TEMP01.DBF' SIZE 100M 
2  /
Tablespace altered.

SQL> ALTER TABLESPACE TEMP_STATS ADD TEMPFILE 'C:\oracle\oradata\DBPIMPL\TEMP_STATS01.DBF' SIZE 50M 
2  /
Tablespace altered.

SQL> ALTER TABLESPACE TEMP_PAY ADD TEMPFILE 'C:\oracle\oradata\DBPIMPL\TEMP_PAY01.DBF' SIZE 50M 
2  /
Tablespace altered.

SQL> SELECT COUNT(*) FROM DBA_OBJECTS WHERE STATUS <> 'VALID';
  COUNT(*)
----------
         0

SQL> create spfile from pfile;
File created.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup
ORACLE instance started.
Total System Global Area  177282424 bytes
Fixed Size                   454008 bytes
Variable Size             142606336 bytes
Database Buffers           33554432 bytes
Redo Buffers                 667648 bytes
Database mounted.
Database opened.

SQL> exit

Now you can connect to RMAN using target control file and take a full database backup.

C:\>rman

Recovery Manager: Release 9.2.0.8.0 - Production

Copyright (c) 1995, 2002, Oracle Corporation.  All rights reserved.

RMAN> CONNECT TARGET /

connected to target database: DBPIMPL (DBID=1527816262)

RMAN> SHOW ALL;

using target database controlfile instead of recovery catalog
RMAN configuration parameters are:

CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # defaultCONFIGURE SNAPSHOT CONTROLFILE NAME TO 'C:\ORACLE\PRODUCT\9.2.0\DB_1\DATABASE\SNCFDBPIMPL.ORA'; # default

RMAN> CONFIGURE CONTROLFILE AUTOBACKUP ON;

new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
new RMAN configuration parameters are successfully stored

RMAN> CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT   'C:\oracle\BACKUP\RMAN\DBPIMPL\%d_%U_%T' MAXPIECESIZE 2 G;

new RMAN configuration parameters:
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT   'C:\oracle\BACKUP\RMAN\DBPIMPL\%d_%U_%T' MAXPIECESIZE 2 G;
new RMAN configuration parameters are successfully stored

RMAN>
RMAN> show all;

RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT   'C:\oracle\BACKUP\RMAN\DBPIMPL\%d_%U_%T' MAXPIECESIZE 2 G;
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'C:\ORACLE\PRODUCT\9.2.0\DB_1\DATABASE\SNCFDBPIMPL.ORA'; # default

RMAN> BACKUP DATABASE PLUS ARCHIVELOG;

Starting backup at 07-APR-09
current log archived
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=10 devtype=DISK
channel ORA_DISK_1: starting archive log backupset
channel ORA_DISK_1: specifying archive log(s) in backup setinput archive log thread=1 sequence=1 recid=1 stamp=683558677
channel ORA_DISK_1: starting piece 1 at 07-APR-09
channel ORA_DISK_1: finished piece 1 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_01KBSHOL_1_1_20090407 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 07-APR-09

Starting backup at 07-APR-09
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00001 name=C:\ORACLE\ORADATA\DBPIMPL\SYSTEM01.DBF
input datafile fno=00002 name=C:\ORACLE\ORADATA\DBPIMPL\UNDOTBS01.DBF
input datafile fno=00006 name=C:\ORACLE\ORADATA\DBPIMPL\TOOLS01.DBF
input datafile fno=00005 name=C:\ORACLE\ORADATA\DBPIMPL\INDX01.DBF
input datafile fno=00003 name=C:\ORACLE\ORADATA\DBPIMPL\CWMLITE01.DBF
input datafile fno=00004 name=C:\ORACLE\ORADATA\DBPIMPL\DRSYS01.DBF
......
input datafile fno=00024 name=C:\ORACLE\ORADATA\DBPIMPL\USERS01.DBF
channel ORA_DISK_1: starting piece 1 at 07-APR-09
channel ORA_DISK_1: finished piece 1 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_02KBSHOP_1_1_20090407 comment=NONE
channel ORA_DISK_1: starting piece 2 at 07-APR-09
channel ORA_DISK_1: finished piece 2 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_02KBSHOP_2_1_20090407 comment=NONE
channel ORA_DISK_1: starting piece 3 at 07-APR-09
channel ORA_DISK_1: finished piece 3 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_02KBSHOP_3_1_20090407 comment=NONE
channel ORA_DISK_1: starting piece 4 at 07-APR-09
channel ORA_DISK_1: finished piece 4 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_02KBSHOP_4_1_20090407 comment=NONE
channel ORA_DISK_1: starting piece 5 at 07-APR-09
channel ORA_DISK_1: finished piece 5 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_02KBSHOP_5_1_20090407 comment=NONE
channel ORA_DISK_1: starting piece 6 at 07-APR-09
channel ORA_DISK_1: finished piece 6 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_02KBSHOP_6_1_20090407 comment=NONE
channel ORA_DISK_1: starting piece 7 at 07-APR-09
channel ORA_DISK_1: finished piece 7 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_02KBSHOP_7_1_20090407 comment=NONE
channel ORA_DISK_1: starting piece 8 at 07-APR-09
channel ORA_DISK_1: finished piece 8 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_02KBSHOP_8_1_20090407 comment=NONE
channel ORA_DISK_1: starting piece 9 at 07-APR-09
channel ORA_DISK_1: finished piece 9 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_02KBSHOP_9_1_20090407 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:25:43
Finished backup at 07-APR-09

Starting backup at 07-APR-09
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archive log backupset
channel ORA_DISK_1: specifying archive log(s) in backup setinput archive log thread=1 sequence=2 recid=2 stamp=683560224
channel ORA_DISK_1: starting piece 1 at 07-APR-09
channel ORA_DISK_1: finished piece 1 at 07-APR-09
piece handle=C:\ORACLE\BACKUP\RMAN\DBPIMPL\DBPIMPL_03KBSJ90_1_1_20090407 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 07-APR-09

Starting Control File Autobackup at 07-APR-09
piece handle=C:\ORACLE\PRODUCT\9.2.0\DB_1\DATABASE\C-1527816262-20090407-00 comment=NONE
Finished Control File Autobackup at 07-APR-09

RMAN>exit;

c:\>

This completes the cloning of Auxiliary Database, similar to that of the Target Database.

Winding-Up RMAN Implementation for Production Database

While I was taking up the RMAN Backup performance issue along with support, I also posted the query on OTN Forums (http://forums.oracle.com/forums/message.jspa?messageID=3320320#3320320).

I carried out the tablespace RMAN backup on the Database Server and then copied it to Blade5 SAN location. When I did a normal copy it took only 2 minutes as opposed to 21 minutes.

This was what one contributor had to say on OTN Forum:
"The rman backup process is not only a copy process but also a check on the integrety of the written backup piece. Might be there are a lot of context switches involved in this process which can, due to latency on the network result in bad performance.(The connection to the bad performing storage is over IP as I understand it)Regards,Tycho"

Well, in any case, there definately was no solution coming out from either sides (Oracle Support and IT Admins) to resolve the performance issue and pronouce RMAN implementation successful. Assuring that <\\testdb\orcl> location would be used as a staging location to backup the RMAN files to DDS tapes, I went ahead and scheduled the RMAN full database backups.

So now, I have a Recovery Catalog Instance through which I am taking daily RMAN Backups of Production Instance, Development Instance and Test Instance. And once all the 3 backups are completed, I am taking RMAN No catalog /target Backup of Recovery Catalog Database on shared SAN folder. It takes 25 minutes to generate 700 MB RMAN Backup files of the Recovery Catalog Database.

Here is the source of windows batch file that carries out the scheduled RMAN backups:

REM#####################################################################
REM # Name:   testdb_prod_rman_hot_backup_dsk.bat
REM # Description: ORCL
REM # Generic   Hot backup script that accepts the SID
REM # Parameters:  Instance Name
REM##################################################################
REM # Check if SID was Passed
if (%1)==() goto USAGE

REM # assign ORACLE_SID to passed SID
set ORACLE_SID=%1

REM # assign a date we can use as part of the logfile
set DTE=%date:~-4%%date:~3,2%%date:~0,2%_%time:~0,2%%time:~3,2%

REM # Take the spaces out of the date string so we can use in our file name
set DTE=%DTE: =%

REM # set the script locationREM # Set location of where you will put script
set SCRIPTLOC=C:\oracle\BACKUP\Scriptscd %SCRIPTLOC%

REM # Set the Temporary command file that we will write our rman script to
set CMDFILE=%SCRIPTLOC%\%ORACLE_SID%_rman_hot_backup_dsk.cmd

REM # Remove the temporary file if it exists
del /q %CMDFILE%

REM # Set the logfile directory
REM # Change based on where you want the backup logs to go
set LOGPATH=C:\oracle\BACKUP\logs
set LOGFILE=%ORACLE_SID%_rman_hot_backup_dsk_%DTE%.log
set LOG=%LOGPATH%\%LOGFILE%

REM # Set the path of the backup location
REM # Change based on disk path for you disk based backups (ORCL)
set BACKLOC=\\testdb\%ORACLE_SID%

REM # Set to your ORACLE_HOME location
set ORACLE_HOME=C:\oracle\product\9.2.0\db_1

# Set the rest of the Oracle Environment# based on our ORACLE_HOME
set LIBPATH=%ORACLE_HOME%/lib
set LD_LIBRARY_PATH=%ORACLE_HOME%/lib
set TNS_ADMIN=%ORACLE_HOME%/network/admin

REM Set our PATH with the ORACLE_HOME so that we have a good
REM clean environment to work with
set PATH=%ORACLE_HOME%/bin;%ORACLE_HOME%/OPatch;%PATH%

REM # Write Temporary command file for RMAN backup
echo # >> %CMDFILE%
echo # display current configurations and list of backups >> %CMDFILE%
echo SHOW ALL ; >> %CMDFILE%
echo # >> %CMDFILE%
echo LIST BACKUP SUMMARY ; >> %CMDFILE%
echo # >> %CMDFILE%
echo # lets start clean for our hot backup by doing full crosschecks >> %CMDFILE%
echo change archivelog all crosscheck ; >> %CMDFILE%
echo crosscheck backup of database; >> %CMDFILE%
echo crosscheck backup of controlfile; >> %CMDFILE%
echo # >> %CMDFILE%
echo delete noprompt expired archivelog all ; >> %CMDFILE%
echo delete noprompt expired backup ; >> %CMDFILE%
echo delete noprompt obsolete ; >> %CMDFILE%
echo # >> %CMDFILE%
echo CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '\\testdb\orcl\%%d_%%U_%%T' MAXPIECESIZE 2 G ; >> %CMDFILE%
echo # >> %CMDFILE%
echo # Archive current log file >> %CMDFILE%
echo SQL 'alter system archive log current' ; >> %CMDFILE%
echo # >> %CMDFILE%
echo # Backup our database >> %CMDFILE%
echo BACKUP DATABASE tag FULL_%DTE% ; >> %CMDFILE%
echo # >> %CMDFILE%
echo # Backup our archive log files that have not been backed up  >> %CMDFILE% 
echo BACKUP ARCHIVELOG ALL DELETE INPUT tag ARCH_%DTE% ; >> %CMDFILE%
echo # >> %CMDFILE%
echo # put controlfile to trace >> %CMDFILE%
echo sql 'ALTER DATABASE BACKUP CONTROLFILE TO TRACE' ; >> %CMDFILE%
echo # >> %CMDFILE%
echo # report and delete unrecoverable and obsolete >> %CMDFILE%
echo REPORT UNRECOVERABLE ; >> %CMDFILE%
echo REPORT OBSOLETE ORPHAN ; >> %CMDFILE%
echo # >> %CMDFILE%
echo DELETE NOPROMPT OBSOLETE ; >> %CMDFILE%
echo DELETE NOPROMPT EXPIRED BACKUP ; >> %CMDFILE%
echo # >> %CMDFILE%
echo CROSSCHECK BACKUPSET ; >> %CMDFILE%
echo # >> %CMDFILE%
echo # list backup details and summary >> %CMDFILE%
echo LIST BACKUP ; >> %CMDFILE%
echo LIST BACKUP SUMMARY ; >> %CMDFILE%

REM # Run the RMAN backup without using the recovery catalog
%ORACLE_HOME%\bin\rman target=sys/@ catalog rman/@ cmdfile=%CMDFILE% msglog=%LOG%

REM # Remove our temporary command file
del /q %CMDFILE%

REM # Goto end of script as we are done
goto END

:USAGE
echo "Must Pass ORACLE_SID as parameters."
goto END

:END

In addition to the daily full database backup, and as per Anwar's suggestion/comment to my 1st RMAN Implementation post on this blog, I scheduled the backup of archive logs at an interval of 3 hours during working hours.

The scheduled jobs are running perfectly fine, and I was able to perform Cloning to set up Test Instance using production RMAN backup. I will be posting more on how to Clone/Duplicate a Database using RMAN Duplicate Command.

Published my first Oracle Technical Whitepaper

Dear Friends & Colleagues,

Today, with the grace of Almighty, I have achieved one of the major milestones of my career.

I have successfully published my Oracle Technical Whitepaper titled "My Journey from 9.0.1 to 9.2.0.8" on Oracle Customer Knowledge Exchange (https://communities.oracle.com/). The paper is based on my work/accomplishments as an Oracle Database Administrator here in Integral Services Company WLL. On this blog, you know it as "Journey from 9.0.1 to 9.2.0.8: Part I-IV".

Though the subject, primarily, is based on the migration of Oracle Database from one version to another, I have tried to touch the various areas that were somehow related to the subject.

The access to the published document in Oracle Customer Knowledge Exchange would require customer logon credentials. So, you can email me at zk.oracle@gmail.com for the PDF file.