Summary
Error
Issue : DP 9.x DEBUG OB2DBG* files is still generated from an old / non-existing session.
Workaround - manual delete of newly created "old" OB2DBG* files.
DP DEBUG parameters have been disabled and checked on all possible area - omnirc, trace, GUI, scheduler, DP CLI, etc.
No new DP DEBUGS is created for new sessions.
The one being created (as per customer example) is session 2017/03/07-2. It has been checked April 10, and definitely more than a month has passed.
Suspect is a hang process that is still running.
Cause
Hang oracle.exe process
First to check are DP processes and checking the DP DEBUGS on the DA side. Nothing on the CM side.
-rwxrwx---+ 1 root Admin 6209162672 Mar 19 12:40 OB2DBG_4476_2017-03-07-2_BDA-NET_PROD-DB.acme.com_4440-10072_DBbackup.txt
-rwxrwx---+ 1 root Admin 10221852 Mar 18 13:23 OB2DBG_4476_2017-03-07-2_BDA-NET_PROD-DB.acme.com_4440-10472_DBbackup.txt
.
.
-rwxrwx---+ 1 root Admin 23597416 Mar 18 13:18 OB2DBG_4476_2017-03-07-2_BDA-NET_PROD-DB.acme.com_4440-9736_DBbackup.txt
-rwxrwx---+ 1 root Admin 25400088 Mar 17 13:21 OB2DBG_4476_2017-03-07-2_BDA-NET_PROD-DB.acme.com_4440-9952_DBbackup.txt
-rwxrwx---+ 1 root Admin 12818283 Mar 20 01:33 OB2DBG_4476_2017-03-07-2_OB2BAR_SBT_CHANNEL_PROD-DB.acme.com_4440-10308_DBbackup.txt
-rwxrwx---+ 1 root Admin 21698960 Mar 21 01:34 OB2DBG_4476_2017-03-07-2_OB2BAR_SBT_CHANNEL_PROD-DB.acme.com_4440-17008_DBbackup.txt
.
.
-rwxrwx---+ 1 root Admin 37172303 Mar 18 13:24 OB2DBG_4476_2017-03-07-2_OB2BAR_SBT_CHANNEL_PROD-DB.acme.com_4440-8648_DBbackup.txt
-rwxrwx---+ 1 root Admin 1008317402 Mar 19 13:01 OB2DBG_4476_2017-03-07-2_OB2BAR_SBT_CHANNEL_PROD-DB.acme.com_4440-9392_DBbackup.txt
Take note of the highlighted “4440” in RED.
As per DP DEBUG file definition, this is the PROCESS ID – which is technically one half – the second one is the “spawned” process ID.
Checking the tasklist output :
Image Name PID Session Name Session# Mem Usage
========================= ======== ================ =========== ============
oracle.exe 4440 Services 0 2,953,480 K
You have actually two “oracle.exe”
Image Name PID Session Name Session# Mem Usage
========================= ======== ================ =========== ============
oracle.exe 5644 Services 0 144,752 K
oracle.exe 4440 Services 0 2,953,480 K
The Data Protector Oracle integration links the Oracle database management software with Data Protector. From the Oracle point of view, Data Protector represents a media management software. On the other hand, the Oracle database management system can be seen as a data source for backup, using media controlled by Data Protector.
Components
The software components involved in backup and restore processes are:
- The Oracle Recovery Manager (RMAN)
- The Data Protector Oracle integration software
How does the integration work?
ob2rman.pl executes RMAN, which directs the Oracle server processes on the target database to perform backup, restore and recovery. RMAN maintains the required information about the target databases in the recovery catalog, the Oracle central repository of information, and in the control file of a particular target database.
So basically what happen is that when you run the next DP/Oracle backup, the two oracle.exe will accept the request from RMAN. The valid “oracle.exe” enables you to execute and complete the backup. The “problem oracle.exe” (PID 4440) will run but it does not do any backup (I/O read) since and it just create the DEBUG files. For some reason from the previous 07 Mar incident, you still have this oracle.exe PID 4440 still running when it should no longer exists.
Fix
RECOMMENDATION:
Request for a schedule to do a ORACLE SERVER reboot (restart).
The main objective is to clear/clean the process table of any hang/orphaned/zombie processes. With this server reboot / restart, follow your normal shutdown and startup procedure so that everything will be clean – can close any open DATABASES and ensure no users are logged-in (and also NO DP BACKUPS or other sessions or open DP GUI).