DP 9.x DEBUG OB2DBG* files is still generated from an old / non-existing session.

  • KM02785794
  • 03-May-2017
  • 25-Mar-2021

Summary

Issue with DP DEBUG OB2DBG* files even with debugging is permanently disabled. Hang or orphan process is still creating debug files.

Error

Issue : DP 9.x DEBUG OB2DBG* files is still generated from an old / non-existing session.

Workaround - manual delete of newly created "old" OB2DBG* files.

DP DEBUG parameters have been disabled and checked on all possible area - omnirc, trace, GUI, scheduler, DP CLI, etc.

No new DP DEBUGS is created for new sessions.

The one being created (as per customer example) is session 2017/03/07-2. It has been checked April 10, and definitely more than a month has passed.

Suspect is a hang process that is still running.

Cause

Hang oracle.exe process

First to check are DP processes and checking the DP DEBUGS on the DA side. Nothing on the CM side.

-rwxrwx---+ 1 root  Admin 6209162672 Mar 19 12:40 OB2DBG_4476_2017-03-07-2_BDA-NET_PROD-DB.acme.com_4440-10072_DBbackup.txt
-rwxrwx---+ 1 root  Admin   10221852 Mar 18 13:23 OB2DBG_4476_2017-03-07-2_BDA-NET_PROD-DB.acme.com_4440-10472_DBbackup.txt
.
.
-rwxrwx---+ 1 root  Admin   23597416 Mar 18 13:18 OB2DBG_4476_2017-03-07-2_BDA-NET_PROD-DB.acme.com_4440-9736_DBbackup.txt
-rwxrwx---+ 1 root  Admin   25400088 Mar 17 13:21 OB2DBG_4476_2017-03-07-2_BDA-NET_PROD-DB.acme.com_4440-9952_DBbackup.txt
-rwxrwx---+ 1 root  Admin   12818283 Mar 20 01:33 OB2DBG_4476_2017-03-07-2_OB2BAR_SBT_CHANNEL_PROD-DB.acme.com_4440-10308_DBbackup.txt
-rwxrwx---+ 1 root  Admin   21698960 Mar 21 01:34 OB2DBG_4476_2017-03-07-2_OB2BAR_SBT_CHANNEL_PROD-DB.acme.com_4440-17008_DBbackup.txt
.

.
-rwxrwx---+ 1 root  Admin   37172303 Mar 18 13:24 OB2DBG_4476_2017-03-07-2_OB2BAR_SBT_CHANNEL_PROD-DB.acme.com_4440-8648_DBbackup.txt
-rwxrwx---+ 1 root  Admin 1008317402 Mar 19 13:01 OB2DBG_4476_2017-03-07-2_OB2BAR_SBT_CHANNEL_PROD-DB.acme.com_4440-9392_DBbackup.txt

Take note of the highlighted “4440” in RED.

As per DP DEBUG file definition, this is the PROCESS ID – which is technically one half – the second one is the “spawned” process ID.

Checking the tasklist output :

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
oracle.exe                    4440 Services                   0  2,953,480 K

You have actually two “oracle.exe”

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
oracle.exe                    5644 Services                   0    144,752 K
oracle.exe                    4440 Services                   0  2,953,480 K

The Data Protector Oracle integration links the Oracle database management software with Data Protector. From the Oracle point of view, Data Protector represents a media management software. On the other hand, the Oracle database management system can be seen as a data source for backup, using media controlled by Data Protector.

Components
The software components involved in backup and restore processes are:

- The Oracle Recovery Manager (RMAN)
- The Data Protector Oracle integration software

How does the integration work?

ob2rman.pl executes RMAN, which directs the Oracle server processes on the target database to perform backup, restore and recovery. RMAN maintains the required information about the target databases in the recovery catalog, the Oracle central repository of information, and in the control file of a particular target database.


So basically what happen is that when you run the next DP/Oracle backup, the two oracle.exe will accept the request from RMAN. The valid “oracle.exe” enables you to execute and complete the backup. The “problem oracle.exe” (PID 4440) will run but it does not do any backup (I/O read) since and it just create the DEBUG files. For some reason from the previous 07 Mar incident, you still have this oracle.exe PID 4440 still running when it should no longer exists.

Fix

RECOMMENDATION:

Request for a schedule to do a ORACLE SERVER reboot (restart).

The main objective is to clear/clean the process table of any hang/orphaned/zombie processes. With this server reboot / restart, follow your normal shutdown and startup procedure so that everything will be clean – can close any open DATABASES and ensure no users are logged-in (and also NO DP BACKUPS or other sessions or open DP GUI).