How do I rebuild a corrupted Sentinel Sonic database?

  • 3406033
  • 02-Nov-2006
  • 26-Apr-2012

Environment

Sentinel
Windows 2000 sp4
Windows 2003 sp1
Solaris 9

Situation

Sentinel Sonic message bus database corrupt
Symptom #1
Unable to connect to system with the following errors in DAS_0.0.log file:
SonicConnection.internalConnect] Failed to connect to Sonic Broker: java.net.ConnectException: Connection refused: tcp://system:10012||com.esecurity.common.communication.strategy.sonicstrategy.SonicStrategy|traceError|
Symptom #2
Database query failing with the following error:
"ORA-01502: index 'ESECDBA.EVT_EVT_ID_IDX' or partition of such index is in unusable state"

Resolution

Possible Cause #1

The database is corrupt, most likely during startup or shutdown.

In order to further diagnose this problem cd to $ESEC_HOME/3rdparty/SonicMQ6.1/bin and run the startcontainer.sh script. If there is a problem with the the broker a message like the following will appear:

java.lang.NullPointerException

and

java.lang.RuntimeException: SonicMQ Broker start failure

If exceptions or failures of this kind are seen the sonic database will need to be purged and rebuilt. Use the following steps.

To un-corrupt the database

1.su to root and cd to /etc/rc3.d

2.Move the S98sentinel to xS98Sentinel and If present, move S99wizard to xS99wizard

3.Reboot. The above will insure that no Sentinel processes are running.

4.Login as esecadm and cd to $ESEC_HOME/3rdparty/SonicMQ6.1

5.Back up the SonicMQDB directory and the Sonic log directory

6.cd to $ESEC_HOME/3rdparty/SonicMQ6.1/bin

7.Purge the database, run ./dbtool.sh -d a

8.Reload the database, run ./dbtool.sh -c a

9.cd to $ESEC_HOME/scripts and start the Sentinel processes, run ./sentinel.sh start

10.Look for errors.

11.Start Wizard (if you have one), cd to $WORKBENCH_HOME and run ./agent-manager.sh start

12.su to root and move xS98sentinel to S98Sentinel and If applicable move xS99wizard to S99wizard.

Possible Cause #2

Partition operation was performed on a partition that contains data. The local indexes need to be rebuilt.

To rebuild the local indexes

1.Connect to SQL*Plus as ESECDBA.

2.At the SQL prompt (SQL>) enter:

SQL> select p.index_name, p.partition_name from user_ind_partitions p, user_indexes i where p.index_name = i.index_name and p.status <>'USABLE';

SQL> alter index rebuild partition;