Environment
Novell Log Manager 1.2 Search
Situation
Event search was returning zero results.
The server0.0 through server0.9.log files would fill up instantly upon restart with these messages.
Thu May 26 15:03:33 MST 2011|SEVERE|IndexedLogComponent.LoggerThread|Unknown.unknown Unable to log event.; Exception java.lang.ArrayIndexOutOfBoundsException;
Searching over the past 24 hours we found events would be returned but only up until a certain point in time. From that point on, there were zero events.
The customer had made no changes other than restarting the server.
The server0.0 through server0.9.log files would fill up instantly upon restart with these messages.
Thu May 26 15:03:33 MST 2011|SEVERE|IndexedLogComponent.LoggerThread|Unknown.unknown Unable to log event.; Exception java.lang.ArrayIndexOutOfBoundsException;
Searching over the past 24 hours we found events would be returned but only up until a certain point in time. From that point on, there were zero events.
The customer had made no changes other than restarting the server.
Resolution
The key to the problem is the error message being repeated in the
log files. The IndexedLogComponent is specifically related to the
event data indexing. Since the search was trying to go through
multiple files and their indexes, it was failing when hitting the
ArrayIndexOutOfBounds exception on the individual bad index. Since
we cannot be sure which specific index it is, we need to make an
educated guess based off the time the events stop showing up in a
search from the WebUI. Based off of this, you can extrapolate the
approximate date of the index, i.e. 05/25/2011 and start checking
from there.
Built into the ccsbase code is an index check and rebuild option. To run a check on a specific index, you need to run the following command;
java -classpath $APP_HOME/lib/ccsbase.jar esecurity.ccs.comp.event.indexedlog.IndexedLogCheck <path to index i.e. /var/opt/novell/sentinel_log_mgr/eventdata/UUID/>
Note that the command is IndexedLogCheck which indicates we are only checking the indexes. In this case, we would run the following;
java -classpath $APP_HOME/lib/ccsbase.jar esecurity.ccs.comp.event.indexedlog.IndexedLogCheck /var/opt/novell/sentinel_log_mgr/eventdata/20110525_408E7E50-C02E-4325-B7C5-2B9FE4853476/
Make sure to check each index for the time frame you feel they might contain corrupted indexes. If the index is fine, you will see;
Errors: 0
Warnings: 0
at the end. If you see any errors or warnings you should contact Novell support for information on how to rebuild the indexes.
Built into the ccsbase code is an index check and rebuild option. To run a check on a specific index, you need to run the following command;
java -classpath $APP_HOME/lib/ccsbase.jar esecurity.ccs.comp.event.indexedlog.IndexedLogCheck <path to index i.e. /var/opt/novell/sentinel_log_mgr/eventdata/UUID/>
Note that the command is IndexedLogCheck which indicates we are only checking the indexes. In this case, we would run the following;
java -classpath $APP_HOME/lib/ccsbase.jar esecurity.ccs.comp.event.indexedlog.IndexedLogCheck /var/opt/novell/sentinel_log_mgr/eventdata/20110525_408E7E50-C02E-4325-B7C5-2B9FE4853476/
Make sure to check each index for the time frame you feel they might contain corrupted indexes. If the index is fine, you will see;
Errors: 0
Warnings: 0
at the end. If you see any errors or warnings you should contact Novell support for information on how to rebuild the indexes.