Environment
NetIQ Sentinel 7.1 Sentinel Server
Situation
The Server0.0.log has numerous entries of the following SEVERE error.
on Jul 29 17:22:41 EDT 2013|SEVERE|Raw Data task monitor Timer|esecurity.ccs.comp.event.rawdata.RawDataConsumer.moveOnlineCompressedRawDataFilesToArchive
Error RAWDATA0001(RawData): Error moving raw data file /var/opt/novell/sentinel/data/rawdata/online/3C41F682-C87B-1030-AAEE-0050569E00A6/2013-07/10-0600.zip to archive /gac-archive/40ED9DA0-2044-102F-8C36-0050569E00A4/rawdata_archive.
Root cause: /var/opt/novell/sentinel/data/rawdata/online/3C41F682-C87B-1030-AAEE-0050569E00A6/2013-07/10-0600.zip (No such file or directory) (java.io.FileNotFoundException)
com.novell.sentinel.util.logging.error.rawdata.RAWDATA0001: Error moving raw data file /var/opt/novell/sentinel/data/rawdata/online/3C41F682-C87B-1030-AAEE-0050569E00A6/2013-07/10-0600.zip to archive /gac-archive/40ED9DA0-2044-102F-8C36-0050569E00A4/rawdata_archive.
at esecurity.ccs.comp.event.rawdata.RawDataConsumer.moveOnlineCompressedRawDataFilesToArchive(RawDataConsumer.java:840)
at esecurity.ccs.comp.event.rawdata.RawDataConsumer$RawDataArchivingTask.run(RawDataConsumer.java:707)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: /var/opt/novell/sentinel/data/rawdata/online/3C41F682-C87B-1030-AAEE-0050569E00A6/2013-07/10-0600.zip (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(Unknown Source)
at esecurity.base.util.FileUtil.copyFile(FileUtil.java:168)
at esecurity.base.util.FileUtil.copy(FileUtil.java:697)
at esecurity.base.util.FileUtil.move(FileUtil.java:686)
at esecurity.ccs.comp.event.rawdata.RawDataConsumer.moveOnlineCompressedRawDataFilesToArchive(RawDataConsumer.java:825)
... 3 more
Error RAWDATA0001(RawData): Error moving raw data file /var/opt/novell/sentinel/data/rawdata/online/3C41F682-C87B-1030-AAEE-0050569E00A6/2013-07/10-0600.zip to archive /gac-archive/40ED9DA0-2044-102F-8C36-0050569E00A4/rawdata_archive.
Root cause: /var/opt/novell/sentinel/data/rawdata/online/3C41F682-C87B-1030-AAEE-0050569E00A6/2013-07/10-0600.zip (No such file or directory) (java.io.FileNotFoundException)
com.novell.sentinel.util.logging.error.rawdata.RAWDATA0001: Error moving raw data file /var/opt/novell/sentinel/data/rawdata/online/3C41F682-C87B-1030-AAEE-0050569E00A6/2013-07/10-0600.zip to archive /gac-archive/40ED9DA0-2044-102F-8C36-0050569E00A4/rawdata_archive.
at esecurity.ccs.comp.event.rawdata.RawDataConsumer.moveOnlineCompressedRawDataFilesToArchive(RawDataConsumer.java:840)
at esecurity.ccs.comp.event.rawdata.RawDataConsumer$RawDataArchivingTask.run(RawDataConsumer.java:707)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: /var/opt/novell/sentinel/data/rawdata/online/3C41F682-C87B-1030-AAEE-0050569E00A6/2013-07/10-0600.zip (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(Unknown Source)
at esecurity.base.util.FileUtil.copyFile(FileUtil.java:168)
at esecurity.base.util.FileUtil.copy(FileUtil.java:697)
at esecurity.base.util.FileUtil.move(FileUtil.java:686)
at esecurity.ccs.comp.event.rawdata.RawDataConsumer.moveOnlineCompressedRawDataFilesToArchive(RawDataConsumer.java:825)
... 3 more
Resolution
To resolve this issue upgrade to Version: 7.1.0.1_904. Once upgraded the issue will be resolved from this point forward, however the past errors need to be addressed. You will need to update the raw_data_info_file table in the postgres db so that the status shows ARCHIVED for the files (partitions) that have already been moved. This table can be updated using pssql.
Log into an SSH session and do an su -novell NOTE: Make sure that you do a su -novell before running these pssql commands. The root user will not work.
If the file has been archived and you need to change the file status to an archived state run the following query. Just to make sure you are editing the correct rows in the raw_data_info_file table in postgres, you should use the errors in the server0.0.log to determine which guid to plug into the query below.
NOTE: if you are cutting and pasting these commands, double check the syntax after pasting the command.
1. log in as root
2. su -novell
3. go to /opt/novell/sentinel/bin
4. ./db.sh sql SIEM dbauser ‘’update raw_data_files_info set state = 'ARCHIVED' where file_name = ‘/71832380-7603-102F-88CA-000C295A7242/2012-08/01-1500.zip'”
The above query is an example of updating the status to ARCHIVED. If you were to do one row at a time the guid that would be used is in the SEVERE events in the server0.0.log. However if you want to update mulitple rows the below query will also work. You will need to edit it so that a month of rows are changed. What you want to avoid doing is updating a row that is active and truly hasn't been archived which means when everything is working properly you should only have active rows for the last couple of hours.
If you want to update everything for the month of lets say july 2013 the query would look like this.
./db.sh sql SIEM dbauser ‘’update raw_data_files_info set state = 'ARCHIVED' where file_name like ‘%2013-07/%.zip'”
After you have completed updating the rows in the raw_data_info_file table run the following query to confirm the ARCHIVED status for the month of 09.
./db.sh sql SIEM dbauser "select file_name, state from raw_data_files_info where file_name like '%2013-09/%.zip'"
the resulting query will have entries similiar to the following.
/0738DED0-E3EE-102F-B247-000C29C01195/2013-09/11-2300.zip | ARCHIVED
/0738DED0-E3EE-102F-B247-000C29C01195/2013-09/12-0000.zip | ARCHIVED
/0738DED0-E3EE-102F-B247-000C29C01195/2013-09/12-0800.zip | ARCHIVED
/0738DED0-E3EE-102F-B247-000C29C01195/2013-09/12-0900.zip | ARCHIVED
/833424E2-E475-102F-88D1-000C29C01195/2013-09/03-1500.zip | COMPRESSED
/833424E0-E475-102F-A943-000C29C01195/2013-09/03-1500.zip | COMPRESSED
/833424E2-E475-102F-84F2-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/833424E0-E475-102F-A946-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/DBB80503-C52A-1030-A083-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/E6694016-BFD8-1030-B558-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/833424E2-E475-102F-AFA8-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/0738DED0-E3EE-102F-B247-000C29C01195/2013-09/12-1000.zip | ARCHIVED
/0738DED0-E3EE-102F-B247-000C29C01195/2013-09/12-0000.zip | ARCHIVED
/0738DED0-E3EE-102F-B247-000C29C01195/2013-09/12-0800.zip | ARCHIVED
/0738DED0-E3EE-102F-B247-000C29C01195/2013-09/12-0900.zip | ARCHIVED
/833424E2-E475-102F-88D1-000C29C01195/2013-09/03-1500.zip | COMPRESSED
/833424E0-E475-102F-A943-000C29C01195/2013-09/03-1500.zip | COMPRESSED
/833424E2-E475-102F-84F2-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/833424E0-E475-102F-A946-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/DBB80503-C52A-1030-A083-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/E6694016-BFD8-1030-B558-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/833424E2-E475-102F-AFA8-000C29C01195/2013-09/09-1500.zip | COMPRESSED
/0738DED0-E3EE-102F-B247-000C29C01195/2013-09/12-1000.zip | ARCHIVED
NOTE: The COMPRESSED status indicates the file has not been moved to archive. This is normal if it is for the current hour.
Also look at the current server0.0.log to confirm that the SEVERE errors moving raw data files is not longer occurring.
Cause
This was a known issue in Sentinel 7.1. There is a table in the postgres database named raw_data_info_file that keeps track of the raw data files. When the raw data is moved to the archive location the status in the table should change to archived. If that doesn't happen then sentinel thinks the file is still in the local storage and continues to try to move it. However since the file is no longer in local storage there will be errors when Sentinel is trying to find it. Before we resolved the issue the biggest problem it caused is a lot of events eating up the log files. Other than that there was no need to worry about it in most cases. Although in some larger environments it was best to get upgraded just in case the continuous logging caused performance issues.