The agent is not picking up FTP events in real time. (NETIQKB72206)

When trying to monitor FTP logs, the agent is not picking up FTP events in real time. The events eventually appear, but there is a 2 or 3 minute lag before events show up.

SM seems to be reading the FTP logs for a while, but then it stops. If the FTP service is restarted, the agent starts reading the logs again.

Consider looking at the buffering of the IIS FTP log entries and see if a smaller buffer size makes sense in your environment. Consult the Microsoft support site and documentation for details and be aware of the performance implications associated with such changes.

 

You can also change the poll interval for the SM FTP applog provider to a smaller interval by creating a DWORD registry entry "FilePollInterval" in the following hive:

 

HKEY_LOCAL_MACHINE\SOFTWARE\NetIQ\Security
Manager\Configurations\<ConfigGroup Name>\Operations\Agent\Event
Providers\Application Log\<GUID of the applog provider>\<GUID of the
folder>

 

The value of the FilePollInterval is in seconds. Modify the value to ?30? to configure the provider to use 30-second pulling intervals.

There are two main factors that contribute to this behavior:

 

·          IIS FTP log buffering:

 

SM?s FTP applog provider monitors the FTP logs by reading the FTP log file. However, IIS uses log buffering technology, where it writes 64k of logs into a buffer before writing/flushing it to disk. If the monitored FTP site activities do not generate more than 64k of log entries, the logs stay in the buffer until the site is restarted or the log files are rolled in the next rollup schedule (set to once per day by default).

 

Until the actual logs are written/flushed to the physical log file on disk, the SM FTP applog provider can?t see the new log entries. Therefore, the provider appears to not be reading any new log entries, but in reality, those log entries have not made it to disk, where the provider can see it.

 

There are ways to control the buffering technology in IIS sites. These methods range from changing the buffer size to disabling the buffering altogether, where all new log entries will be written to disk immediately. The buffering settings are mainly controlled through registry entries such as DisableLogBuffering. Consult the Microsoft support site and documentation for further details on these controls.

 

·          IIS FTP pre-allocated log files:

 

The SM FTP applog provider determines if it needs to read new entries in a log file using two mechanisms:

 

o    It will register a file notification event, which means that if the file changed its size, name, or last write time, the operating system will notify the provider that there are new entries or changes to the file.

 

o    It will wait for the notification for 5 mins, and if no notification has been triggered, it will attempt to read the latest entries anyway, regardless of whether there was a change or not.

 

The IIS logging system will pre-allocate 64k chunks of empty space into the logging file and replace those spaces with actual log entries when new log entries are flushed into the disk.

 

The file last write time and size won't change, and since the file name never changes, a file change notification evet won?t be triggered by the operating system to the SM FTP applog provider. The SM FTP applog provider will have to rely on its pulling interval of 5 mins to see if there are new entries in the FTP log file. Therefore, the SM FTP log provider will look like it?s not getting data in real time.

Formerly known as NETIQKB72206