Syslog Proxy is dropping events

  • 3062373
  • 01-Nov-2006
  • 26-Apr-2012

Environment


Sentinel 5.1.3 Agent Manager
Syslog Proxy

Situation

After upgrading to Sentinel 5.1.3, syslog events are no longer being sent to the database. Connections that use TCP are still sending events but connections using UDP stop sending events.

There are no errors in the syslog_trace0.0.log or syslog_wrapper.log that indicate a problem. First you need to verify that the events are being sent to the syslog connections. You can do this by using any tracing utility such as ethereal or Sniffer etc...


Resolution

This will be addressed in a hotfix for Sentinel 5.1.3

There are two potential workarounds. The first is to set up a batch file or cron job to stop the syslog connector and restart it every 3-4 hours depending on Event Per Second rates.

The other option is to copy in the syslog.jar and the syslogconnector.jar from 5.1.2 as well as the config files. If you choose this option you should understand that you will now be dropping events if the eps reaches a point that fills the buffer.

Additional Information

Change: In Sentinel 5.1.3 the Syslog was enhanced to avoid dropping of events. In the past if the buffer became full it would simply drop events until buffer space could be cleared up to accommodate the incoming messages. An enhancement was made to the Syslog connector in 5.1.3 to help avoid dropping events. One part of this change was the ability for the buffer to send a wait or sleep message to the connections for TCP and UDP. The reason for the wait/sleep command is to allow the buffer time to clear up some space and to avoid dropping events. The bug lies in the fact that the individual connections would wait to send messages to the buffer until it sent a "wake up" message. Unfortunately, the buffer was only programmed to send the "wake up" message to TCP connections so all UDP connections would never start sending events again until the Syslog was stopped and started again.