BSMC 9.2x - About logfile rollover management with Log File policies

  • KM01571099
  • 14-May-2015
  • 14-May-2015

Summary

How the BSM Connector 9.2x manages rollover of logfiles monitored by Log File policies

Question

Users of BSM Connector 9.2x might have the need to monitor some logfiles in some of their critical servers. This can be achieved with "Event > Log File" policies.

However there might be some concerns about these monitored logfiles that may grow too much and fill the file system where they are located. For this reason, a rollover mechanism like the following one is widely used among customers:

myfile.log
myfile.log.1
myfile.log.2
myfile.log.3
...

where the main logfile myfile.log gets a new extension as soon as it reaches a particular size.

What it is unclear is how to manage this situation with "Event > Log File" policies because it is not clear if the pointer of the policy to the file is lost when the logfile rolls between two intervals because this would lead to events not sent from the BSMC to the target server, i.e. a BSM/OMi.

Customers who come from the classic OM world were used to work with the opcle process of the classic OA (Operations Agent) which offers the following method to work with dynamic logfiles like our customer's ones (source: https://softwaresupport.hp.com/group/softwaresupport/search-result/-/facetsearch/document/KM822367):

------- snip --------
A) The latest entries  added to the old logfile will be processed and the new logfile (same name, different i-node)  will be read after 20 intervals (due to OPC_LE_CHECK_INODE variable default value) and all the info from the old log file would be processed:

-  do not enable the check "Close after read"
-  keep the default value FALSE for variable OPC_LE_CALC_HASH in the Agent configuration
------- snip --------

where OPC_LE_CALC_HASH has this description:

If the value of OPC_LE_CALC_HASH is TRUE, opcle detects the file is appended or overwritten by calculating hash of last line and verifying
some random check points.If opcle detects the file is overwritten, then it reads the file from beginning.

But, now with the BSMC, the logfile monitoring is not carried out by opcle anymore; it is opcgeni which is responsible for this scenario.
The Bonline help section related to the “Event > Logfile” policies doesn't show anything related to this and the “Event > Logfile” policy type doesn't offer anymore the possibility to enable/disable the “Close file after Reading” option as it currently happens in the OMW/OMU/OML policy editors for Logfile Encapsulator policies.

However, this option is present the “Event > XML” policy type. Notice also that this policy type offers the possibility to work with wildcards ( <*> ) to match different files with similar names.

Answer

Actually the “Event > Log File” policies are the EMS/SiteScope monitors – the source file handling and reading is therefore managed by the EMS engine of the BSMC not by the OA running underneath.

The flow is the following. The policy created in BSMC is transformed into the EMS monitor and sent/deployed via web services into the SiteScope. The EMS monitor reads the log file (locally or remotely) and writes the output into a temporary policy xml file (located in %OvDataDir%tmp). Then opcgeni reads the temp xml file(s), creates messages and sends them to OMi.

So opcgeni has nothing to do with the source log files. As for opcgeni itself, yes, it can handle the rollover of xml files.

For the EMS part, this is what the online help section shows:

Integrating Data With BSM Connector > Log File Policies > Log File Policy User Interface > Configuring the Data Source in Log File Policies

->How to configure the log file source

Some pattern matching is mentioned:

Optionally, you can use a regular expression to insert date and time variables. For example, you can use a syntax of s/ex$shortYear$$0month$$0day$.log/ to match date-coded IIS log file names. For details, see Special Substitution for File Path.

If you click on the link:

BSM Connector Date Variables are useful for matching content as part of a regular expression. The date variables can also be used as a special substitution to dynamically create file paths for log file policies. This is useful for accessing date-coded files and directories where the file path is updated automatically based on system date information. BSM Connector is an example of an application that creates date-coded log files. The log file names include some form of the year, month, and day as part of the file name, such as File2001_05_01.log, where the year, month, and date are included.

Based on this example, a new file is created each day. Checking the creation, size, or content of the current days file would normally require the file path of the log file policy to be manually changed each day. Using the BSM Connector date variables and special substitution, BSM Connector can automatically update the file path to the current day's log file. By knowing the pattern used in naming the files, you can construct a special substitution string similar to a regular expression that substitutes portions of the system date properties into the file path.

For example if the absolute file path to the current day's log file in a log file policy is:

D:/Production/Webapps/Logs/File2001_05_01.log

the log file for the following day would be:

D:/Production/Webapps/Logs/File2001_05_02.log

You can construct a special substitution expression to automatically update the file path used by the policy, with the following syntax:

s/D:\/Production\/Webapps\/Logs\/File$year$_$0month$_$0day$.log/

The substitution requires that the expression start with a lower-case s and that the expression is enclosed by forward slashes /.../. Forward slashes that are part of the file path must be escaped by adding the back slash (\) character as shown. The BSM Connector date variables are separated by the underscore character literals. BSM Connector checks the system time properties each time the policy runs and substitutes with applicable values into the file path before accessing the file.

While the special substitution syntax is similar in syntax to the substitution syntax used in regular expressions, they are not the same. While all of the BSM Connector date variables can be used in match content regular expressions, the special substitution discussed here can not be used as part of a match content expression.