Integrating Data With BSM Connector > Structured Log File Policies > Structured Log File Policy User Interface > Configuring the Data Source in Structured Log File Policies

Configuring Data Source in Structured Log File Policies

The source page of the structured log file policy editor enables you to specify which log file the policy reads. You can also configure the policy to extract data, to which the log file structure is applied, from the log file. The policy retains that structured data for later reuse in other pages of the policy editor.

Learn More

Tasks

UI Descriptions

UI Element Description
Structured Logfile source

Log File Path / Name

Path and name of the structured log file that the policy reads.

Note: BSM Connector cannot process log files that are larger than 2 GB.

Polling Interval

Determines how often the policy reads the structured log file (in days, hours, minutes, and seconds). This period of time is the polling interval. The larger polling interval is set, the less performance is needed. However, more memory is used (this depends on the amount of the data in the log file). Setting the polling interval below 30 seconds is not recommended, the default setting is usually appropriate.

Note that a policy begins to evaluate data after the first polling interval passes, unless the Read from beginning (first time) or Read from beginning (always) radio button is selected, which results in evaluation of the already existing data at the policy activation. A shorter polling interval is better when you are testing a policy.

Default value: 5 minutes

Note: Make sure that you set this value to a minimum of 15 seconds to be able to save the policy.

Logfile Character Set

Name of the character set used by the structured log file that the policy reads.

Note: It is important to choose the correct character set. If the character set that the policy is expecting does not match the character set in the structured log file, pattern matching may not work, and the event details can have incorrect characters or be truncated in OMi. If you are unsure of which character set is used by the structured log file that the policy reads, consult the documentation of the program that writes the file.

Default value: UTF-8

Send event if log file does not exist

BSM Connector sends an event if the specified structured log file does not exist.

Default value: not selected

Close after reading

If you select this option, the file handle of the structured log file closes and reopens again after the polling interval. The file is read from the last position. If this file had a rollover in the meantime, it is read from the beginning. If the name of the structured log file changes, and a new file was started in the meantime, the policy continues to read the new structured log file and the original structured log file data is lost.

If you do not select this option, the file handle remains and is read entirely each time, unless there is a newer file with the same name (or name pattern). In that case the original structured log file is read to the end, and then the newer file is read. Therefore, no data is lost.

Consider the following example: a policy reads the structured log file system.log.While there is still some unread data in the system.log file, it is renamed to system_Monday.log, and a new version of the system.log file is written.

In case this option is selected, the unread data from the system_Monday.log file remains unread and the system.log file is read entirely.

In case this option is not selected, the unread data from the new system.log file is read after the reading of the system_Monday.log file is completed.

Default value: not selected

Read Mode

The read mode of a structured log file policy indicates whether the policy processes the entire file or only new entries.

Read from last position. The policy reads only new—appended—entries written in the structured log file while the policy is activated. If the file decreases in size between readings, then the entire file is read. Entries that are added to the file when the policy is disabled are not processed by the policy.

Choose this option if you are concerned only with entries that occur when the policy is enabled.

Advantage: No chance of reading the same entry twice. (Unless the file decreases in size because some entries were deleted.)

Disadvantage: Entries written to file while the policy is disabled or the agent is not running are not processed by the policy.

Read from beginning (first time). The policy reads the complete structured log file each time the policy is activated or the agent restarts. This ensures that all entries in the file are compared with the rules in the policy. Each successive time that the policy reads the file, only new (appended) entries in the file are processed.

Choose this option if you want to ensure that every existing and future entry in the file is processed by the policy while it is activated.

Advantage: Every existing and future entry in the file will be processed by the policy.

Disadvantage: Duplicate entries can occur if an activated policy is deactivated and reactivated, or if the agent stops and restarts.

Read from beginning (always). The policy reads the complete structured log file every time it detects that the file has changed. The policy scans the file at the specified polling interval. If no change is detected, the file is not processed. Any entries overwritten while the agent is not running or the policy is deactivated will not be evaluated by the policy.

Choose this option if the policy reads a file that is overwritten, rather than appended.

Advantage: Ensures that files that are overwritten are correctly processed.

Disadvantage: Only valid for files that are overwritten, rather than appended.

Note: Every policy reads the same structured log files independently from any other policies. This means, for example, that if "Policy 1" with read mode Read from beginning (first time) is activated and "Policy 2" with the same read mode already exists, "Policy 1" still reads the entire file after it has been activated.

Default value: Read from last position

Sample Data

Loads the log file into BSM Connector. The log file can be loaded from Server or from the local file system.

Note: BSM Connector can only load a maximum of 50 MB of sample data.

Opens the Structured logfile sample data dialog box. This dialog box contains the following tabs:

  • Raw data

    Raw data tab displays the actual lines of the uploaded log file marked with the numbers. For example:

    1380004749|tcpc113.RIESLING.INTERN|LogicalDisk|C:|% Free Space|66.379264831543|Microsoft.Windows.Server.2008.LogicalDisk

  • Structured data

    Structured data tab displays the lines of the uploaded log file after the log file pattern is applied. These lines are now divided by the fields that were identified by the applied pattern. These field can be used throughout the rest of the policy configuration, for example, as Input Data Properties from the Sample Data in the Mappings and the Defaults tabs. For example, for the above stated example log line, these fields would be the following:

    timestamp|hostname|entitytype|entityid|countername|
    countervalue|scomtype

 

Logfile Structure
Log File Pattern
(for events only)

A pattern by which the log file's structure is extracted, and which will be used in all other policy operations.This pattern should comply with the standard pattern definition used by all HP Operations Managerproducts (OM pattern). For example, this structure could like as follows: <*.timestamp>\|<*.hostname>\|<*.entitytype>\|<*.entityid>\|
<*.countername>\|<*.countervalue>\|<*.scomtype>

For more information about how this structure is extracted from a log file, see Configuring Data Source in Structured Log File Policies.

For more information about the OM pattern matching details, see Pattern Matching in Policy Rules.

Data Fields
(for metrics only)
identify using OM Pattern

Identifying the log file's structure components by using the OM pattern. For example:

<*.timestamp>\|<*.hostname>\|<*.entitytype>\|
<*.entityid>\|<*.countername>\|<*.countervalue>\|
<*.scomtype>

This field can contain the data that matches the whole line or just the static part of the log line. For more information about the OM pattern matching details, see Pattern Matching in Policy Rules.

Note: In case a value in the identify using static fields field is already specified, the value of this field is ignored.

identify using static fields

Identifying the log file's structure components by using static fields. For example:

timestamp,hostname,entitytype,entityid,
countername,countervalue,scomtype

For more information on static fields, see Defining a log file structure by using static fields (Metric only).

Tip: This is a recommended method (applies also for recurring fields) for extracting a structure from the log file due to performance reasons.

Recurring Fields
(for metrics only)

A word list that contains the recurring part from the log line. Each recurrence creates a record in the store. For example:

countername,countervalue

For more information about the recurring fields, see Using recurring fields in defining a log file structure (Metric only).

Data Field Separator
(for metrics only)
The separator that is used as a data separator in the log file.
Line Start Indicator (for events only)

Line Start Pattern

 

This field enables you to differentiate the structured log file entries based on their logical relationship, regardless of their span in the log file. You can do this by identifying a line start indicator from a log file, and then specifying the matched line start pattern by using the OM pattern matching language. For more information, see Setting the Line Start Indicator (Event only).