Summary
Adding a notification attribute to a message forwarded by the NNMi intergation via Web Services
Question
WSDL Integration is configured between NNMi 8.11 and OMU 8.31. NNMi events are successfully received by OMU. However, the integration events are showing a source of "MSI:j12345678" and we cannot process them with OMU policies. We need to identify the proper configuration to process these events with OMU policies so that we can enable "notification" at the condition level.
Answer
In order to add the notification attribute it is necessary to further process the event on the server. The easiest way to process this event without writing a MSI application is to use Correlation Composer.
1. Create or modify /etc/opt/OV/share/conf/OpC/mgmt_sv/msiconf to include:
j* -1
opcecm 100
This determines the order which the instances (that is other processes which use the message manager message stream) message manager (opcmsgm) processes which instance which. The j* -1 means use the Web Services instance before anything else. The event correlation manager process (opcecm) will be processed later, else what usually happens is that opcecm will register before the Web Service (WS) instance. In this case, the WS interface will not be processed by opcecm.
2. Enable the message stream interface (MSI) for the server:
Actions > Server > Configure > Send all messages to MSI > select divert all messages
3. Install and assign the Correlation Composer polices to the server:
Actions > Server > Assign Templates > Add "Correlation Composer Group"
Actions > Server > Install/Update Server Templates
4. Check that opcecmas and opcecm are running on the server (opposed to opceca on a managed node).
5. Create the fact store or use the attached fact store. It should be something like:
The policy type is enhance and is defined as:
Name: MSIintercept.fs.
Definition:
Alarm signature: Field=GROUP Operator=matches value"NNMi" (with test in double quotes)
Variables: name=TRUE Type=Constant value=1
Select the "Enhance Always" option
Note: the Alarm signature in this example will match message where the group matches NNMi. That may need to be changed to meet the requirement.
New Alarm tab:
Alter specification
Field=NOTIFICATION Mode=Replace Value=TRUE
Save the new fact store under /etc/opt/OV/share/conf/OpC/mgmt_sv/CO/ only because when using "ovocomposer -install -ms" all the fact stores will be shown in a list (see below).
Start the correlation composer interface by setting DISPLAY and running ovocomposer -ui.
6. Install the fact store to the OMU server instance:
/opt/OV/bin/OpC/ovocomposer -install -fs MSIintercept.fs -fs yourseveritychange.fs -ms
Note: yourseveritychange.fs is only needed if there are composer policies. If this the only one, then just MSIintercept.fs.
OR install and select the fact stores manually:
# /opt/OV/bin/OpC/ovocomposer -install -ms
List of Correlator stores in /etc/opt/OV/share/conf/OpC/mgmt_sv/CO/:
0) empty.fs
1) MSIintercept.fs
2) demo.fs
Enter numbers (comma separated) to select stores:1,2
Merge /etc/opt/OV/share/conf/OpC/mgmt_sv/CO/MSIintercept.fs and /etc/opt/OV/share/conf/OpC/mgmt_sv/CO/demo.fs into ecs_comp.fs.out.1
Merging of Correlator Stores successful.
Note: More information can be found in the /var/opt/OV/share/log/ecs/csmerge.log
NOTE: To disable Composer - either install "empty.fs" or deassign "Correlation Composer" from the policy assigned to the server.
Check to see if it's working by looking at the message properties - the events from NNMi should now have notification attribute set.
7. Troubleshooting: Check the fact store policies are loaded:
7.1 Check for "ecs_comp" in the ecsmgr -i 11 -stats output (great for troubleshooting):
# ecsmgr -i 11 -stats
Engine Statistics -
input.inputFilters = [((), (), ())]
engineInstance = 11
currentTime = 20090325181510.000000Z
enginelog.errors = 4
enginelog.warnings = 1
enginelog.info = 4
Stream Statistics -
Stream "default" -
default.in.input = 23
default.in.new = 3
default.out.output = 23
default.out.discarded = 3
default.out.undecided = 0
default.out.errors = 0
default.original.output = 20
default.policy.num = 20
Circuit Statistics -
Circuit "ecs_comp" - # << this is the ecs_comp circuit name
ecs_comp.in.input = 0 # << these next lines show how many
ecs_comp.in.new = 0 events have been forwarded into
ecs_comp.out.output = 0 and processed by ecs_comp.
ecs_comp.out.discarded = 0
ecs_comp.out.undecided = 0
ecs_comp.out.errors = 0
ecs_comp.original.output = 0
7.2 Check that both polices have been loaded by opcecm - you can also check by making sure the severities have not changed and that the NNMi events are being forwarded to the notification service:
# ecsmgr -i 11 -fact_dump ecs_comp /tmp/dump.out
Then look through /tmp/dump.out and look for the two fact store names (just vi the file, and seach for some keywords like NNMi and severity reduction).
NNMi forwards events to ovtomcatB, which then in turn forward the events to opcmsgm (message manager) by connecting directly to opcmsgm's MSI. This is why msiconf is important; if not set, opcecm (server correlation process) will usually connect to opcmsgm first. This is because the connection to ovtomcatB is only established when there is something to send over. In this case, we need ovtomcatB forward the message to opcmsgm before opcecm (which contains composer) processes the event.
The WS MSI instance with name starts /var/opt/OV/share/tmp/OpC/mgmt_sv/j<instance>q and pipe file /var/opt/OV/share/tmp/OpC/mgmt_sv/j<instance>p (typically something like j34115824q j34115824p), but it keeps on changing and this is a problem. It's a problem because if events are forwarded into ECS as the order which events are forwarded on the MSI may be important and needs to be defined in /etc/opt/OV/share/conf/OpC/mgmt_sv/msiconf. It's difficult to keep the j*q/p files synchronised with the msiconf file.
QCCR1A89888 describes this problem and registering the interface name in the OVO database and making the interface name something obvious. QCCR1A89888 does not resolve any problems other than internal event processing/registration which does not effect how the policies are processed.