Summary
This documsent is intended to provide a high level description of NA device change detection, attribution and some troubleshooting tips.
Reference
There are different methods that NA uses to populate the "Changed by" field after a change has been made to a device.
We can start mentioningthosemethods to detect changes to a device configuration such as:
-
Syslog messages
-
AAA log reading
-
Internal proxy
From these methods, NA uses a number of different inputs to determine who actually made a change on the device. This information provides the most likely user responsible for the change. In order of priority, the following information is used:
-
User who scheduled a password change that was run on the device.
-
User who scheduled a software update that was run on the device.
-
User who deployed a configuration to the device.
-
User who ran a script on the device.
-
User who connected to the device via NA’s proxy.
-
User information gathered from AAA logs.
-
User information parsed out of a syslog message.
NA assigns a change attribution to a device interaction that is higher in the priority list in which one means the highest priority.
For example, if a user schedules a password change while another user had proxied to the device during the same time period, if a change had been detected, that change would be assigned to the user who had scheduled the password change.
Use Casesto populate the "Changed by" field:
1. The first is when a user deploys a device configuration change through the NA Web interface. If an NA user makes a configuration change using the Edit & Deploy configuration task, this device change configuration will be assigned to the user that ran the task. That user will be attributed to the change in the "Changed by" field.
2. A user logs into a device from the device home page by clicking "Connect" and then clicking on either of the connection methods. The changes that are made on the device while the user is logged in will be attributed to that user.
3. If a user logs directly onto a device (outside of NA) and makes a change, then NA will try to attribute the change in the device configuration in a couple more ways:
3.1 When the device is added to NA, NA will set the device to send syslog messages to NA for "real time change detection". NA has a built-in syslog server which is listening on a specific network port (514 by default) and gathering and analyzing syslog packets in the Network. If NA detects a configuration change syslog pattern; it will trigger a snapshot task and then assign that change to the user reported on that syslog message. It's important to recognize that what is sent depends on the syslog setting that is set up in the device. Some questions to ask would be:
Are syslog messages sent when the user logs off (this seems to be the default for most devices)?
Are syslog messages sent when a user leaves the config mode or when an actual change is made?
3.2 When a recurring scheduled snapshot task or a group snapshot task is scheduled to run in NA, it will gather all the device configuration changes. However, the task will assign "N/A" to the configuration change. Since NA has no way to verify who made the change, NA will put "N/A" in the "Changed by" field.
This being said, you can see changes associated to "N/A" in one of the following scenarios:
* The device is not sending any syslog message to NA when a configuration change is made. In this case, when NA takes a snapshot (scheduled snapshots) it will associate the changes to "N/A". You need to verify that the device supports syslog and is configured to send syslog messages to NA.
In order to enable the change detection feature in the Network Automation System, click on Admin -> Administrative Setting -> Configuration Mngt.
Enable "Change Detection" by clicking the checkbox and then specify the syslog patterns you expect your system will be looking for in your network. Then make sure that the syslog server is started in Admin --> Start/Stop Services.
Note: Syslog messages are UDP packets that could be lost in your network according to its workload.
Attribution is a very complex feature. To successfully attribute a particular user to a particular change, there are many factors that come into play.
Here are a few examples of what is important:
Are multiple users allowed to make changes on a device at the same time?
Is syslog set up on the device?
What triggers the device to send out a syslog message?
Was the NA syslog host set up on the device?
Is syslog set up on the device?
What triggers the device to send out a syslog message?
Was the NA syslog host set up on the device?
How often does NA do a group snapshot that includes the device?
Remember that NA only does one snapshot in 10 minutes (default). If a group snapshot is done and a user logs out of the device within the 10 minute window, he/she will not be attributed to the change and the change will be picked up at the next scheduled snapshot.
How busy is the network? It is a good question, remember that syslog messages are UDP and UDP packets are "best effort" delivery and are high on the drop priority. Also is valid to think is there are any firewalls that could be blocking traffic?
If a resolution is not found for an N/A attribution by the above rules, try to get more details; if Change Detection is enabled (as described above) and the device has logging set to the NA server, then increase the syslog logging option levels.
How busy is the network? It is a good question, remember that syslog messages are UDP and UDP packets are "best effort" delivery and are high on the drop priority. Also is valid to think is there are any firewalls that could be blocking traffic?
If a resolution is not found for an N/A attribution by the above rules, try to get more details; if Change Detection is enabled (as described above) and the device has logging set to the NA server, then increase the syslog logging option levels.
The following logging options set to TRACE in Admin – Troubleshooting are necessary to start in any investigation related to syslog messages.
*feature/changedetection
*external/syslog
*external/syslog
Here is the lines we receive in the jboss_wrapper.log file when changed is performed:
You can see how the HPNA identifies the device pattern (SYS-5-CONFIG_l)
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Syslog Message: <189>1497: *Apr 13 06:07:44.973: %SYS-5-CONFIG_I: Configured from console by root on vty0 (172.16.1.242) -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: can use sender device IP?:true -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Got IP: 172.16.1.242 senderIP: 172.16.1.210 rfcCompliant: false user: null -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Lets make one more shot and try if it wasn't relay, who send message from ip: 172.16.1.210 -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 00 ChangeDetectionEJB: Will process change detection for all devices in all realms with the ip 172.16.1.210 -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Starting Syslog User Lookup, user list size=3 -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Input message [<189>1497: *Apr 13 06:07:44.973: %SYS-5-CONFIG_I: Configured from console by root on vty0 (172.16.1.242)] -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Examining pattern [SYS-5-CONFIG.*onfigured from .* by (\S+) on] -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Pattern match = true -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Syslog user match [u:root:u] -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Examining pattern [SYS-5-CONFIG.*Configured from .* by vty.*\(([a-fA-F0-9:\.]+)\)] -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Pattern match = false -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Examining pattern [SYS-5-CONFIG.*Configured from tftp://([a-fA-F0-9:\.]+)] -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Pattern match = false -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Master list created, size=1 -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Matching AAAUserName field found [u:root:u] -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Saving Syslog User [u:root:u] -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: DeviceIP: null DeviceID: 1851 AgentIP: null AgentType: syslog User: u:root:u UserID: null Comment: <189>1497: *Apr 13 06:07:44.973: %SYS-5-CONFIG_I: Configured from console by root on vty0 (172.16.1.242) SleepTime: null -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: DeviceIP or DeviceID are not null. -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: userID is 1 -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: deviceID is 1851 -
2015/04/24 15:04:24 INFO [STDOUT] {feature/changedetection} [WorkerThread#0[127.0.0.1:38699]] 10 ChangeDetectionEJB: Found a task, updating it id: 2654941 device id: 1851 -
If you confirmed that the device is sending a syslog message and the above settings are correct but the changes are still being assigned to "N/A", then set the logging options to trace, then reproduce the problem by making a change on a device that you saw the changes as "N/A". When the task is done, you can zip up the files (Admin --> Troubleshooting --> download troubleshooting information and open a support case with the corresponding logs.