Summary
The message "FLOWRECORD Object Pool of size 399999 has been exhausted" is seen on the NNMi user interface under "Configuration" -> "NNM iSPI Performance for Traffic Configuration" -> "Traffic Health" page. The most likely reason is high packet/flow rates, or inadequately sized hardware, or a combination of both. The error can be resolved by changing the flow record pool size, or by adding adequate hardware to the setup.
Error
In an environment where NNMi iSPI Performance for Traffic has been deployed,flushing of records from the Traffic SPI Leaf Components to the Traffic SPI Master Component stops, and consequently, Traffic SPI's report generation also stops.
The message "FLOWRECORD Object Pool of size 399999 has been exhausted" is seen on the NNMi user interface under "Configuration" -> "NNM iSPI Performance for Traffic Configuration" -> "Traffic Health" page (see screenshot).
Cause
The most likely causes for this error are:-
1. High record rates/flow rates being received by the Leaf Collector in question.
2. Insufficient memory on the Traffic SPI Leaf Server to process incoming records.
Fix
The flow record pool and the top N flow record pool are used for Traffic SPI's internal algorothmic processing and reporting. It is generally recommended that the value of top N flow record pool size be greater than the value of flow record pool size by a factor of 2.5.
Generally, the default values are enough to keep the operations running smoothely.
However, in uncommon situations, these pools may run out of space.
(Before proceeding any further, it would be best to read KM1462830, which describes the process of tuning Traffic SPI's memory requirements. While the workaround/fix mentioned below may address the problem on a short-term basis, tuning the setup based on packet rates is the best way to address the problem on a long term basis).
To address this situation, first find out the rate at which the Traffic SPI Leaf Collector in question is receiving records (use the "nnmtrafficflowanalysistool.ovpl" command, please read KM1372507 for more details).
Keeping the record rate in mind, navigate to the Traffic SPI Leaf Server's "conf" directory, and open the file named "nms-traffic-leaf.address.properties".
•UNIX: cd /var/opt/OV/shared/traffic-leaf/conf/
•Windows: Navigate to "%trafficInstallDir%\shared\traffic-leaf\conf\"
The "nms-traffic-leaf.address.properties" file contains 3 parameters which are used to configure the memory usage for data processing in Traffic SPI:-
1.datagram.pool.size
2.flowrecord.pool.size
3.topn.flowrecord.pool.size
Since the error indicates that the flow record pool has been exhausted, we need to change the value for the "flowrecord.pool.size" parameter.
The following formula is used to calculate the "datagram.pool.size" value:-
flowrecord.pool.size = 2.5 x (number of records/second) x 60
For example, if the incoming record rate is 500 records per second, then:-
flowrecord.pool.size = 2.5 x (500) x 60 = 75000
(Consequently, topn.flowrecord.pool.size = 2.5 x (75000) = 187500)
So change the value of the "flowrecord.pool.size" parameter based on the above formula, save the file, and restart the Traffic SPI Leaf Services.
Constantly monitor the Traffic Health page to see if the warning changes its status to "CLOSED".
Generally, the default values are enough to keep the operations running smoothely.
However, in uncommon situations, these pools may run out of space.
(Before proceeding any further, it would be best to read KM1462830, which describes the process of tuning Traffic SPI's memory requirements. While the workaround/fix mentioned below may address the problem on a short-term basis, tuning the setup based on packet rates is the best way to address the problem on a long term basis).
To address this situation, first find out the rate at which the Traffic SPI Leaf Collector in question is receiving records (use the "nnmtrafficflowanalysistool.ovpl" command, please read KM1372507 for more details).
Keeping the record rate in mind, navigate to the Traffic SPI Leaf Server's "conf" directory, and open the file named "nms-traffic-leaf.address.properties".
•UNIX: cd /var/opt/OV/shared/traffic-leaf/conf/
•Windows: Navigate to "%trafficInstallDir%\shared\traffic-leaf\conf\"
The "nms-traffic-leaf.address.properties" file contains 3 parameters which are used to configure the memory usage for data processing in Traffic SPI:-
1.datagram.pool.size
2.flowrecord.pool.size
3.topn.flowrecord.pool.size
Since the error indicates that the flow record pool has been exhausted, we need to change the value for the "flowrecord.pool.size" parameter.
The following formula is used to calculate the "datagram.pool.size" value:-
flowrecord.pool.size = 2.5 x (number of records/second) x 60
For example, if the incoming record rate is 500 records per second, then:-
flowrecord.pool.size = 2.5 x (500) x 60 = 75000
(Consequently, topn.flowrecord.pool.size = 2.5 x (75000) = 187500)
So change the value of the "flowrecord.pool.size" parameter based on the above formula, save the file, and restart the Traffic SPI Leaf Services.
Constantly monitor the Traffic Health page to see if the warning changes its status to "CLOSED".