High I/O Disk Utilization that remains at 100% when running any Traffic Report

  • Document ID:KM00790574
  • Creation Date:14-Mar-2014
  • Modified Date:21-Mar-2014
    • Micro Focus Products:
      network node manager ispi performance for metrics 9.20
      network node manager ispi performance for metrics 9.21

Summary

This document explains how to troubleshoot and identify the possible reasons why at the moment of running any Traffic Report, the disk I/O utilization raises and remains at 100%

Error

 In the dbCheckIndexesReport.txt file the following can be seen:
…
          Line 3999: Attr   NOT_OK Flat style FP   255             f_Raw_InterfaceTrafficMetrics.Destination Host IP Address
          Line 4000: Attr   NOT_OK Flat style FP   255             f_Raw_InterfaceTrafficMetrics.Destination Host Name
          Line 4048: Attr   NOT_OK Flat style FP   255             f_Raw_InterfaceTrafficMetrics.Source Host IP Address
          Line 4049: Attr   NOT_OK Flat style FP   255             f_Raw_InterfaceTrafficMetrics.Source Host Name
…
EXTENSIONPACK: Interface_Traffic
==================================
…
f_Raw_InterfaceTrafficMetrics            Attr columns:      OK(102) NOT_OK(4)
…
EXTENSIONPACK SUMMARY: Interface_Traffic  OK(448) NOT_OK(4)
statusALL.ovpl output shows:
…
Tables/Extension Packs with Non-Compliant Indexes
=========================================================================================
Extension Pack : Interface_Traffic
Table Name                              Non-Compliant-Column-Count  Checked On
f_Raw_InterfaceTrafficMetrics           4                           Thu Feb 20 12:14:27 2014

Cause

As far as the hardware goes, there are situations which some systems never pages to disk from memory (e.g. disks on a P2000 array and set to RAID 1+0). This could point to database issues, not system resource issues. The mentioned messages seem to point to a database issue. Bad SQL statements and bad database indexes may cause high disk I/O.

Fix

1.  Run the dbCheckIndexes.ovpl with the table name only:
dbCheckIndexes.ovpl -r -f -t f_Raw_InterfaceTrafficMetrics 
        
2.  Run the dbCheckIndexes.ovpl with the Extension Pack:
dbCheckIndexes.ovpl -r -f -p Interface_Traffic

3.  Run the dbCheckindexes.ovpl with no options to confirm everything is ok:
dbCheckIndexes.ovpl

4.  Check with statusALL.ovpl that all the EP (Extension Pack) are ok
5.  From the browser clear the cache and cookies and close it.
6.  Access the NPS (Network Performance Server) UI, run a couple of Traffic Reports to confirm they are now running correctly and the I/O Disk Utilization is now ok.

Notes
Check the disk performance with the command: iostat -x -4
The  iostat  command  is used for monitoring system input/output device
       loading by observing the time the devices are  active  in  relation  to
       their average transfer rates. The iostat command generates reports that
       can be used to  change  system  configuration  to  better  balance  the
       input/output load between physical disks.