Network Node Manager i (NNMi) - Linux: Upgrade to version 9.20 is successful, but install logs show a failure caused by "java.io.IOException: error=7, Argument list too long".

  • KM00442630
  • 31-May-2013
  • 13-Mar-2015

Summary

On Linux systems, when NNMi version 9.10, at a patch level meeting the NNMi version 9.20 upgrade criteria, is upgraded to version 9.20, the operation is successful. However, install logs report a script execution failure at "opt/OV/newconfig/nms/9.20/installScripts/z-nnm-final-install.xml", caused by a "java.io.IOException" which complains about an "Argument list too long" situation.

Error

Install logs show the following messages:

Wednesday, April 17, 2013 7:47:26 PM GMT+00:00 - ERROR: Could not execute
StackTrace:
com.hp.ov.nms.installconfig.core.ConfigException: 
Script failed: /opt/OV/newconfig/nms/9.20/installScripts/z-nnm-final-install.xml
...
at com.hp.ov.nms.installconfig.InstallConfigMain$1.run(InstallConfigMain.java:135)
Caused by: /opt/OV/newconfig/nms/9.20/installScripts/z-nnm-final-install.xml:42: Execute failed: 
java.io.IOException: Cannot run program "chmod" (in directory "/opt/OV/newconfig/nms/9.20/installScripts"): java.io.IOException: error=7, Argument list too long
at org.apache.tools.ant.taskdefs.ExecuteOn.runExec(ExecuteOn.java:488)
...
at com.hp.ov.nms.installconfig.core.AbstractAntStep.runAnt(AbstractAntStep.java:137)
... 7 more
Caused by: java.io.IOException: Cannot run program "chmod" (in directory "/opt/OV/newconfig/nms/9.20/installScripts"): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
...
at org.apache.tools.ant.taskdefs.ExecuteOn.runExec(ExecuteOn.java:476)
... 20 more
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
...
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 30 more

Wednesday, April 17, 2013 7:47:26 PM GMT+00:00 - ERROR: com.hp.ov.nms.installconfig.initializer.Install error:  Step 3 - ComponentInstallScriptStep

Cause

The message that points to the cause of this failure is:-
"Caused by: /opt/OV/newconfig/nms/9.20/installScripts/z-nnm-final-install.xml:42: Execute failed: java.io.IOException: Cannot run program "chmod" (in directory "/opt/OV/newconfig/nms/9.20/installScripts"): java.io.IOException: error=7, Argument list too long".
 
Given below is the block of text from lines 38 to 52 of the file "/opt/OV/newconfig/nms/9.20/installScripts/z-nnm-final-install.xml", with line number 42 highlighted in bold:-
 
<!--  /var/opt/OV/shared/nnm/databases/custompoller -->
        <chmod perm="2775">
            <dirset dir="${com.hp.ov.DataDir}/shared/nnm/databases/custompoller"/>
        </chmod>
       
<chmod perm="664">
            <fileset dir="${com.hp.ov.DataDir}/shared/nnm/databases/custompoller"/>
        </chmod>
        <chgrp file="${com.hp.ov.DataDir}/shared/nnm/databases/custompoller" group="nmsgrp">
            <dirset dir="${com.hp.ov.DataDir}/shared/nnm/databases/custompoller"/>
            <fileset dir="${com.hp.ov.DataDir}/shared/nnm/databases/custompoller"/>
        </chgrp>
        <chown file="${com.hp.ov.DataDir}/shared/nnm/databases/custompoller" owner="nmsproc">                      
            <dirset dir="${com.hp.ov.DataDir}/shared/nnm/databases/custompoller"/>
            <fileset dir="${com.hp.ov.DataDir}/shared/nnm/databases/custompoller"/>
        </chown>
    
A part of the upgrade process involves setting permissions on the directory "/var/opt/OV/shared/nnm/databases/custompoller", but in this case the set-permissions operation failed, because the chown command seemed to have gotten too many values in its arguments list. The reason is that under the "/var/opt/OV/shared/nnm/databases/custompoller" directory are 2 more directories:-
1. final
2. working
These 2 directories are the file dumps for all custom poller collections configured in the software, and at a point in time, they may contain thousands of "*.csv.gz" files.
The upgrade process's script tries to set permissions to this directory, and its sub-directories/files, using the chmod command.
However, the operation fails because chmod has kernel specific limits on how many files it can change permissions for, at one go.

Fix

In a situation when such messages are encountered in the install-upgrade logs, simply set the user and group permissions manually for the custompoller directory:-
#chown -R nmsproc /var/opt/OV/shared/nnm/databases/custompoller
#chgrp -R nmsgrp /var/opt/OV/shared/nnm/databases/custompoller
 
By default, the directory permissions for the "/var/opt/OV/shared/nnm/databases/custompoller" directory, and its "final" and "working" sub-directories are as follows:-
>>For the custompoller directory:-
# pwd
/var/opt/OV/shared/nnm/databases
# ll
total 40
drwxrwsr-x  3 nmsproc  nmsgrp 4096 Feb  7 08:24 custompoller

>>For the final and working directories:
# pwd
/var/opt/OV/shared/nnm/databases/custompoller/export
# ll
drwxrwxrwx 2 nmsproc nmsgrp 28672 May  8 21:32 final
drwxrwxrwx 2 nmsproc nmsgrp  4096 Feb  7 08:24 working