Environment
NetIQ Access Manager 4.0
Situation
Access Manager 4.0.1 freshly installed on SLES 11 SP3 platform with the Admin Console(AC), Identity Server (IDP) and Access Gateway (AG) each on a spearate server. The normal installation worked fine, but the HF 3 patch afterwards
failed. The HF3 scripts were run from an NFS mount and not locally. The documentation has a disclaimer about remote installs but seems to be related to disk space which was not a concern for us.
Once the patch was installed on the AC and the IDP (we didn't do this on the AG because of the issues we saw), the key AC/IDP services failed to start. Looking into the catalina.out, errors indicated that java couldn't be found. Checking into /opt/novell, we saw that java had a symlink to /opt/novell/jdk1.7.0_25 (older java version), however looking into the bin subfolder showed no java application there.
Feb 18, 2015 5:00:57 PM com.netiq.nam.patching.Util execCmd
FINE: The process /opt/novell/nam/update/HF3/patchjdk.sh apply returned 0.
Feb 18, 2015 5:00:57 PM com.netiq.nam.patching.ProcessDrainer run
FINE: rm: cannot remove `/opt/novell/jdk1.7.0_25/bin/.nfs00000000002ef6790000001a': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/jli/.nfs000000000075b32e00000019': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/server/.nfs00000000012a521b00000013': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs0000000001388e2000000018': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs00000000012a520a00000017': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs00000000012a520b00000016': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs00000000012a521600000015': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs00000000012a521700000014': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/ext/.nfs0000000000d40d2c00000012': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/.nfs00000000012fa02900000011': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/.nfs00000000009d40be00000010': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/.nfs00000000009d40bf0000000f': Device or resource busy
chmod: cannot access `/opt/novell/java/bin/*': No such file or directory
chmod: cannot access `/opt/novell/java/jre/bin/*': No such file or directory
This
amount is recommended to ensure ample space for logging in a production
environment.
This
disk space must be local and not remote.
Once the patch was installed on the AC and the IDP (we didn't do this on the AG because of the issues we saw), the key AC/IDP services failed to start. Looking into the catalina.out, errors indicated that java couldn't be found. Checking into /opt/novell, we saw that java had a symlink to /opt/novell/jdk1.7.0_25 (older java version), however looking into the bin subfolder showed no java application there.
Looking into the patch logs, I see this:
Feb 18, 2015 5:00:57 PM com.netiq.nam.patching.Util execCmd
FINE: The process /opt/novell/nam/update/HF3/patchjdk.sh apply returned 0.
Feb 18, 2015 5:00:57 PM com.netiq.nam.patching.ProcessDrainer run
FINE: rm: cannot remove `/opt/novell/jdk1.7.0_25/bin/.nfs00000000002ef6790000001a': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/jli/.nfs000000000075b32e00000019': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/server/.nfs00000000012a521b00000013': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs0000000001388e2000000018': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs00000000012a520a00000017': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs00000000012a520b00000016': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs00000000012a521600000015': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/amd64/.nfs00000000012a521700000014': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/ext/.nfs0000000000d40d2c00000012': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/.nfs00000000012fa02900000011': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/.nfs00000000009d40be00000010': Device or resource busy
rm: cannot remove `/opt/novell/jdk1.7.0_25/jre/lib/.nfs00000000009d40bf0000000f': Device or resource busy
chmod: cannot access `/opt/novell/java/bin/*': No such file or directory
chmod: cannot access `/opt/novell/java/jre/bin/*': No such file or directory
It
explains why the -25 jdk directory is still there.
Resolution
The HF3 patch comes with a second, updated jdk in opt/novell/jdk1.7.0_72, with all
the appropriate files but in all cases with the x flag missing on the relevant executables. This was on both the AC and the IDP. Setting the x flag on those executables and changing the symlink for java to the ‑72
directory fixed the issues.
Cause
If you look into the /opt/novell/nam/patching/bin/patch file, you will see
that we actually start /opt/novell/java/bin/java to install the patch. However,
during the patch installation, we delete the directory where we started java from,
which means that of course the library files are open. While this works on a
unix file system, and many Linux file systems including SLES mounts, this is really something you should not do. The issue here is that this does not work with a NetApp mount, as netapp will create these temporary files.