IDP server fails to load successfully after restart

Access Manager setup and working fine. Administrator tried to upload some new custom pages and a custom auth class and restarted novell-idp. This procedure was done on both nodes in the cluster as it appeared that tomcat started ok on the first server they applied the updates to. However, after the restart, users could not access the Identity server to login .. turns out that the nidp server app did not actually startup within Tomcat.

 Initial thought was that the custom class had broken something but removing it completely made no difference.

 Looking through the catalina log files, the startup sequence shows something like:

 INFO: Deploying web application directory /opt/novell/nam/idp/webapps/nidp

Feb 23, 2015 4:31:45 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deployment of web application directory /opt/novell/nam/idp/webapps/nidp has finished in 56 ms

 

Note; how it claims that nidp loaded in 56 ms when it normally takes around 10 secs at this customer (needs to contact Admin Console config store, read all the info, initialise the classes, etc).

 We would also expect to see a message like "WARNING: [SetPropertiesRule]{Context/Manager} Setting property 'saveOnRestart' to 'false' did not find a matching property." which is read from /META-INF/context.xml - but that is missing.

 

So we were suspicious of the symbolic link. We moved the symbolic link from /opt/novell/nam/idp/webapps/nidp that pointed to /opt/novell/nids/lib/webapp/. We then created a real directory called nidp instead of the symbolic link and copied the real nidp contents to that directory. It then loaded perfectly.

Set the correct permissions in webapps directory. It turns out that /opt/novell/nids/lib/webapp had the wrong permissions for some reason. Quite possibly the customer changed the permissions by mistake. Anyway, that directory was missing the "x" right i.e. permissions on the directory where 644 instead of 755.