Environment
NetIQ Access Manager 3.2
NetIQ Access Manager 3.2 IR1 applied
NetIQ Access Manager 3.2 Access Gateway Appliance configured
NetIQ Access Manager 3.2 IR1 applied
NetIQ Access Manager 3.2 Access Gateway Appliance configured
Situation
Access Manager setup and working well - users can access protected resources behind the Access Gateway Appliance (AGA) without problems after authenticating at the Identity (IDP) server. AFter a few days of working fine, two if six AGA servers stopped responding - users could not even hit public resources on the proxy without browser timeout errors, despite the listener on TCP 443 still being active. An Apache Timeout message is displayed in the Admin Console under Health Status.
After rebooting all 6 AGA servers four came up fine but 2 still showed the apache timeout issue.
After rebooting all 6 AGA servers four came up fine but 2 still showed the apache timeout issue.
Resolution
Free up space on the Apache server, as there was no room for any files to be created. Turns out that logging was enabled at the ESP level where entries were getting written to the /opt/ directory (part of root file system).
Looking at the error_log file showed more details. For example, the following enntries indicated that there was no space left on the dis. The main issue here was the fact that the ESP logs were written to the /opt/ patha nd this will be addressed in 3.2 SP1.
<amLogEntry> 2012-09-14T16:02:54Z ERROR AGM: AM#204654001: AMDEVICEID#ag-63385D794A185F08: ApacheGatewayManager: reconfigure(), EXCEPTION (Error processing configuration parameters - java.io.IOException: No space left on device) applying existing configuration </amLogEntry>
<amLogEntry> 2012-09-14T16:12:54Z ERROR AGM: AM#204654001: AMDEVICEID#ag-63385D794A185F08: ApacheGatewayManager: translateConfiguration(), EXCEPTION (java.io.IOException: No space left on device) Attempting to read and process product configuration file-/opt/novell/nam/mag/webapps/agm/WEB-INF/config/current/config.xml :output directory/opt/novell/nam/mag/webapps/agm/WEB-INF/config/apache2/ </amLogEntry>
<amLogEntry> 2012-09-14T16:12:54Z ERROR AGM: AM#204654001: AMDEVICEID#ag-63385D794A185F08: ApacheGatewayManager: com.novell.nacm.agm.AGMConfigurationException: java.io.IOException: No space left on device
at com.novell.nacm.agm.conf.CustomizedErrorPages.doConfigure(CustomizedErrorPages.java:74)
Looking at the error_log file showed more details. For example, the following enntries indicated that there was no space left on the dis. The main issue here was the fact that the ESP logs were written to the /opt/ patha nd this will be addressed in 3.2 SP1.
<amLogEntry> 2012-09-14T16:02:54Z ERROR AGM: AM#204654001: AMDEVICEID#ag-63385D794A185F08: ApacheGatewayManager: reconfigure(), EXCEPTION (Error processing configuration parameters - java.io.IOException: No space left on device) applying existing configuration </amLogEntry>
<amLogEntry> 2012-09-14T16:12:54Z ERROR AGM: AM#204654001: AMDEVICEID#ag-63385D794A185F08: ApacheGatewayManager: translateConfiguration(), EXCEPTION (java.io.IOException: No space left on device) Attempting to read and process product configuration file-/opt/novell/nam/mag/webapps/agm/WEB-INF/config/current/config.xml :output directory/opt/novell/nam/mag/webapps/agm/WEB-INF/config/apache2/ </amLogEntry>
<amLogEntry> 2012-09-14T16:12:54Z ERROR AGM: AM#204654001: AMDEVICEID#ag-63385D794A185F08: ApacheGatewayManager: com.novell.nacm.agm.AGMConfigurationException: java.io.IOException: No space left on device
at com.novell.nacm.agm.conf.CustomizedErrorPages.doConfigure(CustomizedErrorPages.java:74)