New authentications are being denied due to low system memory.

Periodically, new connections to proxy services were failing with the error.

Restarting the Access Gateway would temporarily resolve the problem

The AGs each had 8GB of memory. As the error was regarding authentications rather than proxy connections, it was suspected that this was java memory issue used by the embedded service provider.

Default Java memory is 1GB and had already been increased to 2GB but the problem persisted.

Enabling Statistics Logging on the IDP cluster will also enable statistics in the catalina log for the eSP on the Access Gateway. A recommended logging interval value for a production system would be 600 (seconds).

These statistics are then printed to the log every ten minutes and looks like

We see this is a busy system.

Total memory is 2.11 exp 9 which equates to the -Xmx2048m  (max Heap Memory) value configured in server.xml.

In the above statistic we only have 15% of memory available.

The error mentioned above states "Threshold 10 Current: 6.109713"

The Threshold of 10 is defined in server.xml

and the eSP will limit further authentication when free memory goes below 10 % and throttling will begin as we see in this example.

   Free Memory: 1.36996032E8
   Total Memory: 2.11759923E9
    Percent Free: 6.4694033
    System Throttle:
   Due to Low Memory: (Request Blocked)

The solution here is to allocate more than 2GB memory to the java heap. As the server has 8GB RAM and is a dedicated AG appliance, 4GB is sufficient for the operating system and we can allocate 4GB to the java heap.

As we know from our baseline that we will use at least 2GB for java, we should allocate at least this value at startup (-Xms) to improve performance.

The following line was added to /opt/novell/nam/mag/conf/tomcat.conf to resolve the problem.