Environment
Situation
Customer reported slow and in-responsive LAG boxes.
The catalina.out file showed the following:
INFO: queuing alert: Severe: Log file(s) have crossed 10000263090 bytes!
After a couple of those warnings with increasing size of the log the following was seen:
SEVERE: All threads (150) are currently busy, waiting. Increase maxThreads (150) or check the servlet status.
When those huge log files where log rotated they caused the box to go into high utilization and consume all of the threads making the box become in-responsive.
Under/var/opt/novell/tomcat5/webapps/nesp/WEB-INF/logs it was seen that huge log files in XML format existed.
The Access Manager configuration showed under the IDP, General, Logging page that only the component Application was set to Info.
Customer had a lot of policies and all the policy evaluation info was logged causing the log file to grow very fast to gigabytes in size.
Resolution
Additional Information
It increases processing load, and on a day-to-day basis, it should be enabled only to log error conditions and system warnings.
If a specific problem occurs, component file logging can be set to info or config to gather the information needed to isolate and repair the detected problem.
When the problem is resolved, component file logging should be reconfigured to log only error conditions and system warnings.