Environment
Novell Access Management 3 Netware Access Gateway
Novell Access Management 3 Support Pack 1 applied
Situation
The Linux Access Gateway (LAG) proxy would continue to restart
automatically. The only indication that this was happening was the
alerts that were sent to the administrator when the services
started and stopped. All users reported that the system was running
fine.
Grabbing the coredump, the backtrace of the issue as seen with gdb showed the following:
Thread 1 (process 18081):
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x40244f9b in __write_nocancel () from /lib/tls/libc.so.6
#2 0x401f6576 in _IO_new_file_write () from /lib/tls/libc.so.6
#3 0x401f6275 in new_do_write () from /lib/tls/libc.so.6
#4 0x401f652f in _IO_new_do_write () from /lib/tls/libc.so.6
#5 0x401f6fb2 in _IO_new_file_sync () from /lib/tls/libc.so.6
#6 0x401ec0b7 in fflush () from /lib/tls/libc.so.6
#7 0x454fe8bc in SendOriginServerHTTPRequest (clientRequest=0x80c46004, methodName=0xa1717020 "GET", methodLength=3) at /home/src/vishnu/53.x/legacy/s_proxy/client.c:2218
#8 0x454fea05 in SendHTTPClientRequest (clientRequest=0x80c46004, status=0) at /home/src/vishnu/53.x/legacy/s_proxy/client.c:657
#9 0x4555876f in RES_ConnectToHost (requestBlock=0x80c46004) at /home/src/vishnu/53.x/legacy/s_proxy/ipres.c:768
It appears to be an issue logging entries to the log files used by the LAG proxy.We do have the log rollover for some of the LAG files - along the lines of what we have for ics_dyn.log. Looking at the /etc/logrotate.d/syslog file one can see entries for :
/var/log/ics_dyn.log
/var/log/laghttpheaders
/var/log/lagsoapmessages
Grabbing the coredump, the backtrace of the issue as seen with gdb showed the following:
Thread 1 (process 18081):
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x40244f9b in __write_nocancel () from /lib/tls/libc.so.6
#2 0x401f6576 in _IO_new_file_write () from /lib/tls/libc.so.6
#3 0x401f6275 in new_do_write () from /lib/tls/libc.so.6
#4 0x401f652f in _IO_new_do_write () from /lib/tls/libc.so.6
#5 0x401f6fb2 in _IO_new_file_sync () from /lib/tls/libc.so.6
#6 0x401ec0b7 in fflush () from /lib/tls/libc.so.6
#7 0x454fe8bc in SendOriginServerHTTPRequest (clientRequest=0x80c46004, methodName=0xa1717020 "GET", methodLength=3) at /home/src/vishnu/53.x/legacy/s_proxy/client.c:2218
#8 0x454fea05 in SendHTTPClientRequest (clientRequest=0x80c46004, status=0) at /home/src/vishnu/53.x/legacy/s_proxy/client.c:657
#9 0x4555876f in RES_ConnectToHost (requestBlock=0x80c46004) at /home/src/vishnu/53.x/legacy/s_proxy/ipres.c:768
It appears to be an issue logging entries to the log files used by the LAG proxy.We do have the log rollover for some of the LAG files - along the lines of what we have for ics_dyn.log. Looking at the /etc/logrotate.d/syslog file one can see entries for :
/var/log/ics_dyn.log
/var/log/laghttpheaders
/var/log/lagsoapmessages
Resolution
Apply the LAG update from nam3sp1ir2.tar.gz (AccessManager 3
Support pack 1 Interim Release 2).
We have included the log file roll over option for all of ics_dyn.log, lagsoapmessages and laghttpheaders. Initially, the logrotate utility was as part of the system's daily cron jobs, but later it was moved to hourly cron jobs so that it can verify the log files size every 1 hr and rotate them.
In addition to that, there is a script /chroot/lag/opt/novell/support/tools/dscheck.sh which can check the disk space usage and can alert the system if the disk usage crosses that threshold (90% by default). The administrator can add the script to a cron job if needed.
We have included the log file roll over option for all of ics_dyn.log, lagsoapmessages and laghttpheaders. Initially, the logrotate utility was as part of the system's daily cron jobs, but later it was moved to hourly cron jobs so that it can verify the log files size every 1 hr and rotate them.
In addition to that, there is a script /chroot/lag/opt/novell/support/tools/dscheck.sh which can check the disk space usage and can alert the system if the disk usage crosses that threshold (90% by default). The administrator can add the script to a cron job if needed.