Linux Access Gateway Proxy crashing downloading very large files (>500 MB) from Web server

  • 7007494
  • 06-Jan-2011
  • 29-Apr-2013

Environment

Novell Access Manager 3.1 Support Pack 2 applied
Novell Access Manager 3.1 Linux Access Gateway
Linux Access Gateway (LAG) setup on a VMWare ESX 4 platform
SLES9 based LAG running 2GB of RAM and 50GB hard disk

Situation

Basic Access Manager functionality working fine - users can access LAG protected resources after having authenticated at the Identity (IDP) server.

When a user tries to download very large files at peak times of the day through the LAG, the LAG sometimes restarts. The ics_dyn log file indicates that the proxy is running low on memory ('scache' warnings prior to restart). The issue is reproducible by stress testing the system with multiple users downloading large files (>500MB) at the same time.

Resolution

Increase the memory allocated for the LAG virtual machine to 4GB. AFter doing this, stress tests could no longer duplicate the issue.

Additional Information

gdb backtrace for the crash shows:

#0 nkEnterDebugger () at nksutil.c:693
#1 0x4f01fab6 in InternalConsistencyError (errorString=0x4f1120a4"DataBuffer::allocate cannot return a buffer") at
/home/schoi/build_sles9/LinuxAccessGateway~AccessManager3.1_SP2/LinuxAccessGateway/legacy/s_proxy/start.c:2827
#2 0x4f041f56 in DataBuffer::allocate (numberOfDataBuffers=1,setAvailToEnd=false) at /home/schoi/build_sles9/LinuxAccessGateway~AccessManager3.1_SP2/LinuxAccessGateway/legacy/s_proxy/s_cos/cosBuf.cpp:117
#3 0x4f042924 in CacheObject::appendNewDataFragmentAndBuffer (this=0xaaea8ab0,dataStream=4294967295,f=@0xaae525c0) at /home/schoi/build_sles9LinuxAccessGateway~AccessManager3.1_SP2/LinuxAccessGateway/legacy/s_proxy/s_cos/cosCO.cpp:2466
#4 0x4f0443ec in CacheObject::appendData (this=0xaaea8ab0, dataStream=CO_ENTITY_DATA_STREAM, RequestedWriteAmount=3544, writeData=0xaf54a960 "�\214\224Z�[)<8U%�030Li��\214t~\032\002\0220;\036", endOfDataStream=false, reserveSpace=false) at /home/schoi/build_sles9/LinuxAccessGateway~AccessManager3.1_SP2/LinuxAccessGateway/legacy/s_proxy/s_cos/cosCO.cpp:2957
#5 0x4f088727 in CachedWebItem::receiveData (this=0xaa617714, pBufSeg=0xaf56fbb0) at cosCO.hpp:1491
#6 0x4f08d648 in CachedWebItem::writeData (this=0xaa617714, pBufSeg=0xaf56fbb0) at /home/schoi/build_sles9/LinuxAccessGateway~AccessManager3.1_SP2/LinuxAccessGateway/legacy/s_proxy/s_cos/webcache.cpp:4932
#7 0x4f819230 in HttpFillResponseDataStreamManager::finishProcessData (context=0xad4b1724, event=0xae41da44) at
:
: