Environment
iManager 3.2.5
eDirectory 9.2.5
eDirectory 9.2.5
Situation
When the Tomcat or Java components of iManager crash often an application core file is required by support. This not only shows the current stack of the application but also shows all the threads, their state and what function they are in. Below are the steps to setting up for an iManager application core file.
Resolution
A. Prepare for a coredump
Modify the /etc/systemd/coredump.conf and increase the ProcessSizeMax and ExternalSizeMax to larger than the potential size of the process being cored.
EXAMPLE:
[Coredump]
#Storage=external
#Compress=yes
ProcessSizeMax=10G
ExternalSizeMax=20G
#JournalSizeMax=767M
#MaxUse=
#KeepFree=
2. Verify cores are enabled. Typing ulimit -c in a shell should return "unlimited"
If not, for init.d, https://support.microfocus.com/kb/doc.php?id=7023055.
If it is systemd refer to https://support.microfocus.com/kb/doc.php?id=3078409
Example: For systemd verify the /etc/systemd/system.conf file contains the following lines:
DumpCore=yes
DumpCore=yes
DefaultLimitCORE=infinity
These are normally on by default post SLES 12 SP3 and SLES 15 SP1.
The first TID shows how to quickly test whether you are now receiving coredumps.
3. Set the malloc variable in Tomcat so the stack will be clean.
Modify /etc/opt/novell/tomcat9/tomcat9.conf and add at the beginning:
MALLOC_CHECK_=3
export MALLOC_CHECK
Stop then start Tomcat. Do not restart.
Cores will usually go to /var/lib/systemd/coredump. If they cannot be found type: coredumpctl info and look for the path in the Storage field.
Optional Test:
Once this is set up one can test this by manually generating a core file from Java.
- Get the PID: ps -eaf | grep java
- Trigger a segmentation fault on the Java process: kill -s SIGSEGV <PID>
Give it a little time. You can see the status of the core file write using: coredumpctl info
4. After a core file is available decompress it.
Example: unxz corefile.xz
If in LZ4 format use the lz4 command line utility to decompress the file.
B. Open the core file
Load the core file using gdb and dump the core file information:
gdb /opt/netiq/common/jre/bin/java -core [path_to_core]
Modify /etc/opt/novell/tomcat9/tomcat9.conf and add at the beginning:
MALLOC_CHECK_=3
export MALLOC_CHECK
Stop then start Tomcat. Do not restart.
Cores will usually go to /var/lib/systemd/coredump. If they cannot be found type: coredumpctl info and look for the path in the Storage field.
Optional Test:
Once this is set up one can test this by manually generating a core file from Java.
- Get the PID: ps -eaf | grep java
- Trigger a segmentation fault on the Java process: kill -s SIGSEGV <PID>
Give it a little time. You can see the status of the core file write using: coredumpctl info
4. After a core file is available decompress it.
Example: unxz corefile.xz
If in LZ4 format use the lz4 command line utility to decompress the file.
B. Open the core file
Load the core file using gdb and dump the core file information:
gdb /opt/netiq/common/jre/bin/java -core [path_to_core]
set logging on
bt
info threads
thread apply all where
set logging off
quit
y
When an actual crash occurs the logged output of gdb.txt can be found in the current directory and sent to Support.
bt
info threads
thread apply all where
set logging off
quit
y
When an actual crash occurs the logged output of gdb.txt can be found in the current directory and sent to Support.