How do you troubleshoot lengine cores on SLES, Red Hat and Solaris?

Lengine is coring on Unix or Linux.

How do you troubleshoot lengine cores?

Lengine may core if it is misconfigured, or if there is some other module or component that lengine depends on that is crashing. Some things to look at:

* If you have an mdb.conf file, make sure that the syntax used and the IP address used is correct. You may want to rename or remove the mdb.conf file and see if the problem persists. By default, an mdb.conf file is not used or created.

* Look at your memory usage and parameters. Please review KB 3581698 titled "What are the values memory values for the Secure Logging Server?" and make sure the values you have entered are correct.

* You have a misconfigured channel. If you are going to a database, perhaps you can disable and/or delete the channel and make the file channel the default channel. If one of your channels is misconfigured, then you should go in and configure it properly.

* Download and install the latest version of Novell Audit. Many times there are issues that are resolved in the latest version of Novell Audit. If you are running Novell Nsure Audit 1.0.x, make sure that you are on the latest 1.0.3 code available at https://support.novell.com/filefinder . If you search for "naudit", you will see a list of all the Novell Audit fixes. Also, if you are running 1.0.3 you may want to consider moving to 2.0.2. If you are running Novell Audit 2.0, or 2.0.1, you may want to consider moving to Novell Audit 2.0.2. You may want to go to https://support.novell.com/filefinder and review the fixes available for Novell Audit. Again, search for "naudit" from the above URL and you should see all of the Novell Audit fixes.

If none of the above items help, then it is time to call into Novell Technical Services (NTS) and get some assistance. This TID will outline the things that you will need to do in order to make your time spent with NTS most productive. Much of the information contained in this TID came from http://gaim.sourceforge.net/gdb.php . If you want further information, please review the sourceforge link above.

1.) Obtain a debug version of Novell Audit: Core files on Linux are not like coredump files on NetWare or memory.dmp files on Windows. In production versions of Novell Audit code for SLES, Red Hat, and Solaris, the symbols are stripped out of the code. There is no way to add those symbols in after-the-fact like you can on other platforms (NetWare and Windows). Therefore you must obtain a debug version of Novell Audit from a Novell Technical Support engineer. Novell does not provide a public location for end users to obtain this debug code. You can only obtain it by opening a support incident with NTS. You should obtain this debug code from your Technical Support Engineer and run it.

2.) Make sure that GDB is installed on your server. GDB stands for GNU DeBugger. Please do the following to see if you have GDB installed on your server:

bash-2.05# pkginfo -i |grep gdb
system SFWgdb gdb - GNU source-level debugger

# rpm -qa |grep gdb
gdb-6.3-16.4

If you do not have GDB installed, please review your server files and obtain the gdb package and install it.

3.) Setting ulimit size. The core file that is generated may be large. What you need to do is make sure you have the ability to save a large core file. Do do that, you need to set the following:

bash-2.05# ulimit -c unlimited

This will allow a core file to be created.

4.) Validate the core file belongs to Novell Audit. Typically the core file will reside in the location where lengine is installed. On Solaris, that location is /opt/NOVLnaudit/ . On SLES that location is /opt/novell/naudit/ . Here is how you find out that the core file belongs to lengine:

bash-2.05# file core
core: ELF 32-bit MSB core file SPARC Version 1, from'lengine'

The above line validates that this core file belongs to lengine. Some of the CPU information may be different depending on the platform and type of CPU you are running. The above core file was created on the Sun SPARC platform.

5.)Running GDB and opening a core file. Now it is time to open your core file. Please do the following:

/opt/sfw/bin/gdb lengine core

SLES 8, 9, 10 or Red Hat:

gdb lengine core

You will see a lot of output on your screen similar to the following:

GNU gdb 5.0
Copyright 2000 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.9"...
(no debugging symbols found)...
Core was generated by `./lengine'.
Program terminated with signal 11, Segmentation Fault.
Reading symbols from /usr/lib/libmdb.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libmdb.so
Reading symbols from /usr/lib/libdl.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libdl.so.1

You may have to hit the key a few times to get to the end. Once your core file has been opened, you should have a prompt that looks something like this:

#0 0xfefcf308 in WNGetInt32 () from /opt/novell/eDirectory/lib/libndssdk.so.1
(gdb)

6.) Obtaining a backtrace. Now that we have the core file opened, we want to obtain a backtrace. At the prompt type either backtrace or bt. GDB should list all of the functions that were called leading up to the crash:

(gdb) backtrace
#0 0xfefcf308 in WNGetInt32 () from /opt/novell/eDirectory/lib/libndssdk.so.1
#1 0xfefced64 in SizeOfReferral ()
from /opt/novell/eDirectory/lib/libndssdk.so.1
#2 0xfefa8b4c in __1cP_ExpandReferral6Fpkcppc_i_ ()
from /opt/novell/eDirectory/lib/libndssdk.so.1
#3 0xfefa8d64 in DCConnectToReferral ()
from /opt/novell/eDirectory/lib/libndssdk.so.1
#4 0xff087114 in MDBDSInit () from /usr/lib/mdb/mdbds.so
#5 0xff381c68 in _end () from /usr/lib/libmdb.so
#6 0xff382bc0 in MDBDumpValueStruct () from /usr/lib/libmdb.so
#7 0xff382e9c in MDBInit () from /usr/lib/libmdb.so
#8 0x34974 in main ()
(gdb)

Get a copy of all this output and send it into your Novell Technical Support Engineer.

7.) Exiting GDB. To exit GDB, simply type "quit" (without quotes) and hit the key.

(gdb) quit
bash-2.05#