Environment
Novell Open Enterprise Server 11 (OES 11) Linux
Novell Open Enterprise Server 2 (OES 2) Linux
Novell Open Enterprise Server 1 (OES 1) Linux
SUSE Linux Enterprise Server 10
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 9
Situation
Steps for capturing a core file for the Novell Messenger Messaging Agent or Archive Agent.
Resolution
The steps in this TID were taken from the TID 3054866 - "How to obtain Application Coredumps."
The following steps should be taken to prepare for capturing a core dump:
- Disable the limit for the maximum size of a core dump file
- Configure a fixed location for storing core dumps
- Disable AppArmor
- Enable core dumps for setuid and setgid processes
The quick step guide for this is as follows:
-
Edit /etc/init.d/novell-nmma and add the following line near the top:
ulimit -c unlimited - In a terminal window run
install -m 1777 -d /var/local/dumps - Run
echo "/var/local/dumps/core.%e.%p"> /proc/sys/kernel/core_pattern - Run
rcapparmor stop - Run
sysctl -w kernel.suid_dumpable=2 - (Re)start the Messaging Agent (rcnovell-nmma restart)
To verify that the core is setup correctly try killing the messenger agent's PID using the kill -11 command
kill -11 <pid number>
This should create a new core file in the /var/local/dumps directory.
These steps are described in detail below.
Additional Information
Terminology
For Unix systems, the term "core dump" generally refers to a dump of the state of an individual process. The term "core dump" is also used in the context of NetWare, but in that context it refers to a dump of the state of a complete system. The equivalent of a NetWare core dump for Unix systems is typically called a "system crash dump" or "kernel crash dump".
Core dump analysis
Core dumps can be a valuable source of information in certain troubleshooting scenarios. Detailed analysis of core dumps requires knowledge of the programming language(s) the problematic application (and the libraries and plugins it depends on) is written in, as well as a development environment that matches the environment in which the core dump was taken in terms of architecture as well as application, library and plugin versions. The supportconfig tool (https://www.novell.com/communities/node/2332) provides a convenient way of capturing this environment information.
As core dump analysis requires specialist skills and can be a highly complex process, it should generally only be attempted when more straightforward troubleshooting steps (in particular, applying relevant patches) have not brought relief.
When do core dumps occur?
A process will crash if it receives a signal (which it is not ignoring, or which it cannot ignore) and for which it has not set up a signal handler routine. For example, most processes generally do not set up a signal handler to handle SIGSEGV and will crash when the kernel sends them this signal to indicate that they have tried to perform an incorrect memory access. When this situation occurs and certain environmental settings are right, the kernel will write a core dump file for the process.
Preparation step: Disable the limit for the maximum size of a core dump file
On Unix systems, limits can be set for various resources available to processes. These limits can be displayed through"ulimit -a". One limitable value is the size of core dumps. To ensure a complete core dump can be written, run "ulimit -c unlimited" prior to starting the application of which the core dump needs to be captured. Changing the limit does not affect running processes and only applies to processes started the current shell.
To change limits permanently, install the ulimit package, configure the limits in the/etc/sysconfig/ulimitconfiguration file and reboot the system for the changed limits to take effect.
Preparation step: Configure a fixed location for storing core dumps
By default, the kernel writes core dump files in the current working directory of the crashing process (if file system permissions and ACLs allow). This can make it difficult to locate core dump files. To store core dumps in a fixed directory, first create a suitable directory, say /var/local/dumps:
To additionally make this configuration change persistent over reboots, add a line
For additional details on the format of the core_pattern parameter, refer to/usr/src/linux/Documentation/sysctl/kernel.txt in the kernel sources.
Preparation step: Disable AppArmor
AppArmor application security is based on learning the normal/good behaviour of an application and then preventing the application from performing operations that do not fit the learned behaviour. As writing a core dump is typically not part of an application's normal/good behaviour, AppArmor is likely to prevent a core file from being written.
Refer to the AppArmor documentation for details on how to disable AppArmor selectively, or execute
Preparation step: Enable core dumps for setuid and setgid processes
Unix systems provide a facility, the setuid and setgid bits, to execute processes under a different user id or group id than that of the user/group starting the process. Such a process may be dealing with data which should not be directly accessible to the invoking user/group and if a core dump were written for such a process, that data could be leaked. For this reason, the kernel does not write core dumps for setuid/setgid processes by default. Similarly, by default the kernel will not write core dumps for processes running as the root user when doing so could leak information. This default behaviour can be overridden through the kernel.suid_dumpable sysctl:
To additionally make this configuration change persistent over reboots, add a line
Refer to /usr/src/linux/Documentation/sysctl/fs.txt in the kernel sources for details on this sysctl.
Manually triggering a core dump
In some troubleshooting scenarios, it may be necessary to trigger the generation of a core dump manually. This can be done by sending a signal that generates a coredump to the process, for example SIGABRT. To send SIGABRT to process 1234, run
System crash dumps
System crash dumps are beyond the scope of this document; refer to TID 3374462 - Configure kernel core dump capture for information on them. Also be aware that the directory used to store application core dump have to be different from the directory to store system crash dumps. See KB 7010096.
For Unix systems, the term "core dump" generally refers to a dump of the state of an individual process. The term "core dump" is also used in the context of NetWare, but in that context it refers to a dump of the state of a complete system. The equivalent of a NetWare core dump for Unix systems is typically called a "system crash dump" or "kernel crash dump".
Core dump analysis
Core dumps can be a valuable source of information in certain troubleshooting scenarios. Detailed analysis of core dumps requires knowledge of the programming language(s) the problematic application (and the libraries and plugins it depends on) is written in, as well as a development environment that matches the environment in which the core dump was taken in terms of architecture as well as application, library and plugin versions. The supportconfig tool (https://www.novell.com/communities/node/2332) provides a convenient way of capturing this environment information.
As core dump analysis requires specialist skills and can be a highly complex process, it should generally only be attempted when more straightforward troubleshooting steps (in particular, applying relevant patches) have not brought relief.
When do core dumps occur?
A process will crash if it receives a signal (which it is not ignoring, or which it cannot ignore) and for which it has not set up a signal handler routine. For example, most processes generally do not set up a signal handler to handle SIGSEGV and will crash when the kernel sends them this signal to indicate that they have tried to perform an incorrect memory access. When this situation occurs and certain environmental settings are right, the kernel will write a core dump file for the process.
Preparation step: Disable the limit for the maximum size of a core dump file
On Unix systems, limits can be set for various resources available to processes. These limits can be displayed through"ulimit -a". One limitable value is the size of core dumps. To ensure a complete core dump can be written, run "ulimit -c unlimited" prior to starting the application of which the core dump needs to be captured. Changing the limit does not affect running processes and only applies to processes started the current shell.
To change limits permanently, install the ulimit package, configure the limits in the/etc/sysconfig/ulimitconfiguration file and reboot the system for the changed limits to take effect.
Preparation step: Configure a fixed location for storing core dumps
By default, the kernel writes core dump files in the current working directory of the crashing process (if file system permissions and ACLs allow). This can make it difficult to locate core dump files. To store core dumps in a fixed directory, first create a suitable directory, say /var/local/dumps:
install -m 1777 -d /var/local/dumps
(1777 corresponds to world writable, deletable only by owner), next instruct the kernel to store dumps there:echo"/var/local/dumps/core.%e.%p"> /proc/sys/kernel/core_pattern
The kernel will expand %e to the name of the crashing process and %p to the PID.To additionally make this configuration change persistent over reboots, add a line
kernel.core_pattern=/var/local/dumps/core.%e.%p
to the /etc/sysctl.conf configuration file.For additional details on the format of the core_pattern parameter, refer to/usr/src/linux/Documentation/sysctl/kernel.txt in the kernel sources.
Preparation step: Disable AppArmor
AppArmor application security is based on learning the normal/good behaviour of an application and then preventing the application from performing operations that do not fit the learned behaviour. As writing a core dump is typically not part of an application's normal/good behaviour, AppArmor is likely to prevent a core file from being written.
Refer to the AppArmor documentation for details on how to disable AppArmor selectively, or execute
rcapparmor stop
to disable AppArmor completely (at the cost of loosing AppArmor's protection).Preparation step: Enable core dumps for setuid and setgid processes
Unix systems provide a facility, the setuid and setgid bits, to execute processes under a different user id or group id than that of the user/group starting the process. Such a process may be dealing with data which should not be directly accessible to the invoking user/group and if a core dump were written for such a process, that data could be leaked. For this reason, the kernel does not write core dumps for setuid/setgid processes by default. Similarly, by default the kernel will not write core dumps for processes running as the root user when doing so could leak information. This default behaviour can be overridden through the kernel.suid_dumpable sysctl:
sysctl -w kernel.suid_dumpable=2
To additionally make this configuration change persistent over reboots, add a line
kernel.suid_dumpable=2
to the /etc/sysctl.conf configuration file.Refer to /usr/src/linux/Documentation/sysctl/fs.txt in the kernel sources for details on this sysctl.
Manually triggering a core dump
In some troubleshooting scenarios, it may be necessary to trigger the generation of a core dump manually. This can be done by sending a signal that generates a coredump to the process, for example SIGABRT. To send SIGABRT to process 1234, run
kill -ABRT 1234
or to send SIGABRT to all running firefox-bin processes, runkill -ABRT $(pidof firefox-bin)
System crash dumps
System crash dumps are beyond the scope of this document; refer to TID 3374462 - Configure kernel core dump capture for information on them. Also be aware that the directory used to store application core dump have to be different from the directory to store system crash dumps. See KB 7010096.