Environment
All Novell Linux
products
Situation
A core dump of a process needs to
be captured for troubleshooting purposes.
Resolution
To understand capturing systemd service core files on SLE12, see TID7017137 - How to obtain systemd service core dumps.
The following steps should be taken to prepare for capturing a core dump:
The following steps should be taken to prepare for capturing a core dump:
- Disable the limit for the maximum size of a core dump file
- Configure a fixed location for storing core dumps
- Disable AppArmor
- Enable core dumps for setuid and setgid processes
- Runulimit -c unlimited
- Runinstall -m 1777 -d /var/local/dumps
- Runecho "/var/local/dumps/core.%e.%p"> /proc/sys/kernel/core_pattern
- Runrcapparmor stop
- Runsysctl -w kernel.suid_dumpable=2
- (Re)start problematic
processes.
Additional Information
Terminology
For Unix systems, the term "core dump" generally refers to a dump of the state of an individual process. The term "core dump" is also used in the context of NetWare, but in that context it refers to a dump of the state of a complete system. The equivalent of a NetWare core dump for Unix systems is typically called a "system crash dump" or "kernel crash dump".
Core dump analysis
Core dumps can be a valuable source of information in certain troubleshooting scenarios. Detailed analysis of core dumps requires knowledge of the programming language(s) the problematic application (and the libraries and plugins it depends on) is written in, as well as a development environment that matches the environment in which the core dump was taken in terms of architecture as well as application, library and plugin versions. The supportconfig tool (https://www.novell.com/communities/node/2332) provides a convenient way of capturing this environment information.
As core dump analysis requires specialist skills and can be a highly complex process, it should generally only be attempted when more straightforward troubleshooting steps (in particular, applying relevant patches) have not brought relief.
When do core dumps occur?
A process will crash if it receives a signal (which it is not ignoring, or which it cannot ignore) and for which it has not set up a signal handler routine. For example, most processes generally do not set up a signal handler to handle SIGSEGV and will crash when the kernel sends them this signal to indicate that they have tried to perform an incorrect memory access. When this situation occurs and certain environmental settings are right, the kernel will write a core dump file for the process.
Preparation step: Disable the limit for the maximum size of a core dump file
On Unix systems, limits can be set for various resources available to processes. These limits can be displayed through"ulimit -a". One limitable value is the size of core dumps. To ensure a complete core dump can be written, run "ulimit -c unlimited" prior to starting the application of which the core dump needs to be captured. Changing the limit does not affect running processes and only applies to processes started the current shell.
To change limits permanently, install the ulimit package, configure the limits in the/etc/sysconfig/ulimit configuration file and reboot the system for the changed limits to take effect.
Preparation step: Configure a fixed location for storing core dumps
By default, the kernel writes core dump files in the current working directory of the crashing process (if file system permissions and ACLs allow). This can make it difficult to locate core dump files. To store core dumps in a fixed directory, first create a suitable directory, say /var/local/dumps:
To additionally make this configuration change persistent over reboots, add a line
For additional details on the format of the core_pattern parameter, refer to /usr/src/linux/Documentation/sysctl/kernel.txt in the kernel sources.
Preparation step: Disable AppArmor
AppArmor application security is based on learning the normal/good behaviour of an application and then preventing the application from performing operations that do not fit the learned behaviour. As writing a core dump is typically not part of an application's normal/good behaviour, AppArmor is likely to prevent a core file from being written.
Refer to the AppArmor documentation for details on how to disable AppArmor selectively, or execute
Preparation step: Enable core dumps for setuid and setgid processes
Unix systems provide a facility, the setuid and setgid bits, to execute processes under a different user id or group id than that of the user/group starting the process. Such a process may be dealing with data which should not be directly accessible to the invoking user/group and if a core dump were written for such a process, that data could be leaked. For this reason, the kernel does not write core dumps for setuid/setgid processes by default. Similarly, by default the kernel will not write core dumps for processes running as the root user when doing so could leak information. This default behaviour can be overridden through the kernel.suid_dumpable sysctl:
To additionally make this configuration change persistent over reboots, add a line
Refer to /usr/src/linux/Documentation/sysctl/fs.txt in the kernel sources for details on this sysctl.
Manually triggering a core dump
In some troubleshooting scenarios, it may be necessary to trigger the generation of a core dump manually. This can be done by sending a signal that generates a coredump to the process, for example SIGABRT. To send SIGABRT to process 1234, run
System crash dumps
System crash dumps are beyond the scope of this document; refer to KB 3374462 - Configure kernel core dump capture for information on them. Also be aware that the directory used to store application core dump have to be different from the directory to store system crash dumps. See KB 7010096.
For Unix systems, the term "core dump" generally refers to a dump of the state of an individual process. The term "core dump" is also used in the context of NetWare, but in that context it refers to a dump of the state of a complete system. The equivalent of a NetWare core dump for Unix systems is typically called a "system crash dump" or "kernel crash dump".
Core dump analysis
Core dumps can be a valuable source of information in certain troubleshooting scenarios. Detailed analysis of core dumps requires knowledge of the programming language(s) the problematic application (and the libraries and plugins it depends on) is written in, as well as a development environment that matches the environment in which the core dump was taken in terms of architecture as well as application, library and plugin versions. The supportconfig tool (https://www.novell.com/communities/node/2332) provides a convenient way of capturing this environment information.
As core dump analysis requires specialist skills and can be a highly complex process, it should generally only be attempted when more straightforward troubleshooting steps (in particular, applying relevant patches) have not brought relief.
When do core dumps occur?
A process will crash if it receives a signal (which it is not ignoring, or which it cannot ignore) and for which it has not set up a signal handler routine. For example, most processes generally do not set up a signal handler to handle SIGSEGV and will crash when the kernel sends them this signal to indicate that they have tried to perform an incorrect memory access. When this situation occurs and certain environmental settings are right, the kernel will write a core dump file for the process.
Preparation step: Disable the limit for the maximum size of a core dump file
On Unix systems, limits can be set for various resources available to processes. These limits can be displayed through"ulimit -a". One limitable value is the size of core dumps. To ensure a complete core dump can be written, run "ulimit -c unlimited" prior to starting the application of which the core dump needs to be captured. Changing the limit does not affect running processes and only applies to processes started the current shell.
To change limits permanently, install the ulimit package, configure the limits in the/etc/sysconfig/ulimit configuration file and reboot the system for the changed limits to take effect.
Preparation step: Configure a fixed location for storing core dumps
By default, the kernel writes core dump files in the current working directory of the crashing process (if file system permissions and ACLs allow). This can make it difficult to locate core dump files. To store core dumps in a fixed directory, first create a suitable directory, say /var/local/dumps:
install -m 1777 -d
/var/local/dumps
(1777 corresponds to world
writable, deletable only by owner), next instruct the kernel to
store dumps there:echo"/var/local/dumps/core.%e.%p">
/proc/sys/kernel/core_pattern
The kernel will expand %e to the name of the
crashing process and %p to the PID.To additionally make this configuration change persistent over reboots, add a line
kernel.core_pattern=/var/local/dumps/core.%e.%p
to the /etc/sysctl.conf
configuration file.For additional details on the format of the core_pattern parameter, refer to /usr/src/linux/Documentation/sysctl/kernel.txt in the kernel sources.
Preparation step: Disable AppArmor
AppArmor application security is based on learning the normal/good behaviour of an application and then preventing the application from performing operations that do not fit the learned behaviour. As writing a core dump is typically not part of an application's normal/good behaviour, AppArmor is likely to prevent a core file from being written.
Refer to the AppArmor documentation for details on how to disable AppArmor selectively, or execute
rcapparmor stop
to disable AppArmor completely (at
the cost of loosing AppArmor's protection).Preparation step: Enable core dumps for setuid and setgid processes
Unix systems provide a facility, the setuid and setgid bits, to execute processes under a different user id or group id than that of the user/group starting the process. Such a process may be dealing with data which should not be directly accessible to the invoking user/group and if a core dump were written for such a process, that data could be leaked. For this reason, the kernel does not write core dumps for setuid/setgid processes by default. Similarly, by default the kernel will not write core dumps for processes running as the root user when doing so could leak information. This default behaviour can be overridden through the kernel.suid_dumpable sysctl:
sysctl -w
kernel.suid_dumpable=2
To additionally make this configuration change persistent over reboots, add a line
kernel.suid_dumpable=2
to the /etc/sysctl.conf
configuration file.Refer to /usr/src/linux/Documentation/sysctl/fs.txt in the kernel sources for details on this sysctl.
Manually triggering a core dump
In some troubleshooting scenarios, it may be necessary to trigger the generation of a core dump manually. This can be done by sending a signal that generates a coredump to the process, for example SIGABRT. To send SIGABRT to process 1234, run
kill -ABRT 1234
or to send SIGABRT to all running firefox-bin processes,
runkill -ABRT $(pidof
firefox-bin)
System crash dumps
System crash dumps are beyond the scope of this document; refer to KB 3374462 - Configure kernel core dump capture for information on them. Also be aware that the directory used to store application core dump have to be different from the directory to store system crash dumps. See KB 7010096.
Change Log
20090312 ajohansson - Add info on how to make the suid_dumpable
setting persistent across reboots
2016-01-08 Jason Record - Added SLES12 systemd service link
2016-01-08 Jason Record - Added SLES12 systemd service link
