core.<number> file found in filesystem

  • 7003302
  • 18-May-2009
  • 29-Apr-2013

Environment

Novell Open Enterprise Server 1 (OES 1) Linux
Novell Open Enterprise Server 2 (OES 2) Linux
SUSE Linux

Situation

When browsing the filesystem a file with the name of core.x or core.<number> is found.  This file may vary in size from very small (kilobytes) to very large (gigabytes or larger).  The file may be found repeatedly over time in the same location.

Resolution

In Linux a 'core' file is written when a program stops unexpectedly or when it is instructed to stop and write a core when sent SIGNAL 11.  The latter method can be done with the 'kill' command while the former can happen anytime the program does something unexpected and unhandled.  Corrupted memory, an attempt to allocate too much memory, a divide-by-zero exception, or many other issues can cause the core to be written to the applications working directory.

Server applications like eDirectory, OpenLDAP, or GroupWise will typically write to a specific directory each time as defined by the application's working directory.  This directory is often found via the 'cwd' symlink located in the proc pseudo-filesystem.  To find this for a process named 'ndsd' (eDirectory) the following could be done:

ls -l /proc/`pgrep ndsd`/cwd

If you have rights to view where this symlink is pointed you may find it is pointed to a location like:

/var/opt/novell/eDirectory/treename/servername/data/dib

If eDirectory were to crash for some reason a core file would likely be written to this location.  The core files are written, per a kernel-configuration, with the PID of the dying process appended to the file so instead of having a file named 'core' the files are named 'core.<pidOfProcess>'.  To find out which process wrote the core after the fact (when the PID of the process is no longer known) use the 'file' command as follows:

file /path/to/core.9628
/path/to/core.9628: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/bin/someProgram'

In this case we can see that /usr/bin/someProgram wrote the file and that the application is a 64-bit application.

As a note of caution if core files build up they can, and will, eventually fill the filesystem.  Most applications do not core regularly so this is not usually an issue but since the PID by default does change on every invocation of a process it is likely that they will not overwrite one another and will instead continue to take space until that is no longer possible.  If an application is regularly coring there is either a misconfiguration or a bug and that should be reported to the creator of the product after appropriate troubleshooting steps have been taken.

For those coming from a NetWare background a coredump could be written when a process would abnormally end (ABEND).  A coredump from NetWare and a core from a Linux process are similar in that both are used to troubleshoot why a process crashed.  Because of differences in memory management the files are notexactly the same in their structure.  A coredump from NetWare usually means all of the kernel memory is included because NetWare runs everything at ring-0 (zero) where Linux uses user mode applications extensively, even for root-owned processes.  A core file in Linux can, therefore, be very small for a small application.