Environment
- Access Manager Gateway 4.5 on Linux
- Access Manager Gateway 4.5 Appliance
Situation
- Creating core dumps for troubleshooting Access Gateway Proxy issues (Apache) is not working as documented
Resolution
- run the Access Gateway in debug mode: "/etc/init.d/novell-apache2 start debug"
- check all running HTTP processes: ps aux | grep httpd
-------------------------------------------------------------------
root 16106 0.0 0.0 9292 1640 pts/1 S+ 11:32 0:00 grep --color=auto httpd
root 17765 0.0 0.4 256760 19484 ? Ssl Sep20 0:17 /opt/novell/apache2/sbin/httpd -D allowServerDebug
wwwrun 18472 0.0 0.4 1245220 20036 ? Sl Sep20 1:08 /opt/novell/apache2/sbin/httpd -D allowServerDebug
wwwrun 18499 0.0 0.4 3682372 19336 ? Sl Sep20 0:05 /opt/novell/apache2/sbin/httpd -D allowServerDebug
-------------------------------------------------------------------- - run on one of the HTTP processes a kill with "SIGSEGV" signal: kill -11 17765 in order to force a core dump
- check the folder "var/lib/systemd/coredump" for new core dumps (controlled by systemd
--------------------------------------------------------------------
total 524 -rw-r----- 1 root root 525460 Sep 24 11:33 core.httpd.0.d7ea5b289bcd45399c14d3082b75f207.17765.1569317589000000.xz
-------------------------------------------------------------------- - Reviewing a coredump:
- install the symbol files:"rpm -Uvh novell-nacm-apache-debuginfo-2.4.37-450105.x86_64.rpm" stored for example (NAM45 install package) at: "novell-access-gateway-4.5.0.1-5/mag/rpms"
- with the Access Gateway Appliance add a SLES12 DVD as repro and install gdb (not provided by the NAM channel)
- Changing the documentation on how to run coredumps with systemd based Linux OS and adding gdb to the update channel as been addressed to engineering
- Note: Do not forget to turn off running Apache in debug mode after troubleshooting your issues has been finished
Additional Information
- from: "https://www.freedesktop.org/software/systemd/man/systemd-coredump.html"
- Description
systemd-coredump@.service is a system service that can acquire core dumps from the kernel and handle them in various ways. The systemd-coredump executable does the actual work. It is invoked twice: once as the handler by the kernel, and the second time in the systemd-coredump@.service to actually write the data to the journal.
When the kernel invokes systemd-coredump to handle a core dump, it runs in privileged mode, and will connect to the socket created by the systemd-coredump.socket unit, which in turn will spawn an unprivileged systemd-coredump@.service instance to process the core dump. Hence systemd-coredump.socket and systemd-coredump@.service are helper units which do the actual processing of core dumps and are subject to normal service management.
Core dumps can be written to the journal or saved as a file. Once saved they can be retrieved for further processing, for example in gdb(1).
By default, systemd-coredump will log the core dump including a backtrace if possible to the journal and store the core dump itself in an external file in /var/lib/systemd/coredump.
The behavior of a specific program upon reception of a signal is governed by a few factors which are described in detail in core(5). In particular, the core dump will only be processed when the related resource limits are sufficient.
It is also possible to invoke systemd-coredump with --backtrace option. In this case, systemd-coredump expects a journal entry in the journal Journal Export Format on standard input. The entry should contain a MESSAGE= field and any additional metadata fields the caller deems reasonable. systemd-coredump will append additional metadata fields in the same way it does for core dumps received from the kernel. In this mode, no core dump is stored in the journal.