Unmounting Runtime Directory Failed During Shutdown

  • 7023942
  • 13-Jun-2019
  • 02-Jul-2019

Environment

SUSE Linux Enterprise Server 12

Situation

A system was upgraded from SLES 11 SP4 to SLES 12 during shutdown the following error can be seen: 

dracut: Disassembling device-mapper devices , Command failed

this error can be seen on s390x but also on x86 when enabling debugging during shutdown. Detailed output: 

Apr 25 16:32:38 linux-aoie systemd[1]: Unmounting Runtime Directory...
Apr 25 16:32:38 linux-aoie systemd[1]: Unmounted Lock Directory.
Apr 25 16:32:38 linux-aoie umount[2531]: umount: /var/run: target is busy
Apr 25 16:32:38 linux-aoie umount[2531]:         (In some cases useful info abou
Apr 25 16:32:38 linux-aoie umount[2531]:          use the device is found by lso
Apr 25 16:32:38 linux-aoie systemd[1]: var-run.mount: Mount process exited, code
Apr 25 16:32:38 linux-aoie systemd[1]: Failed unmounting Runtime Directory.


Resolution

To fix such problem we need to make sure /var/run is pointing to /run
This can be done via two method: 

     1) Interactive by booting the system from SLES DVD and select "Rescue Mode
then mounting the root disk to /mnt 
then remove /mnt/var/run 
next, create a link /mnt/var/run -> /mnt/run 

Note  in the case /var is separate volume then mount var instead of / and adjust above paths 

     2) Automated by using convertfs dracut module: 
run the following command :
dracut --add convertfs -f 

add the following two kernel command line parameters to the kernel Command Line (use 'yast bootloader'): 
rd.convertfs rootflags=rw  systemctl reboot 

After doing above, the result should look something similar to below :

linux-aoie# ll -ld /var/run*
lrwxrwxrwx  1 root root    6 May 13 09:22 /var/run -> ../run
drwxr-xr-x 22 root root 4096 Apr 30 11:05 /var/run.runmove~

And the var-run.mount unit file should not be running: 

linux-aoie# systemctl status var-run.mount 
● var-run.mount - Runtime Directory
   Loaded: loaded (/usr/lib/systemd/system/var-run.mount; static; vendor preset: disabled)
   Active: inactive (dead

Cause

The cause of this issue is duplicated tmpfs Filesystem. The upgraded system end up with two run-time directories:

linux-aoie# mount | grep run 
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /var/run type tmpfs (rw,nosuid,nodev,mode=755)


In SLES 11 SP4 /var/run is used as tmpfs ; In SLES 12 /run is used as tmpfs. /var/run is a link to /run

linux-aoie# ll /var/run
lrwxrwxrwx 1 root root 4 May  2  2017 /var/run -> /run


But when /var/run is a directory (like with SLES 11), var-run.mount systemd unit gets activated and mounts /var/run as tmpfs 

linux-aoie# systemctl status var-run.mount
● var-run.mount - Runtime Directory
   Loaded: loaded (/usr/lib/systemd/system/var-run.mount; static; vendor preset: disabled)
   Active: active (mounted) since Thu 2019-04-25 16:39:46 CEST; 13min ago
    Where: /var/run
     What: tmpfs
  Process: 678 ExecMount=/usr/bin/mount /run /var/run -t bind -o bind (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)


This could be seen also in journalctl output: 

Apr 25 16:39:46 linux-aoie systemd[1]: var-run.mount: Directory /var/run to mount over is not empty, mounting anyway.
Apr 25 16:39:46 linux-aoie systemd[1]: Mounting Runtime Directory...
Apr 25 16:39:46 linux-aoie systemd[1]: Mounted Runtime Directory.


The following lines prevent systemd from mounting /var/run as tmpfs but systemd still mounting /var/run as tmpfs because its a directory. 

linux-aoie# grep Condition /usr/lib/systemd/system/var-run.mount
ConditionPathIsDirectory=/var/run
ConditionPathIsSymbolicLink=!/var/run

Additional Information

Above is a workaround for this problem. And should fix the issue. However SUSE Engineering is working on a fix that might be included in SLES12 SP5 DVD. 

Feedback service temporarily unavailable. For content questions or problems, please contact Support.