Cannot shutdown VM

Matthew Daniel · Feb 9, 2021

I cannot shutdown VMs. This happens after 1032 iterations of VMs being started and shutdown and occurs on both PVE 6.2 and 6.3.

The client OS shuts down but the VM fails to stop. The VNC Console for Ubuntu VMs shows "systemd-shutdown[1]: Failed to finalize DM devices, ignoring" while the VM tries to shutdown. You then have the cancel the Shutdown task via PVE Web GUI and additionally use ‘qm stop’ to switch off the VM.

Syslog is showing many entries of 'qmeventd - Too many open files'.

Checking qmeventd open files count with “lsof | grep qmevent | wc -l” shows approx. 40 open files after I reboot the PVE host server. Each time I start, then shutdown a VM the number of qmeventd open files increases. When it exceeds 1032, you are no longer able to shutdown any running VM.

I’m currently having to manually restart qmeventd to avoid this issue.

Stefan_R · Feb 9, 2021

Hm, that's certainly not supposed to happen. Could you post the actual output of lsof | grep qmevent (after a few VM starts and stops) and maybe pveversion -v just to be sure?

Matthew Daniel · Feb 9, 2021

Stefan_R said:
Hm, that's certainly not supposed to happen. Could you post the actual output of lsof | grep qmevent (after a few VM starts and stops) and maybe pveversion -v just to be sure?

On this host, running 100 VMs increased lsof | grep qmevent | wc -l from 20 to 120. Then after shutting these down, it was still at 120. Output of lsof | grep qmevent after this event is:

Code:

# lsof | grep qmevent
qmeventd   9626                             root  cwd       DIR              253,1      4096          2 /
qmeventd   9626                             root  rtd       DIR              253,1      4096          2 /
qmeventd   9626                             root  txt       REG              253,1     23080     395719 /usr/sbin/qmeventd
qmeventd   9626                             root  mem       REG              253,1    146968     396411 /usr/lib/x86_64-linux-gnu/libpthread-2.28.so
qmeventd   9626                             root  mem       REG              253,1    468944     396398 /usr/lib/x86_64-linux-gnu/libpcre.so.3.13.3
qmeventd   9626                             root  mem       REG              253,1   1824496     396197 /usr/lib/x86_64-linux-gnu/libc-2.28.so
qmeventd   9626                             root  mem       REG              253,1   1166640     401213 /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.5800.3
qmeventd   9626                             root  mem       REG              253,1     51136     394379 /usr/lib/x86_64-linux-gnu/libjson-c.so.3.0.1
qmeventd   9626                             root  mem       REG              253,1    165632     396162 /usr/lib/x86_64-linux-gnu/ld-2.28.so
qmeventd   9626                             root    0r      CHR                1,3       0t0          6 /dev/null
qmeventd   9626                             root    1u     unix 0x0000000080f37cd7       0t0    4284966 type=STREAM
qmeventd   9626                             root    2u     unix 0x0000000080f37cd7       0t0    4284966 type=STREAM
qmeventd   9626                             root    3u     unix 0x00000000863d50da       0t0    4284967 /var/run/qmeventd.sock type=STREAM
qmeventd   9626                             root    4u  a_inode               0,14         0      14524 [eventpoll]
qmeventd   9626                             root    5u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root    6u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root    7u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root    8u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root    9u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   10u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   11u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   12u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   13u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   14u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   15u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   16u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   17u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   18u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   19u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   20u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   21u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   22u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   23u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   24u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   25u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   26u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   27u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   28u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   29u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   30u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   31u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   32u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   33u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   34u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   35u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   36u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   37u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   38u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   39u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   40u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   41u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   42u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   43u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   44u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   45u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   46u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   47u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   48u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   49u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   50u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   51u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   52u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   53u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   54u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   55u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   56u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   57u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   58u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   59u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   60u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   61u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   62u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   63u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   64u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   65u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   66u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   67u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   68u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   69u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   70u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   71u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   72u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   73u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   74u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   75u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   76u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   77u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   78u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   79u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   80u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   81u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   82u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   83u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   84u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   85u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   86u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   87u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   88u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   89u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   90u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   91u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   92u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   93u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   94u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   95u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   96u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   97u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   98u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root   99u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  100u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  101u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  102u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  103u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  104u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  105u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  106u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  107u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  108u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  109u  a_inode               0,14         0      14524 [pidfd]
qmeventd   9626                             root  111u  a_inode               0,14         0      14524 [pidfd]

Matthew Daniel · Feb 9, 2021

Output of pveversion -v is:

Code:

# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-4
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-2
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Stefan_R · Feb 9, 2021

We managed to reproduce the issue here (at least a symtomatically very similar one), and I've just posted a potential fix on the mailing list:
https://lists.proxmox.com/pipermail/pve-devel/2021-February/046970.html

If you're comfortable with building our packages yourself, you can check out our developer documentation here. Otherwise the fix should be available to you in upcoming qemu-server versions.

Matthew Daniel · Feb 10, 2021

Stefan_R said:
We managed to reproduce the issue here (at least a symtomatically very similar one), and I've just posted a potential fix on the mailing list:
https://lists.proxmox.com/pipermail/pve-devel/2021-February/046970.html

If you're comfortable with building our packages yourself, you can check out our developer documentation here. Otherwise the fix should be available to you in upcoming qemu-server versions.

Thanks for your swift action on this issue.

I have setup a test server with your test repo deb [URL]http://download.proxmox.com/debian/pve[/URL] buster pvetest that currently has qemu-server 6.3-4, which still exhibits the problem. I will retest when upcoming versions become available

Matthew Daniel · Feb 12, 2021

Matthew Daniel said:
I will retest when upcoming versions become available

The test repo deb [URL]http://download.proxmox.com/debian/pve[/URL] buster pvetest has qemu-server 6.3-5 today, which I can happily say has fixed my reported problem. Once this is on the production repo, I will test and confirm the fix on my live environment.

Thanks again

Matthew Daniel · Mar 4, 2021

I can confirm the problem solved in latest release:
Kernel Version: Linux 5.4.101-1-pve #1 SMP PVE 5.4.101-1 (Fri, 26 Feb 2021 13:13:09 +0100)
PVE Manager Version: pve-manager/6.3-4/0a38c56f

hmohr · Mar 4, 2021

Nice!
Im using pve 5.2 and whenever i'm not able to shutdown, i login via ssh and use qm stop vmid. Happens to work too!

Search

Search

Cannot shutdown VM

Matthew Daniel

Active Member

Stefan_R

Proxmox Retired Staff

Matthew Daniel

Active Member

Matthew Daniel

Active Member

Stefan_R

Proxmox Retired Staff

Matthew Daniel

Active Member

Matthew Daniel

Active Member

Matthew Daniel

Active Member

hmohr

Member