pveproxy / spiceproxy leaking memory

[ckt]

Recently I found several VMs turned off (killed by oomkiller) due to insufficient system memory. Since the host machine isn't overprovisioned I asked myself what was causing this - and soon found the root cause: pveproxy and spiceproxy have many processes sleeping (some with (shutdown) in their name, according to the process list, being in that state of many days). Running "service pveproxy/spiceproxy restart" free'd up several gigabytes (6 GiB in my case) of system memory.

Does anyone else have this issue? (try running "ps aux" and check if there are many sleeping pveproxy worker (shutdown) processes (and/or their spiceworker equivalent). How can I mitigate this issue? It looks like pveproxy and spiceproxy aren't behaving correctly; shutdown should indicate that the process should go away after a short period of time I guess.

# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.18-1-pve)
pve-manager: 6.0-7 (running version: 6.0-7/28984024)
pve-kernel-5.0: 6.0-8
pve-kernel-helper: 6.0-8
pve-kernel-4.15: 5.4-7
pve-kernel-5.0.21-2-pve: 5.0.21-3
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-5.0.18-1-pve: 5.0.18-3
pve-kernel-4.15.18-19-pve: 4.15.18-45
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.12-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-4
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-8
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1

 

[wolfgang]

Hi,

here we are not aware of this behavior.
Please update your Proxmox VE, because you run on an outdated version.
There are many fixes in the kernel.

 

[RMM]

I've got exactly the same Problem.
That was one of the reasons why I've updated to proxmox 6.1. But the proplem still appears.
The interesting thing is it only happens on two out of 3 servers. They've got the same hardware and software. The amount of virtual machines running are different. On server a the behaviour doesn't happen, but on server b and c.
And server b uses zfs replication to regularly (like every 10 minutes) copy one of the virtual block devices to server c. I've set that up over the gui.
Otherwise I can't imagine any obvious difference.

Restarting the two services frees up the memory again (in my case that were around 8GB)

Here a small part of the process list:

777 ?        S      1:39 spiceproxy worker (shutdown)
778 ? S 1:43 pveproxy worker (shutdown)
933 ? S 0:58 spiceproxy worker (shutdown)
956 ? S 0:56 pveproxy worker (shutdown)
957 ? S 0:57 pveproxy worker (shutdown)
958 ? S 0:57 pveproxy worker (shutdown)
1038 ? S 3:05 spiceproxy worker (shutdown)
1145 ? I< 0:23 [kworker/3:1H-kblockd]
1146 ? SN 0:00 [zthr_procedure]
1147 ? SN 0:00 [zthr_procedure]
1204 ? S 3:07 pveproxy worker (shutdown)
1205 ? S 3:04 pveproxy worker (shutdown)
1206 ? S 3:03 pveproxy worker (shutdown)
1415 ? I< 0:04 [kworker/7:1H-kblockd]
1471 ? I< 0:04 [kworker/0:1H-kblockd]
1939 ? I< 0:00 [rpciod]
1940 ? I< 0:00 [xprtiod]
1941 ? I< 0:00 [iscsi_eh]
1946 ? I< 0:00 [ib-comp-wq]
1947 ? I< 0:00 [ib-comp-unb-wq]
1948 ? I< 0:00 [ib_mcast]
1949 ? I< 0:00 [ib_nl_sa_wq]
1954 ? I< 0:00 [rdma_cm]
2002 ? S 1:34 spiceproxy worker (shutdown)
2003 ? S 1:37 pveproxy worker (shutdown)
2004 ? S 1:38 pveproxy worker (shutdown)
2128 ? I< 0:03 [kworker/1:1H-kblockd]
2129 ? S 3:06 spiceproxy worker (shutdown)
2215 ? I< 0:00 [cryptd]
2216 ? S 0:00 [irq/129-mei_me]
2425 ? S 1:44 spiceproxy worker (shutdown)
2427 ? S 1:45 pveproxy worker (shutdown)
2445 ? Ss 8:06 /lib/systemd/systemd-logind
8640 ? S 2:35 spiceproxy worker (shutdown)
8689 ? S 2:50 pveproxy worker (shutdown)
8690 ? S 2:52 pveproxy worker (shutdown)
8691 ? S 2:48 pveproxy worker (shutdown)
8769 ? S 0:54 spiceproxy worker (shutdown)
8980 ? I 0:00 [kworker/3:2-events]
9178 ? Ssl 3:50 /usr/sbin/rsyslogd -n -iNONE
9221 ? S 2:48 spiceproxy worker (shutdown)
9501 ? S 2:57 spiceproxy worker (shutdown)
9507 ? S 2:58 pveproxy worker (shutdown)
9508 ? S 2:58 pveproxy worker (shutdown)
9579 ? S 3:03 spiceproxy worker (shutdown)
9606 ? S 1:18 spiceproxy worker (shutdown)
9649 ? Ss 0:00 sshd: root@pts/2
9663 ? S 1:18 pveproxy worker (shutdown)
9664 ? S 1:21 pveproxy worker (shutdown)
9665 ? S 1:21 pveproxy worker (shutdown)
9782 ? I 0:00 [kworker/0:1-events]
9786 ? S 3:08 pveproxy worker (shutdown)
9787 ? S 3:05 pveproxy worker (shutdown)
9910 pts/2 Ss 0:00 -bash
10058 ? S 3:27 spiceproxy worker (shutdown)
10452 ? S 1:53 spiceproxy worker (shutdown)
10639 ? S 0:09 pveproxy worker (shutdown)
11551 ? I 0:00 [kworker/4:0-events]
12397 ? S 1:35 pveproxy worker (shutdown)
12499 ? I 0:00 [kworker/3:0-events]
12963 ? S 0:00 spiceproxy worker
13013 ? S 0:01 pveproxy worker
13038 ? S 0:01 pveproxy worker
13039 ? S 0:00 pveproxy worker
13092 ? Ssl 0:04 /usr/sbin/pvefw-logger
13460 ? S 2:08 spiceproxy worker (shutdown)
13933 ? S 0:02 spiceproxy worker (shutdown)
13980 ? S 0:02 pveproxy worker (shutdown)
13981 ? S 0:02 pveproxy worker (shutdown)
13986 ? S 0:40 spiceproxy worker (shutdown)
14324 ? S 2:39 spiceproxy worker (shutdown)
14347 ? S 0:37 pveproxy worker (shutdown)
14348 ? S 0:38 pveproxy worker (shutdown)
14349 ? S 0:37 pveproxy worker (shutdown)
14601 ? S 1:03 spiceproxy worker (shutdown)
14799 ? S 1:09 pveproxy worker (shutdown)
14800 ? S 1:09 pveproxy worker (shutdown)
14871 ? I 0:00 [kworker/5:1-mm_percpu_wq]
15111 ? S 0:00 pvedaemon worker
15249 ? S 1:13 spiceproxy worker (shutdown)
15281 ? S 1:14 pveproxy worker (shutdown)
16226 ? S 0:05 spiceproxy worker (shutdown)
16227 ? S 0:05 pveproxy worker (shutdown)
16447 ? I 0:00 [kworker/6:0-events]
17290 ? S 1:11 spiceproxy worker (shutdown)
17291 ? S 1:13 pveproxy worker (shutdown)
17292 ? S 1:14 pveproxy worker (shutdown)
17652 ? S 2:31 spiceproxy worker (shutdown)
17763 ? S 1:03 spiceproxy worker (shutdown)
17766 ? S 1:08 pveproxy worker (shutdown)
27829 ? S 1:50 spiceproxy worker (shutdown)
27834 ? S 1:58 pveproxy worker (shutdown)
27895 ? S 2:23 pveproxy worker (shutdown)
27896 ? S 2:25 pveproxy worker (shutdown)
28443 ? S 0:00 pickup -l -t unix -u -c
28638 ? S 3:21 spiceproxy worker (shutdown)
28668 ? S 0:32 spiceproxy worker (shutdown)
28670 ? S 0:36 pveproxy worker (shutdown)
28671 ? S 0:37 pveproxy worker (shutdown)
28672 ? S 0:37 pveproxy worker (shutdown)
28740 ? S 1:27 spiceproxy worker (shutdown)
28741 ? S 1:34 pveproxy worker (shutdown)
28742 ? S 1:32 pveproxy worker (shutdown)
28743 ? S 1:33 pveproxy worker (shutdown)
29581 ? I 0:00 [kworker/4:2]
29582 ? I 0:00 [kworker/0:0-events]
29584 ? S 0:55 spiceproxy worker (shutdown)
29598 ? S 0:50 pveproxy worker (shutdown)
29599 ? S 0:49 pveproxy worker (shutdown)
29600 ? S 0:51 pveproxy worker (shutdown)
29668 ? Ssl 0:02 /lib/systemd/systemd-timesyncd
29692 ? Ss 8:35 /lib/systemd/systemd-journald
29954 ? S 1:30 spiceproxy worker (shutdown)
29997 ? S 1:28 pveproxy worker (shutdown)
30172 ? S 2:45 spiceproxy worker (shutdown)
30180 ? S 2:42 pveproxy worker (shutdown)
30353 ? S 1:41 spiceproxy worker (shutdown)
30363 ? S 1:43 pveproxy worker (shutdown)
31016 ? S 3:16 spiceproxy worker (shutdown)
31725 ? Ss 16:56 /lib/systemd/systemd-udevd
31791 ? S 2:29 spiceproxy worker (shutdown)
32012 ? S 2:23 pveproxy worker (shutdown)
32013 ? S 2:25 pveproxy worker (shutdown)
32202 ? S 1:03 spiceproxy worker (shutdown)
32279 ? S 1:23 spiceproxy worker (shutdown)
32321 ? S 2:52 spiceproxy worker (shutdown)
32328 ? S 2:52 pveproxy worker (shutdown)
32329 ? S 2:51 pveproxy worker (shutdown)
32330 ? S 2:56 pveproxy worker (shutdown)
32505 ? I 0:00 [kworker/3:1-mm_percpu_wq]
32506 ? I 0:00 [kworker/3:3]
32595 ? S 0:28 spiceproxy worker (shutdown)
32634 ? S 0:57 pveproxy worker (shutdown)
32635 ? S 0:57 pveproxy worker (shutdown)

And here pve version

proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-4.15: 5.4-12
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.13.8-1-pve: 4.13.8-27
pve-kernel-4.13.4-1-pve: 4.13.4-26
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-15
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2