Memory usage exceeding the allocated amounts

[azhv]

Hello,
Recently we noticed that the memory usage on several hosts is constantly growing on daily basis. After few checks it seems that some VMs are consuming way more memory that they are expected to. Stopping the VM and booting it back on or migrating temporary fixes the issue. Dropping cache didn't helped. The VMs in the example below are set to use only 8GB or 12GB of memory. ( host vm.swappiness is set to 10 ).
I've hit few forum threads on the topic but couldn't find any solution or workaround the problem.
Any ideas/workarounds or hints what may cause the leak are more than welcome.
Thank you.

Spoiler: pveversion -v

proxmox-ve: 6.2-1 (running kernel: 5.4.34-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-1
pve-kernel-helper: 6.2-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.3
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-5
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Spoiler: top

top - 13:54:56 up 49 days, 34 min, 3 users, load average: 14.17, 14.76, 16.42
Tasks: 972 total, 6 running, 966 sleeping, 0 stopped, 0 zombie
%Cpu(s): 6.9 us, 4.3 sy, 0.0 ni, 88.5 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
MiB Mem : 257561.8 total, 2995.8 free, 202386.3 used, 52179.7 buff/cache
MiB Swap: 196608.0 total, 95109.7 free, 101498.3 used. 53280.4 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
69665 root 20 0 51.1g 9.7g 3764 S 5.3 3.8 36771:07 kvm
69565 root 20 0 49.4g 12.6g 4916 S 100.3 5.0 44734:33 kvm
67641 root 20 0 30.4g 6.1g 3644 S 10.2 2.4 13670:03 kvm
67550 root 20 0 25.6g 6.5g 3624 S 8.3 2.6 22649:38 kvm
5410 root 20 0 23.5g 6.3g 3516 S 9.9 2.5 46617:14 kvm
5480 root 20 0 18.1g 6.3g 3496 S 5.6 2.5 43602:44 kvm
36822 root 20 0 17.6g 6.2g 3564 S 104.3 2.5 17138:17 kvm
36890 root 20 0 14.6g 5.3g 3768 S 5.3 2.1 6183:07 kvm
30642 root 20 0 13.1g 11.5g 3616 S 17.2 4.6 9246:44 kvm
28283 root 20 0 13.0g 11.6g 3540 S 3.0 4.6 3682:43 kvm
24079 root 20 0 9796644 6.4g 3636 S 5.9 2.5 1177:17 kvm

Spoiler: meminfo

MemTotal: 263743292 kB
MemFree: 2731984 kB
MemAvailable: 54550868 kB
Buffers: 556148 kB
Cached: 50131692 kB
SwapCached: 3891656 kB
Active: 181658844 kB
Inactive: 65028260 kB
Active(anon): 172993100 kB
Inactive(anon): 23097796 kB
Active(file): 8665744 kB
Inactive(file): 41930464 kB
Unevictable: 302636 kB
Mlocked: 302636 kB
SwapTotal: 201326588 kB
SwapFree: 97396892 kB
Dirty: 1532 kB
Writeback: 0 kB
AnonPages: 192588412 kB
Mapped: 131248 kB
Shmem: 89912 kB
KReclaimable: 3071356 kB
Slab: 10246672 kB