Ballooning doesn't appear to be taking unused RAM back?

[harmonyp]

If I run htop -s PERCENT_MEM on the node I can see there are multiple virtual machines allocated RAM they are not using

Example this VM id 159 is reserving 10.8GB of RAM

However it's usage over the last hour shows as peaking at that almost an hour ago

qm guest exec 159 -- free -m
{
   "exitcode" : 0,
   "exited" : 1,
   "out-data" : "              total        used        free      shared  buff/cache   available\nMem:          10765         810        9606          32         348        9633\nSwap:             0           0           0\n"
}

It's allocated

The nodes overall RAM usage is now at 75% however I noticed this was happening way before 50% RAM usage. As it's getting closer and closer to near 100% usage I noticed the RAM allocated inside this virtual machine is going lower and lower. It's currently showing 10765 MB (as above), yesterday it was 16GB.

I have noticed that this is only currently happening to virtual machines allocated more than 10GB of RAM. A VM that is allocated 8GB (2GB min) shows the full 8GB

qm guest exec 154 -- free -m
{
   "exitcode" : 0,
   "exited" : 1,
   "out-data" : "              total        used        free      shared  buff/cache   available\nMem:           7960        1200        6566           0         193        6525\nSwap:          1457           0        1457\n"
}

My question are

1) Why is this happening? My understanding is Ballooning should only activate once the nodes overall RAM usage is at 80%+
2) Once this does happen it's also my understanding that these virtual machines should still show the full 24GB of RAM and the reserved RAM should be sent back to the node. The only time this virtual machine should show less than 24GB is if the node is completely out of memory and there is nothing for ballooning to take back?

 

[harmonyp]

Running

echo 3 | sudo tee /proc/sys/vm/drop_caches

has reduced the nodes memory from 70% to 55%. It is running ZFS (limited to 64GB) but that does not take account for that much of a drop based off arcstat size reduction.

Is there any way to force the node to reduce the resident set size (RES) of a running VM?

 

[harmonyp]

Could this also be what's causing the issue for me? https://www.mail-archive.com/pve-devel@lists.proxmox.com/msg02453.html other than this I can't find others having this same issue.

I do have an arc limit though

/etc/modprobe.d/zfs.conf

#Max Ram Used by ZFS - 64GB
options zfs zfs_arc_max=68719476736
#Min Ram Used by ZFS - 20GB
options zfs zfs_arc_min=21474836480

arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
21:57:26     4     0      0     0    0     0    0     0    0   19G   20G   122G

pveversion -v

proxmox-ve: 6.4-1 (running kernel: 5.4.114-1-pve)
pve-manager: 6.4-8 (running version: 6.4-8/185e14db)
pve-kernel-5.4: 6.4-3
pve-kernel-helper: 6.4-3
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.114-1-pve: 5.4.114-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.9-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-6
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
pve-zsync: 2.2
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1