Possible memory leak on I9-13900K/PVE 8.0.4 platform

[Kobayashi_Bairuo]

Hi, I did a clean install today using the PVE ISO file from https://enterprise.proxmox.com/iso/proxmox-ve_8.0-2.iso and updated to PVE 8.0.4 via apt after installation. In this version, there seems to be a memory leak issue, when I import virtual machines that worked fine in 8.0.3, these virtual machines all have memory leaks to varying degrees in version 8.0.4. I tried changing the CPU Type but to no avail.

System environment:
CPU: I9-13900K
MEM: 128G DDR5 4400MHZ
MotherBoard: SuperMicro X13SAE-F

Please let me know if you need any information from me. Since PVE 8.0.4 is too new, there is no information about it on Google


root@10-10-0-7:~# pveversion
pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-6-pve)

root@10-10-0-7:~# pveversion -v
proxmox-ve: 8.0.2 (running kernel: 6.2.16-6-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-6-pve: 6.2.16-7
proxmox-kernel-6.2: 6.2.16-7
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.4
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.7
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.3
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-4
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

 

[fiona]

Hi,

when I import virtual machines that worked fine in 8.0.3, these virtual machines all have memory leaks to varying degrees in version 8.0.4.

did you verify this on the same host or were the VMs running on another host before? Please share the VM configuration with qm config <ID> --current. If you have ballooning or KSM enabled, are you sure it's not just the guest in the VMs using more of their assigned memory over time?

 

[fiona]

Can you also check if booting an older kernel solves the issue?

 

[Kobayashi_Bairuo]

I fixed it by reinstalling PVE so can't reproduce it now. The system has a total of 128G memory. Even if all VMs use 100% memory, the memory usage should not exceed 66G, but in fact PVE occupies 105G, and there is still 30G+ memory retention after all VMs are closed. I tried booting the system with PVE-Kernel-6.2.16-3 but it didn't work. I started bolloon on the virtual machine, and he showed that although the VM's Summary page memory has been used 100%, the memory usage of the entire system is still rising, and using htop inside the VM seems to be just a bunch of cache

 

[fiona]

I fixed it by reinstalling PVE so can't reproduce it now. The system has a total of 128G memory. Even if all VMs use 100% memory, the memory usage should not exceed 66G, but in fact PVE occupies 105G, and there is still 30G+ memory retention after all VMs are closed.

Please check which processes are using that 30G of memory when no VM is running, e.g. press Shift+M in top to sort by memory usage.