Slow memory leak in 6.8.12-13-pve

E

ewsclass66

New Member
Oct 24, 2023
6
2
3
Hi all,
I have recently added a host to a cluster, it has a newer kernel version than the others and I am noticing a slow and gradual memory leak which is consuming around an additional 500MB a minute. This continues until the host grinds to a halt and becomes un-operational. This host is identical in spec to the others in the cluster, of which they are not experiencing the issue. This is a Ceph cluster, with OSDs on each host. Looking in htop, no processes are using extreme amounts of memory which points to a kernel memory leak. I am using ZFS for boot disks. ARC looks good:

Code: 
arc_summary -s arc

------------------------------------------------------------------------
ZFS Subsystem Report                            Mon Jul 28 12:52:21 2025
Linux 6.8.12-13-pve                                           2.2.8-pve1
Machine: redacted (x86_64)                             2.2.8-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                     4.1 %  672.8 MiB
        Target size (adaptive):                        73.7 %   11.8 GiB
        Min size (hard limit):                         73.7 %   11.8 GiB
        Max size (high water):                            1:1   16.0 GiB
        Anonymous data size:                          < 0.1 %  280.0 KiB
        Anonymous metadata size:                        0.0 %    0 Bytes
        MFU data target:                               37.5 %  241.8 MiB
        MFU data size:                                 26.3 %  169.6 MiB
        MFU ghost data size:                                     0 Bytes
        MFU metadata target:                           12.5 %   80.6 MiB
        MFU metadata size:                              4.1 %   26.5 MiB
        MFU ghost metadata size:                                 0 Bytes
        MRU data target:                               37.5 %  241.8 MiB
        MRU data size:                                 59.5 %  383.7 MiB
        MRU ghost data size:                                     0 Bytes
        MRU metadata target:                           12.5 %   80.6 MiB
        MRU metadata size:                             10.0 %   64.8 MiB
        MRU ghost metadata size:                                 0 Bytes
        Uncached data size:                             0.0 %    0 Bytes
        Uncached metadata size:                         0.0 %    0 Bytes
        Bonus size:                                     0.6 %    3.8 MiB
        Dnode cache target:                            10.0 %    1.6 GiB
        Dnode cache size:                               0.9 %   14.1 MiB
        Dbuf size:                                      1.0 %    6.7 MiB
        Header size:                                    0.4 %    2.4 MiB
        L2 header size:                                 0.0 %    0 Bytes
        ABD chunk waste size:                           0.2 %    1.1 MiB

ARC hash breakdown:
        Elements max:                                               9.5k
        Elements current:                              99.7 %       9.5k
        Collisions:                                                   19
        Chain max:                                                     1
        Chains:                                                        2

ARC misc:
        Deleted:                                                      17
        Mutex misses:                                                  0
        Eviction skips:                                                1
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                 279.0 KiB
        L2 eligible MFU evictions:                      0.0 %    0 Bytes
        L2 eligible MRU evictions:                    100.0 %  279.0 KiB
        L2 ineligible evictions:                                 4.0 KiB

Code: 
lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   48
  On-line CPU(s) list:    0-47
Vendor ID:                GenuineIntel
  BIOS Vendor ID:         Intel(R) Corporation
  Model name:             Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
    BIOS Model name:      Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz  CPU @ 3.0GHz
    BIOS CPU family:      179
    CPU family:           6
    Model:                85
    Thread(s) per core:   2
    Core(s) per socket:   12
    Socket(s):            2
    Stepping:             4
    CPU(s) scaling MHz:   75%
    CPU max MHz:          3700.0000
    CPU min MHz:          1200.0000
    BogoMIPS:             6000.00
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology
                          nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3
                          dnowprefetch cpuid_fault epb cat_l3 cdp_l3 pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512
                          dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_pkg_req vnmi
                          pku ospke md_clear flush_l1d arch_capabilities
Virtualization features:
  Virtualization:         VT-x
Caches (sum of all): 
  L1d:                    768 KiB (24 instances)
  L1i:                    768 KiB (24 instances)
  L2:                     24 MiB (24 instances)
  L3:                     49.5 MiB (2 instances)
NUMA:                 
  NUMA node(s):           2
  NUMA node0 CPU(s):      0-11,24-35
  NUMA node1 CPU(s):      12-23,36-47
Vulnerabilities:     
  Gather data sampling:   Mitigation; Microcode
  Itlb multihit:          KVM: Mitigation: VMX disabled
  L1tf:                   Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
  Mds:                    Mitigation; Clear CPU buffers; SMT vulnerable
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Mitigation; Clear CPU buffers; SMT vulnerable
  Reg file data sampling: Not affected
  Retbleed:               Mitigation; IBRS
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Mitigation; Clear CPU buffers; SMT vulnerable

Code: 
cat /proc/cmdline
initrd=\EFI\proxmox\6.8.12-13-pve\initrd.img-6.8.12-13-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
I'm going to try upgrading this host to a newer kernel version to see if that helps the issue. A reboot does not seem to fix this and the leak continues once rebooted.

Any suggestions?

New Host:
Code: 
pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-13-pve)
pve-manager: 8.4.5 (running version: 8.4.5/57892e8e686cb35b)
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
ceph: 19.2.2-pve1~bpo12+1
ceph-fuse: 19.2.2-pve1~bpo12+1
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
frr-pythontools: 10.2.2-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.2
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.2
libpve-cluster-perl: 8.1.2
libpve-common-perl: 8.3.2
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.3-1
proxmox-backup-file-restore: 3.4.3-1
proxmox-backup-restore-image: 0.7.0
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.4
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.5
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.12
pve-cluster: 8.1.2
pve-container: 5.3.0
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-4~bpo12+1
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.2
pve-firmware: 3.16-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.5
pve-qemu-kvm: 9.2.0-7
pve-xtermjs: 5.5.0-2
qemu-server: 8.4.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.8-pve1



Other Hosts (No memory leak):
Code: 
proxmox-ve: 8.4.0 (running kernel: 6.8.12-5-pve)
pve-manager: 8.4.5 (running version: 8.4.5/57892e8e686cb35b)
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph: 19.2.2-pve1~bpo12+1
ceph-fuse: 19.2.2-pve1~bpo12+1
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.2
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.2
libpve-cluster-perl: 8.1.2
libpve-common-perl: 8.3.2
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.3-1
proxmox-backup-file-restore: 3.4.3-1
proxmox-backup-restore-image: 0.7.0
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.4
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.5
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.12
pve-cluster: 8.1.2
pve-container: 5.3.0
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-4~bpo12+1
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.2
pve-firmware: 3.16-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.5
pve-qemu-kvm: 9.2.0-7
pve-xtermjs: 5.5.0-2
qemu-server: 8.4.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.8-pve1
Code: 
cat /proc/cmdline
initrd=\EFI\proxmox\6.8.12-5-pve\initrd.img-6.8.12-5-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
The existing hosts are awaiting a reboot to apply the new kernel which I am obviously obstaining from for now.
 
Last edited:
Manually setting the kernel at 6.8.12-5-pve has fixed the issue, so there is something up in the newer release.
 
Hello

any new information on this? We are facing the same issue. We have to clusters with 3 nodes each. Both clusters are on 6.8.12-13-pve. Both clusters should be identical (we can't find any differences at least).

BUT
Cluster 1 has slowly growing memory until OOM
Cluster 2 has no such issue

A kernel downgrade is the "last line of defence" so I hope to find any more information about this. Currently we are rebooting the cluster every two days. smem, slabtop, ps aux. Nothing shows huge processes, no issues in the logs or obvious faulty processes.
 
Cannot confirm mem leak to 6.8.12-13 in 5n cluster with 3x different HW, so maybe it's a driver to a hw used (like eg network card ...).
 
Could you please try updating to the newest kernel 6.8.12-14 and check whether you still notice memory leaks?
 
My bad, I did not mention: kernel 6.8.12-14 is currently available in the no-subscription repository, but not yet in the enterprise repository. If you are affected by the memory leak, you can update the kernel from the no-subscription repository in the meantime. Please let us know whether version 6.8.12-14 fixes the memory leaks.
 
l.leahu-vladucu said:
My bad, I did not mention: kernel 6.8.12-14 is currently available in the no-subscription repository, but not yet in the enterprise repository. If you are affected by the memory leak, you can update the kernel from the no-subscription repository in the meantime. Please let us know whether version 6.8.12-14 fixes the memory leaks.
Click to expand...
We will check on Monday. Thanks so far. Will report back
 
Just to elaborate on this a bit more, there are the following ZFS issues on GitHub:

There's also a pull request that has been merged and integrated in ZFS 2.3.4:

This bug seems to exist since kernel 6.8.12-8-pve (with ZFS 2.2.7). In other words, kernel 6.8.12-6-pve (with ZFS 2.2.6) is the last version before the newest patches that does not have these issues.

This effectively means, as of now:
  1. Proxmox VE 9.0 with kernel 6.14.11-1 contains ZFS 2.3.4, which contains the fix
  2. Proxmox VE 8.x with kernel 6.8.12-14 also contains the fix, backported by us
  3. Proxmox VE 8.x with kernel 6.14.8-2~bpo12+1 does NOT yet contain the fix (it has already been backported by us, but not yet released)

Please note that, as of now, kernels 6.14.11-1 (PVE 9.0) and 6.8.12-14 (PVE 8.x) are not yet available in the enterprise repositories. In that case, feel free to enable the no-subscription repository and upgrade only the proxmox-kernel package:
  • for Proxmox VE 8.x: apt update && apt install proxmox-kernel-6.8
  • for Proxmox VE 9.0: apt update && apt install proxmox-kernel-6.14

Please let me know whether the new versions fixed your out of memory issues, or whether you still have them. Thanks!
 
  • Like
Reactions: UdoB, Johannes S and hb9hil
Good morning

After some testings we do not see that 6.8.12-14 does fix this issue. The leak is still persistent


Code: 
proxmox-ve: 8.4.0 (running kernel: 6.8.12-14-pve)
pve-manager: 8.4.11 (running version: 8.4.11/14a32011146091ed)
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8.12-14-pve-signed: 6.8.12-14
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph: 17.2.8-pve2
ceph-fuse: 17.2.8-pve2
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.2
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.2
libpve-cluster-perl: 8.1.2
libpve-common-perl: 8.3.4
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.7
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.4-1
proxmox-backup-file-restore: 3.4.4-1
proxmox-backup-restore-image: 0.7.0
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.4
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.5
proxmox-widget-toolkit: 4.3.13
pve-cluster: 8.1.2
pve-container: 5.3.0
pve-docs: 8.4.1
pve-edk2-firmware: 4.2025.02-4~bpo12+1
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.2
pve-firmware: 3.16-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.5
pve-qemu-kvm: 9.2.0-7
pve-xtermjs: 5.5.0-2
qemu-server: 8.4.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.8-pve1
 
EDIT: This dnode issue doesn't seem to be an issue here. But the hosts eating memory without any hint which process is eating up that memory. Everything is clean so far.

Code: 
arc_summary -s arc | grep "ARC size\|Target\|Dnode"

ARC size (current):                                     0.5 %  638.3 MiB
        Target size (adaptive):                         6.2 %    7.9 GiB
        Dnode cache target:                            10.0 %   12.6 GiB
        Dnode cache size:                               0.2 %   23.5 MiB
 
Thanks for testing. We had some reports that the newer kernel helped, but also some reports that it didn't - so we'll need to investigate further. It seems there are indeed two different issues.

I tried reproducing the issues over the weekend on a test cluster using Proxmox VE 8.4 and kernel 6.8.12-14-pve, but everything worked well on my end.

At this point I would be grateful if some of you who are able to reproduce the issue would test some kernel versions. In the end, we will probably need to bisect the kernel and find the exact patch that causes issues, but it would help us if we could:
  • find the last correctly working kernel version for each kernel branch
  • find the common patches that have been applied to all kernel branches
  • try to find a common driver, kernel module or setting that might cause the issues you are experiencing

While we can also bisect the kernel without this information, it will take much longer, so we would be happy for any hints in the right direction.

If you would like to help, would it be possible to test with older kernel versions and tell us your experiences? To be precise:
  • With Proxmox VE 8.x and kernel 6.8, it would be interesting to know whether the older kernels 6.8.12-12-pve or 6.8.12-11-pve fix your issues. We currently have a few reports that the issues began with kernel 6.8.12-12-pve, so according to these reports, kernel 6.8.12-11-pve was the last one that did notcause issues. We would however like to confirm this again, if some of you would like to test both versions.
    • EDIT 2: As explained by my colleague here and here, a patched kernel for Proxmox VE 8.x and kernel 6.8 is available: kernel-6.8-ice-memleak-fix-1
    • EDIT 3: We have some reports that the patched kernel kernel-6.8-ice-memleak-fix-1 improves the situation, but does not fix it entirely. We are currently investigating further. However, we would like to confirm this again, so please continue to send us feedback.
  • With Proxmox VE 8.x and kernel 6.14, we have reports that kernel 6.14.8-2-bpo12-pve also contains the issue, but we currently do not have information about the last correctly working kernel version. If you find the last correctly working kernel, please let us know.
    • EDIT 1: since we have some reports from PVE 9.0 that kernel 6.14.4-1+deb13u1-pve did not fix the issue, I'm very sure that the issue also exists in PVE 8.x with kernel 6.14.4-1 (but feel free to test it if you have the possibility). Kernel versions 6.14.0-1 and 6.14.0-2 are older than this and could further be tested.
  • With Proxmox VE 9.0 and kernel 6.14, we have reports that kernels 6.14.11-1 and 6.14.8-2 both contain the issue, but we currently do not have information about the last correctly working kernel version. If you find the last correctly working kernel, please let us know.
    • EDIT 1: we have some reports that even the oldest kernel 6.14.4-1+deb13u1-pve contains the issue - we would however like to confirm this again, so please let us know whether you experience the same.
    • EDIT 2: As explained by my colleague here and here, a patched kernel for Proxmox VE 9.0 and kernel 6.14 is available: kernel-6.14-ice-memleak-fix-1
    • EDIT 3: We have some reports that the patched kernel kernel-6.14-ice-memleak-fix-1 improves the situation, but does not fix it entirely. We are currently investigating further. However, we would like to confirm this again, so please continue to send us feedback.
  • EDIT 4 (applies to both PVE 8.x with 6.8 and 6.14, and 9.0 with 6.14): Dear community, thanks to your valuable feedback, we are now pretty sure that the issue is related to the ICE driver and/or Intel E810 NIC, possibly also related to Jumbo Frames / MTU 9000. We will proceed with bisecting the kernel using this hardware very soon.

If you decide to install an older kernel, you can pin the kernel version either temporarily or permanently.


EDIT 1: Updated the post with information about the kernel versions that contain the issue.

EDIT 2: Updated the post with information about the patched kernel versions.

EDIT 3: Updated the post with the newest information about the patched kernel versions.

EDIT 4: See above.
 
Last edited:
I can confirm I am experiencing the same memory leak issue. I have a 5-node cluster with the following configuration:

  • CPU: AMD EPYC 9124
  • RAM: 192 GB DDR5 ECC
  • Storage: 6 × Samsung PM9A3 NVMe (U.2)
  • NIC: Intel Ethernet Network Adapter E810-CQDA2 Dual port 100 GbE (QSFP28) PCIe 4.0 ×16, LP

The memory leak gradually appears on all nodes. There is no visible growth in user-space processes, so it clearly looks like a kernel/slab leak.


With kernel 6.8.12-11 everything works fine, but the problem starts from 6.8.12-12 and newer. Based on my setup, I strongly believe the cause is related to the ice driver for the Intel E810 NIC.
 
I've got the same issue - not sure if it matters but I have Intel 810 NICs.
Still running 6.8.12-13-pve.

AMD EPYC 7313P
192GB RAM
ZFS mirror for boot
Ceph w. 3x disks
Intel 810 quad port 25G adapter
Subsystem: Super Micro Computer Inc Ethernet Controller E810-C for SFP [15d9:1c40]
Kernel driver in use: ice
Kernel modules: ice

*Updated with NIC info
 
Last edited:
Good evening, everyone.
I can also confirm a memory leak with kernels higher than version 6.8.12-11.
I don't use ZFS, but rather Ceph with a cluster configuration of three hyperconverged nodes.
I have performed several tests by changing kernel versions, upgrading to Proxmox version 9, and using different versions of Ceph (18.2.2, 18.2.7, and 19.2.3).
I also ran tests in physical and virtual environments (I have been working on these tests almost full-time for the past 15 days).
I can confirm that the leak issue is resolved by upgrading to kernel version 6.8.12-11.

The tests carried out in physical and virtual environments seem to rule out the possibility of a driver issue.

I can provide more details on the tests carried out if required.

1757620601938.png
 
Good morning.

I can confirm that on PVE 8.4.12 the Kernel 6.8.12-11-pve solves the memory leak. Kernel 6.8.12-12 and -13 are already affected by the leak.
I manage another 3-node cluster on PVE 9.0.6 with Kernel 6.14.11-1-pve and do not see any leaks here but unfortunately on this cluster there isn't that much load to give a well-founded assessment
 
Confirmation from my setup (PVE 9.0.9, 5 nodes, AMD EPYC 9124, 192 GB RAM, 6× Samsung PM9A3 U.2, Intel Ethernet Network Adapter E810-CQDA2 Dual port 100 GbE (QSFP28) PCIe 4.0 ×16, LP):


  • ZFS only for boot, ARC capped with zfs_arc_max=1G. Main storage is Ceph.
  • With Ceph mostly idle, there is no memory leak or it is extremely small.
  • When running DiskSpd tests on Windows Server 2025 VMs, the leak appears: the more intensive the tests (or the more VMs I run in parallel), the faster the memory leak fills up all RAM across nodes. Userspace looks clean → this is a kernel/slab leak.
  • Downgrading and pinning kernel 6.8.12-11 fully resolves the issue. Under the same workload, memory stays stable.
 
  • Like
Reactions: julien.merle and hb9hil
Thanks for all the information everyone - this is very useful for us. At the moment we think that the memory leaks might be network-related, and with Ceph causing a lot of network traffic, we think this might be the reason why the memory leaks seem to be so extreme. To confirm whether this is true, could you please tell us (if you didn't already tell us above):
  1. Whether you're using Ceph.
  2. Which NIC is used for Ceph (exact NIC model). If multiple NICs are used (e.g. for public/cluster network, or bondings), please tell us the information for all used NICs.
  3. Which driver is used for the NIC (you can find that by checking the output of lspci -nnk and checking "Kernel driver in use").
  4. EDIT (new addition): Which MTU the network interfaces for Ceph are using.

Thanks!
 
Last edited:
  • Like
Reactions: Johannes S
  • YES – Ceph is the main storage backend for all my virtual machines.
  • Network: Intel Ethernet Network Adapter E810-CQDA2, dual-port 100 GbE (QSFP28), PCIe 4.0 ×16, low-profile. First port is dedicated to the public network, second port is dedicated to the cluster network.
  • Hardware/driver details: Intel Ethernet Controller E810-C for QSFP [8086:1592], revision 02. Subsystem: Intel Ethernet Network Adapter E810-C-Q2 [8086:0002]. Kernel driver in use: ice. Kernel module: ice. Both ports (41:00.0 and 41:00.1) are active under the same driver.
 
  • Yes - Ceph RBD for storage backend VM Disk and CephFS for snippets iso
  • Network Intel Corporation Ethernet Controller E810-XXV for SFP [8086:159b] (rev 02) 2x25Gbs 0x8001e4fb 23.0.8 bonding and same network for public and cluster Ceph network
  • driver: ice
    version: 6.8.12-11-pve
    firmware-version: 4.51 0x8001e4fb 23.0.8 (I switched the cluster to the stable kernel version)

More Details:
No memory leak or very low memory leak when OSD services are stopped.
No memory leak or very low memory leak if Proxmox servers are clients of an external Ceph cluster.
 
  • Like
Reactions: waltar
You must log in or register to reply here.
Top Bottom