Might be worth testing the regular 6.14.11-3-pve (2025-09-22T10:13Z) kernel instead of the -ice-fix builds, since it already carries relevant net/mm changes. I’ve since moved from Intel E810 to Mellanox, so can’t verify current ice behavior myself.
Slow memory leak in 6.8.12-13-pve
- Thread starter ewsclass66
- Start date
Might be worth testing the regular 6.14.11-3-pve (2025-09-22T10:13Z) kernel instead of the -ice-fix builds, since it already carries relevant net/mm changes. I’ve since moved from Intel E810 to Mellanox, so can’t verify current ice behavior myself.
As I posted last Thursday, I tested that as well with no change. That kernel does *not* contain the "ice: fix Rx page leak on multi-buffer frames" fix (which is the -ice-fix builds), though of course those also don't fix the issue.
Apparently I spoke too soon on the -ice-fix builds. The buffer_head slab allocations do rise for the first few hours, but eventually seem to flatten out. I see something like 300MB of used buffer_head slab allocations (per slabtop) which of course seems pretty excessive, but total system memory usage appears to be flat over the last six hours with the 6.14 ice-fix build.
Last edited:
Hi,
I have been testing 6.14.11-3-pve since one week onto a 5 nodes cluster with jumbo frames (9000) on Ceph interfaces, and I can still see a memory leak. It depends on each node, one of them is eating his memory much faster than others, but I can't explain why (they are all similar in hardware and configuration). But they are all leaking anyway.
I have been testing 6.14.11-3-pve since one week onto a 5 nodes cluster with jumbo frames (9000) on Ceph interfaces, and I can still see a memory leak. It depends on each node, one of them is eating his memory much faster than others, but I can't explain why (they are all similar in hardware and configuration). But they are all leaking anyway.
Attachments
On a 4-node Ceph cluster (dual 100 GbE Intel E810, MTU 9000) I ran ~16 Windows Server 2025 VMs under continuous diskspd.exe load for a full week (~1 % wear on PM9A3 drives). With kernels 6.8.12-11-pve and 6.14.11-3-pve I saw no memory leaks at all, while the “-icefix” builds still showed a slow increase. I eventually replaced all E810s with Mellanox, and since then memory usage has remained stable.
I will replace 6.14.11-3-pve with 6.8.12-11-pve tomorrow morning on all nodes. Since I have no plan to add charges in the next few days, it will be a good way to compare.
The opt-in kernel 6.17 is now available for Proxmox VE 9.0 in the pve-test and pve-no-subscription repositories, if you would like to test it (official announcement of kernel 6.17 will come soon EDIT: added link to announcement post). Currently, version 6.17.1-1-pve is available, which contains all ICE driver fixes released upstream. If you have the possibility (and especially for non-production and/or non-critical systems), feel free to try the latest kernel 6.17 and let us know whether it improves the situation in any way (or whether it's the same, or worse).
Since it's an opt-in kernel, you will need to install it manually:
If you have any DKMS modules (e.g. Nvidia driver or other external kernel modules), you will also need to install the headers for the new kernel:
While kernels 6.8 and 6.14 also contain backported fixes from newer kernels (backported both by Ubuntu and by us), it might be possible that some further fixes might be available in the newest kernel that have not yet been backported, and might improve the situation.
Since it's an opt-in kernel, you will need to install it manually:
Code:
apt update && apt install proxmox-kernel-6.17If you have any DKMS modules (e.g. Nvidia driver or other external kernel modules), you will also need to install the headers for the new kernel:
Code:
apt update && apt install proxmox-kernel-6.17 proxmox-headers-6.17While kernels 6.8 and 6.14 also contain backported fixes from newer kernels (backported both by Ubuntu and by us), it might be possible that some further fixes might be available in the newest kernel that have not yet been backported, and might improve the situation.
Last edited:
After an hour I see that "dmaengine-unmap-2" is behaving like the other nodes in slabtop now (i.e. buffer_head on top). I suspect the issue is fixed with 6.17, or at least much much improved. We will se how it goes over a couple of days.
Yep, all is well with 6.17.
Node memory usage over the last week:
And the last 12 hours (no increase at all!)
Node memory usage over the last week:
And the last 12 hours (no increase at all!)
Last edited: