Increased latency on disks since 9.2.6 and associated 7.0.x kernel

W

weppa

Renowned Member
Nov 2, 2015
22
9
68
I have a couple of proxmox VE hosts, running 9.2.4 with 7.0 kernel.
These systems are updated regularly (like once a month) with proper reboot

For the last update (9.2.4 to 9.2.6 - on the 7.0 kernel) I have something weird happening that trigger alerts in MUNIN (yes i'm using munin)
The different hosts run on comptely different hardware and datacenter, and all of them display the same behavior, which never happened in the past

Screenshot_3Kz1zKCTPn.png

Basically it's like the disks are lagging, on dev/loop, WRITE IO
The screenshots are on "loop" but the graphs are the same on the real nvme (physical disks) graphs, or any other disk graph.

This behaviour can be seen on
- the host itself (screenshot)
- linux vms
- LXCs

So it propagates on all subsystems

All hosts run on MDADM SSD raid (enterprise) arrays, all systems are not loaded at all (less than n10% CPU usage) on modern and very fast hardware (EPYC)

I'll dig deeper in the hosts logs to try to pinpoint a root cause, but clearly this is specific to 9.2.6 (possible 9.2.5 whihc I skipped) and started happening on a variety of hardware the minut I rebooted the hosts.

Any advice ?
 
Last edited:
  • Like
Reactions: aviti
G

Post in thread 'Opt-in Linux 7.0 Kernel for Proxmox VE 9 available'

gfngfn256 said:
This actually maybe the area of concern, as I remember reading on these forums of users having issues with the latest kernels when using mdadm, which as you can see here, PVE does not support.
Click to expand...
I'm well aware of the limitations of mdraid, but at the same time it's in wide enough use that it would be pretty bad if something was broken with that in particular.

It's also possible that nothing is wrong in reality, and the statistics sources provided by the kernel are off for some reason - in particular I'm using data from /sys/block/<dev>/stat as parsed by the Munin...
C

Post in thread 'Opt-in Linux 7.0 Kernel for Proxmox VE 9 available'

Galaxis said:
Has anyone seen problems with I/O latencies on SSD/NVMEs when using the 7.0.14-6-pve kernel (from pve-no-subscription repos)?
Click to expand...

I experienced it on a 5 node cluster.
Each node used the same hardware with 3 NVME split into 3 partitions:
  • / is md raid 5
  • /vaz/lib/vz is a lvm raid 5, storage used only for backup and template
  • ceph osd, main storage
Proxmox metrics and VM performance do not seems affected but munin start to show latency peeks

Capture d'écran_20260727_100957-1.png
  • Like

Commonalities:
  • mdraid
  • Munin
 
Thanks @Neobin !

In the meantime
proxmox-boot-tool kernel pin 7.0.14-5-pve
proxmox-boot-tool refresh

UPDATE : I can confirm the problem DISAPPEARS with kernel 7.0.14-5

So it's specific to kernels after this one.
 
Last edited:
  • Like
Reactions: aviti, Galaxis and Johannes S
I encountered the same issue when updating from kernel 7.0.2-4-pve to kernel 7.0.14-8-pve. I also use Munin, and the latency graph started showing spikes as you mentioned.

I ran fio performance tests on both kernels and don't notice any major difference (at least not as significant as the graphs suggest).

Bash: 
# 7.0.2-4-pve
+ 4K random
  + read: IOPS=1141k, BW=4456MiB/s (4673MB/s)(240GiB/55149msec)
  + write: IOPS=330k, BW=1289MiB/s (1352MB/s)(75.5GiB/60003msec); 0 zone resets
+ sequential read/write
  + read: IOPS=11.5k, BW=11.3GiB/s (12.1GB/s)(10.0GiB/888msec)
  + write: IOPS=1421, BW=1421MiB/s (1490MB/s)(10.0GiB/7204msec); 0 zone resets

# 7.0.14-8-pve
+ 4K random
  + read: IOPS=1140k, BW=4455MiB/s (4671MB/s)(240GiB/55166msec)
  + write: IOPS=293k, BW=1143MiB/s (1198MB/s)(67.0GiB/60002msec); 0 zone resets
+ sequential read/write
  + read: IOPS=11.4k, BW=11.1GiB/s (11.9GB/s)(10.0GiB/901msec)
  + write: IOPS=1429, BW=1430MiB/s (1499MB/s)(10.0GiB/7162msec); 0 zone resets

I rolled back the kernel 7.0.2-4-pve while waiting for a more permanent solution.
 
Hi @bdr

thanks for chiming in!
Could you please also try if the problem is gone for you on kernel 7.0.14-5, as mentioned by @weppa.

How is your storage / your disks configured? Are you also using mdraid?

Best regards
Jonas
 
Thanks for the reply @j.theisen
I will test by rebooting into kernel 7.0.14-5 tomorrow morning to avoid disrupting any ongoing operations.
And yes I'm using mdraid configured as RAID1 across 2 NVMe drives (/dev/nvme0n1 and /dev/nvme1n1).
Details of the setup:
  • MD RAID arrays configured on partitions
  • LVM directly on top of the MD RAID arrays
  • Logical volumes LVM thin pool for container virtual disks
  • Container disks formated as EXT4
 
At this stage, it clearly seems to be a bug in the kernel and I find it sad that so few people complain, which indicates not many people are monitoring their hardware closely. I have not identified a way to communicate with kernel devs, I mean something from this century and not requiring to subscribe to a mailing list and figure out what's been going on in it for the last 20 years before posting in a cryptic ascii template.
 
There’s no need to worry. The reason you don’t hear such complaints is that everyone has made the wise decision not to use mdraid, which is unsupported.

Does this mean it should be fixed as an obvious bug in an unsupported feature (or that it should be made supported)?

https://pve.proxmox.com/wiki/Software_RAID
 
You are right to underline that I (and some of us) should spend time migrating from mdraid, I'll dig deeper into that.
However, that bug is a bug in the linux kernel (not specific to mdraid, it affect plain disks , loopbacks, etc) , it's even not specific to proxmox (but I discovered the issue thru it), so I'm not sure what the next step is ; the (linux) kernel devs are impossible to reach in any modern way... And I'm not sure a newbie suddenly posting in a linux kernel mailing list will be greeted with great compassion.
Or I'm not looking at the right place.
 
Last edited:
I am experiencing the same issue after upgrading to 7.0.14-11.
I am not using mdadm. This is a cluster of 6 nodes using ceph with enterprise SSDs via HBA controllers.
So PVE has direct access to disks without any type of RAID (software or hardware).
1786966135711.png

I am not seeing any difference on Ceph latency though neither within the VMs (via Ceph).
I don't run any VMs on non-ceph storage to compare against.

I am noticing similar behavior on an older cluster of 3, also with Ceph, but only on the OS NVMe drives. The SATA SSD drives used for Ceph do not show any latency spikes.
There the kernel version is 6.8.12-41.

1786966550690.png

Could it be related to the recent KVM vulnerability (CVE-2026-64561) since both kernel versions include the patch?
L

Thread 'Proxmox and CVE-2026-64561'

Hi,

is the proxmox kernel secured against

CVE-2026-64561​


?

If yes, since what version?

Edit: Nevermind about the recent patch. The latency reports are from July, well before this patch was released.
But, both clusters I mentioned had very high uptimes so they run rather old kernels before jumping to the latest one to cover the KVM vuln.
 
Last edited:
I tried contacting the linux-kernel-block maintainer but I did the unthinkable ; I sent him a short and polite email instead of subscribing to a mailing list full of experts. I'l leave it to someone else to try to have these people care about the issue. I tried.

If anyone is bold enough ;
https://subspace.kernel.org/vger.kernel.org.html
and specifically
https://lore.kernel.org/linux-block/
Good luck and welcome to the 90s.
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back