Increased latency on disks since 9.2.6 and associated 7.0.x kernel

W

weppa

Renowned Member
Nov 2, 2015
19
7
68
I have a couple of proxmox VE hosts, running 9.2.4 with 7.0 kernel.
These systems are updated regularly (like once a month) with proper reboot

For the last update (9.2.4 to 9.2.6 - on the 7.0 kernel) I have something weird happening that trigger alerts in MUNIN (yes i'm using munin)
The different hosts run on comptely different hardware and datacenter, and all of them display the same behavior, which never happened in the past

Screenshot_3Kz1zKCTPn.png

Basically it's like the disks are lagging, on dev/loop, WRITE IO
The screenshots are on "loop" but the graphs are the same on the real nvme (physical disks) graphs, or any other disk graph.

This behaviour can be seen on
- the host itself (screenshot)
- linux vms
- LXCs

So it propagates on all subsystems

All hosts run on MDADM SSD raid (enterprise) arrays, all systems are not loaded at all (less than n10% CPU usage) on modern and very fast hardware (EPYC)

I'll dig deeper in the hosts logs to try to pinpoint a root cause, but clearly this is specific to 9.2.6 (possible 9.2.5 whihc I skipped) and started happening on a variety of hardware the minut I rebooted the hosts.

Any advice ?
 
Last edited:
G

Post in thread 'Opt-in Linux 7.0 Kernel for Proxmox VE 9 available'

gfngfn256 said:
This actually maybe the area of concern, as I remember reading on these forums of users having issues with the latest kernels when using mdadm, which as you can see here, PVE does not support.
Click to expand...
I'm well aware of the limitations of mdraid, but at the same time it's in wide enough use that it would be pretty bad if something was broken with that in particular.

It's also possible that nothing is wrong in reality, and the statistics sources provided by the kernel are off for some reason - in particular I'm using data from /sys/block/<dev>/stat as parsed by the Munin...
C

Post in thread 'Opt-in Linux 7.0 Kernel for Proxmox VE 9 available'

Galaxis said:
Has anyone seen problems with I/O latencies on SSD/NVMEs when using the 7.0.14-6-pve kernel (from pve-no-subscription repos)?
Click to expand...

I experienced it on a 5 node cluster.
Each node used the same hardware with 3 NVME split into 3 partitions:
  • / is md raid 5
  • /vaz/lib/vz is a lvm raid 5, storage used only for backup and template
  • ceph osd, main storage
Proxmox metrics and VM performance do not seems affected but munin start to show latency peeks

Capture d'écran_20260727_100957-1.png
  • Like

Commonalities:
  • mdraid
  • Munin
 
Thanks @Neobin !

In the meantime
proxmox-boot-tool kernel pin 7.0.14-5-pve
proxmox-boot-tool refresh

UPDATE : I can confirm the problem DISAPPEARS with kernel 7.0.14-5

So it's specific to kernels after this one.
 
Last edited:
  • Like
Reactions: Galaxis and Johannes S
You must log in or register to reply here.
Top Bottom
Back