High IO-delay with latest kernel.

[birdy]

I always had the issue of having a high IO-delay for some time after a reboot of my PVE-server. But after a certain time, could be up to 24 hours, things calmed down and IO-delay was ok (avg < .5).

So when I updated (No-Subscr.) to the latest kernel (5.15.39.1) and rebooted I saw the same but thought this would settle down after the expected max 24 hours. But the IO-delay kept high and Zabbix kept nagging me about "Disk read/write request responses" regarding the HDDs (sda/b/c/d > HDD2TB). So after a few days of investigating I decided to switch back to an older kernel (5.15.35.3) and now after a couple of hours things calmed down and I'm back "normal" IO-delays.

Anyone an idea what is causing this?

 

[leesteken]

Are there many more messages per hour in the Syslog (journaltcl) when running the newer kernel maybe?

EDIT: Sometimes a kernel version is more "chatty" with debug info or warnings or there are suddenly many (repetitions of) non-fatal errors reported about PCI passthrough for example.

 

[birdy]

You mean that the drives would be more ¨stressed"?

I doubt that, because PVE is installed (rpool) on the SSDs (ZFS RAID1), the HDDs (ZFS RAID10) are used for data drives for OMV en Zoneminder.

 

[birdy]

Installed the latest updates (including kernel Linux 5.15.39-2-pve) and rebooted and after 8 hours things calmed down and IO delay is back to "normal".

So ok for now but it is still a mystery to me what is happening here.