Proxmox 9.1.4 with very high iodelay

[wagnbeu0]

Hi, I am running since about 1,5 years my homelab using this hardware

Lenovo ThinkCentre M715q
AMD Ryzen 2400
64 GB RAM
1 TB SATA SSD

Until the last weeks i was absolutely happy with it, I have installed ~8 VM (Windows Server, Linux etc.) and about 25 LXC Containers.

The only bottleneck so was was at the end the CPU when tooo many VMS and containers were up and running and producing load.

Since two weeks or so I can see that the system itself is always under very heavy IODELAY. IODELY never gets under 25%, mostly it around 60% making some of my VMs and Containers unusable.

Was there a big change on the Proxmox 9.1.x line which can explain this?

 

[_gabriel]

What about the SSD exact model ?
is ZFS used ?

 

[wagnbeu0]

It is a Samsung SSD870 1 TB, no ZFS

 

[_gabriel]

Have you checked I/O per VM ?
Have you rebooted host to run the latest kernel ?
No backups running ?

 

[wagnbeu0]

IO is being created by some LXC containers, I´m analyzing here.
The current kernel is 6.8.12-13-pve, with newer kernels the system was freezing after some minutes.
No backup running.

 

[wagnbeu0]

But maybe I found the error: Shame on my head. I once created for testing purpose a "Snapshot". After removing this one the iodelay was down to 8%.

Yes, can confirm, it was this container where the snapshot was causing high iodelay. After removing the Snapshot everything is working again as expected.

 

[Impact]

Samsung SSD870

EVO or QVO? Please try the things here.
Can you share cat /etc/pve/storage.cfg and pct config CTIDHERE of the CT with the now deleted snapshot? I'm curious what kind of storage type you used. How old was the snapshot?