IO Pressure Stall - Unexplained

Hi,
you probably just need to migrate/start the VMs fresh with the updated QEMU version:
Code: 
pve-qemu-kvm (10.2.1-2) trixie; urgency=medium
...
  * fix misleading IO wait and pressure stall information (PSI) accounting
    with io_uring-based event loop. A drive with IO thread being idle resulted
    in nearly 100% IO pressure. This is resolved by telling io_uring to not
    count blocking waits towards the IO wait metric when the event loop is
    effectively idle.
 
  • Like
Reactions: ucholak
@fiona
are you saying to downgrade from pve-qemu-kvm (11.0.2-1) to pve-qemu-kvm (10.2.1-2) trixie

Sorry, also when researching how to install only pve-qemu-kvm (10.2.1-2) trixie I see that it mentions that I should also place the package on hold.

apt reinstall pve-qemu-kvm=10.2.1-2
apt-mark hold pve-qemu-kvm

what is the purpose of this.

Lawrence
 
fiona said:
you probably just need to migrate/start the VMs fresh with the updated QEMU version
Click to expand...
Based on the OP, that is unlikely as he recently fresh-booted VM 107 & he has pve-qemu-kvm: 11.0.2-1.

Code: 
--- [1. HOST OS & KERNEL VERSION] ---
proxmox-ve: 9.2.0 (running kernel: 6.8.12-18-pve)
pve-manager: 9.2.4 (running version: 9.2.4/5e5ae681198514d4)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.14-3
proxmox-kernel-7.0.14-3-pve-signed: 7.0.14-3
proxmox-kernel-7.0.12-1-pve-signed: 7.0.12-1
proxmox-kernel-6.8: 6.8.12-29
proxmox-kernel-6.8.12-29-pve-signed: 6.8.12-29
proxmox-kernel-6.8.12-18-pve-signed: 6.8.12-18
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
Is there a reason you are still running kernel: 6.8.12-18-pve, or have you just never rebooted the PVE host?
 
lslamp said:
@fiona
are you saying to downgrade from pve-qemu-kvm (11.0.2-1) to pve-qemu-kvm (10.2.1-2) trixie
Click to expand...
No, I say the issue should be fixed for VMs started with QEMU package versions >= 10.2.1-2.

You can check the running version for the VM with qm status 107 --verbose | grep running-qemu
 
  • Like
Reactions: lslamp
After installing the previous version of pve-qemu-kvm=10.2.1-2 all the IO Pressure Stall alerts went away.

Thanks for the help.
Lawrence
 
lslamp said:
After installing the previous version of pve-qemu-kvm=10.2.1-2 all the IO Pressure Stall alerts went away.
Click to expand...
The fix is also present in the latest version. It is present since version 10.2.1-2. You probably still had guests with an old binary running.
 
Hi @fiona,

In our environment of a few physical hosts recently upgraded to 9.2 we also bumped into that bug. And I cannot confirm that any versions pve-qemu-kvm>=10.2.1-2 made any difference. I used straight forward logic that
Code: 
io=io_uring
triggers IO Pressure Stall load/bug. Then we switched to
Code: 
io=native
and rebooted particular VM to apply hardware changes and the problem went away like by using magic wand. From ~100% stall to near 0% as before upgrade. This is acceptable workaround for now.

Code: 
pve-qemu-kvm (10.2.1-2) trixie; urgency=medium
...
  * fix misleading IO wait and pressure stall information (PSI) accounting
    with io_uring-based event loop. A drive with IO thread being idle resulted
    in nearly 100% IO pressure. This is resolved by telling io_uring to not
    count blocking waits towards the IO wait metric when the event loop is
    effectively idle.

So at least in our env: VM's disks on NFS storage backend: its NOT solved with that version. Is there any exact commit that fixes it? Or some more detailed technical explanation regarding this problem? When is it triggered?

Some of our VMs were more affected (30-40% stall vs 100% stall) than others. Couldn't find a pattern what kind of operation triggers really high IO Pressure Stall.
 
Do you experience actual performance issues? How big is the latency for NFS? How much IO did the less affected and more affected VMs actually perform?
 
VMs that got around 30-40% stall with io=io_uring were noticeably slower but still somewhat responsive, a few of them that had stall value almost 100% all the time were completely unresponsive. Even reboot commands timeouted or any other operations like VNC/Spice console etc. Graph in Proxmox for one of them (Windows 11) looked like this:
Selection_056.png
The other (Linux) just was missing values in graph in proxmox so null values basically. But for that linux host I have also i/o wait graph from Grafana from that time (Telegraf there collecting data):
Selection_057.png

This is all I have here. The graphs returned to "our norm <5ms" when we changed to io=native and VM reboot to apply changes.
 
Last edited:
This is some VM (Linux with artifactory instance) that still has io=io_uring and it looks like that in Proxmox dashboard:
Selection_058.png
It's slower to respond but responding. I am now switching to io=native for that one as well.

As you may notice before 07-08-2026 (so before update) IO Pressure Stall values were near 0% as expected.
This is on: all nodes on up-to-date PVE 9.2.

EDIT: The only noticeable value from the guest point of view with io=io_uring is very high WA (wa: io wait cpu time (or) % CPU time spent in wait (on disk)) that is very high. So the guest system waits for storage somehow. Values range 30.0+ here.
 
Last edited:
Do you have IO thread enabled for your VM disks? Can you share one or two configurations of affected VMs as well as your /etc/pve/storage.cfg? Which kernel version are you running on the host?
 
One affected is really old legacy system with centos kernel 3.10.x series, the other Debian 12 ( 6.1.0-52-amd64) and some Debian's 13 (with 6.12.95+deb13-cloud-amd64) some Ubuntus.

That's overall for VMs kernels. Proxmox nodes are all at 7.0.14-9-pve

All VMs have iothread enabled.

Here is snippet of /etc/pve/storage.cfg:
Code: 
nfs: proxmox_ssd_shared_vol
        export /proxmox_ssd_shared_vol
        path /mnt/pve/proxmox_ssd_shared_vol
        server %IP_ADDRESS_HERE%
        content vztmpl,import,iso,rootdir,backup,images,snippets
        options noatime,noacl,sync,vers=4.1
And many such a volumes mixed hdd/ssd backends over NFS. I hope it helps somehow. This I may share freely.
 
Could you provide the full VM configs with qm config ID so we can see all the settings? Thanks!

Maybe it's related to the sync mount option? From man 5 nfs:
If the sync option is specified on a mount point, any system call that writes data to files on that mount point causes that data to be flushed to the server before the system call returns control to user space. This provides greater data cache coherence among clients, but at a significant performance cost.
Click to expand...
 
How come then storage was performing reliably with same options before an upgrade? And there is presumably related bug "fixed"?
I understand that You may want to see bigger picture. But it's clearly regression/corner case introduced by PVE 9.2.

Example affected VM config snippet if you want to try to reproduce it in lab env:
Code: 
agent: 1,fstrim_cloned_disks=1
bios: seabios
boot: order=scsi0;scsi1
cores: 1
cpu: x86-64-v2-AES
description: Some description
machine: q35,viommu=virtio
memory: 8192
meta: creation-qemu=10.1.2,ctime=1771347817
name: vm-name.int.domain.tld
net0: virtio=BC:24:11:XX:XX:XX,bridge=VLAN_XX,mtu=1500
ostype: l26
rng0: source=/dev/urandom
scsi0: proxmox_ssd_shared_vol:200/vm-200-disk-0.qcow2,discard=on,iothread=1,size=30G,ssd=1
scsi1: proxmox_hdd_shared_vol:200/vm-200-disk-1.qcow2,discard=on,iothread=1,size=1500G
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=4238584c-XXXX-ca09-000d-XXXXXXXXXXXX
sockets: 6
startup: order=3
vga: qxl
vmgenid: ee18162f-4cac-XXXX-88d3-XXXXXXXXXXXX
 
Last edited:
ow-vtech said:
How come then storage was performing reliably with same options before an upgrade? And there is presumably related bug "fixed"?
I understand that You may want to see bigger picture. But it's clearly regression/corner case introduced by PVE 9.2.
Click to expand...
The other issue has been fixed and the fix was confirmed by people who were reporting it. If you got an issue with the packages that include that fix, then it's a different issue.

I'd still try without sync to see if it makes a difference. You need to make sure no guests requiring the storage are running and remount the storage to apply the change.

Did anything on the NFS server side change around the time of the upgrade too or is the configuration there still identical. You can try booting into an older kernel like 6.17 or 6.8 to see if the issue is a kernel regression. You can try downgrading pve-qemu-kvm to 10.0.2-4 to see if the issue was already present in the original release of Proxmox VE 9.
 
fiona said:
I'd still try without sync to see if it makes a difference. You need to make sure no guests requiring the storage are running and remount the storage to apply the change.

Did anything on the NFS server side change around the time of the upgrade too or is the configuration there still identical. You can try booting into an older kernel like 6.17 or 6.8 to see if the issue is a kernel regression. You can try downgrading pve-qemu-kvm to 10.0.2-4 to see if the issue was already present in the original release of Proxmox VE 9.
Click to expand...
Hi, maybe you will get angry but I will refer just to last part of your post and ignore NFS re-config section. Because NFS config side never changed since initial cluster and storage setup so yes it is identical. And confirmed working for year at least. Older kernel before upgrade was 7.0.4 as I remember correctly so was already from 7.x series and no issues then. I will try to empty one node to perform the double-kernel-regression-check. But in reality this kernel version bump was really minor. I understand that it's my homework to do, to rule our scope of other problems rather than claiming its due to upgrade. But all graphs in Proxmox Dashboards started drawing red values for IO Pressure Stall and other graphs "interruptions/anomalies" just after update day for many VMs.

As for the down-grade to 10.0.2-4 this was my first try. I kinda misunderstood this topic while readingo_O. And thought that down-grade is an actual workaround. So this I've tried that first on one node. And with pve-qemu-kvm=10.0.2-4 the issue is there (I made sure to have restarted VM so it runs on down-graded binary of kvm)

Then it must be closely related but separate new issue here. I do not know internals of Proxmox that much yet to debug any further at this point.
 
ow-vtech said:
Hi, maybe you will get angry but I will refer just to last part of your post and ignore NFS re-config section. Because NFS config side never changed since initial cluster and storage setup so yes it is identical. And confirmed working for year at least.
Click to expand...
I won't get angry, I just think that it would still be extra information to know if that is related. I imagine most people with NFS have not set the mount option, and most people with NFS are not reporting issues.

ow-vtech said:
As for the down-grade to 10.0.2-4 this was my first try. I kinda misunderstood this topic while readingo_O. And thought that down-grade is an actual workaround. So this I've tried that first on one node. And with pve-qemu-kvm=10.0.2-4 the issue is there (I made sure to have restarted VM so it runs on down-graded binary of kvm)
Click to expand...
For the original issue, it was a valid workaround.
 
  • Like
Reactions: Johannes S and gfngfn256
You must log in or register to reply here.
Top Bottom
Back