IO Pressure Stall - Unexplained
- Thread starter lslamp
- Start date
I may have a regression as well, but for me it isnt constant high stall, just I have noticed much bigger spikes that started on 24 Sept 2026.
I can see from shell logs, the dist-upgrade was done on 4:33pm Sept 24. Prior to this it was still Proxmox 9.x just an earlier build.
VM's using io_uring, I will switch to native, to see if is an improvement.
I can see from shell logs, the dist-upgrade was done on 4:33pm Sept 24. Prior to this it was still Proxmox 9.x just an earlier build.
VM's using io_uring, I will switch to native, to see if is an improvement.
Hi,
I'm seeing the same symptom as described here, but on a different storage stack, and I could isolate the trigger, so this may help narrow things down.
**Setup**
- proxmox-ve 9.2.0, running kernel 6.8.12-43-pve, pve-qemu-kvm 11.0.3-4
- VM storage: ZFS mirror on 2x NVMe U.3 (KIOXIA KCD8), pool "vmpool"
- Host system on a separate SATA HW-RAID1 (sda, rotational=0, tran=sata)
- Guest: Windows 11, 2 cores, 4 GB RAM, virtio-scsi-single
**Symptom**
With iothread=1 on the disk, host IO pressure is ~95% as long as the VM is running. It is not related to boot-time IO, and it is normal before the VM is started.
All pressure is accounted to the VM's scope, none to the host's system.slice.
**Devices are idle during that time**
- No tasks in D state: 5 samples of `ps -eo state,pid,wchan:32,comm | awk '$1=="D"'` returned nothing
- dmesg: no IO errors, timeouts or resets
- The qemu iothread is in state S, wchan do_sys_io_uring_enter, i.e. idle, not blocked on disk:
**Isolation tests**
Same VM, only these disk options changed, each time a full qm stop + qm start:
iothread=1, aio=io_uring (default) -> IO pressure ~95%
iothread=1, aio=native -> IO pressure ~95%
iothread=0, aio=native -> IO pressure normal
iothread=0, aio=io_uring -> IO pressure normal
So the trigger is iothread=1; with iothread=0 the pressure disappears even with io_uring.
**What I checked / ruled out**
- Kernel treats sda as SSD, so the SATA HW-RAID is not the cause:
- `apt changelog pve-qemu-kvm` shows that the PSI fix "fix misleading IO wait and pressure stall information (PSI) accounting with io_uring-based event loop. A drive with IO thread being idle resulted in nearly 100% IO pressure." is included in the installed 11.0.3-4
- SSD emulation (ssd=1): tried, no change
- Versions:
**Why I think this may be a regression**
This matches the idle io_uring accounting issue fixed in pve-qemu-kvm 10.2.1-2 (staff reply in "IO Pressure Stall issue after upgrade to 9.1"), but with 11.0.3-4, which contains that changelog entry, an idle iothread still produces ~95% IO pressure while iothread=0 does not. So the fix may be incomplete or have regressed in QEMU 11.x. chrcoluk reports in this thread a similar jump after a dist-upgrade on 24 Sept 2026 that went away with aio=native, which could be the same root cause seen with a different workaround.
For anyone hitting this: iothread=0 is a workaround for light workloads, at the cost of IO running in QEMU's main loop.
I'm seeing the same symptom as described here, but on a different storage stack, and I could isolate the trigger, so this may help narrow things down.
**Setup**
- proxmox-ve 9.2.0, running kernel 6.8.12-43-pve, pve-qemu-kvm 11.0.3-4
- VM storage: ZFS mirror on 2x NVMe U.3 (KIOXIA KCD8), pool "vmpool"
- Host system on a separate SATA HW-RAID1 (sda, rotational=0, tran=sata)
- Guest: Windows 11, 2 cores, 4 GB RAM, virtio-scsi-single
**Symptom**
With iothread=1 on the disk, host IO pressure is ~95% as long as the VM is running. It is not related to boot-time IO, and it is normal before the VM is started.
Code:
# cat /proc/pressure/io
some avg10=95.67 avg60=89.03 avg300=45.92 total=6669195006
full avg10=95.55 avg60=88.98 avg300=45.90 total=6659775238
# cgroup breakdown
qemu.slice/101.scope : some avg10=92.92 avg60=95.92 avg300=88.38 / full ~ identical
qemu.slice : some avg10=92.32 avg60=95.80 avg300=88.28 / full ~ identical
system.slice : some/full 0.00 (all avg)All pressure is accounted to the VM's scope, none to the host's system.slice.
**Devices are idle during that time**
Code:
# iostat -x 2 3 (only nvme/sda rows; sample 1 is the average since boot)
avg-cpu: %user %nice %system %iowait %steal %idle
0.18 0.00 0.05 0.54 0.00 99.23
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 18.54 453.08 0.00 0.00 0.07 24.43 28.40 2979.50 0.00 0.00 0.06 104.89 0.96 1588.04 0.00 0.00 0.50 1661.66 0.00 0.00 0.00 2.99
nvme1n1 18.85 457.65 0.00 0.00 0.07 24.28 29.63 2979.50 0.00 0.00 0.06 100.55 0.96 1588.04 0.00 0.00 0.50 1661.66 0.00 0.00 0.00 8.09
sda 0.42 13.80 0.13 23.44 0.48 32.91 2.19 1134.44 0.78 26.29 2.56 518.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.29
avg-cpu: %user %nice %system %iowait %steal %idle
0.02 0.00 0.02 3.13 0.00 96.84
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.50 38.00 0.00 0.00 1.00 76.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05
nvme1n1 0.00 0.00 0.00 0.00 0.00 0.00 0.50 38.00 0.00 0.00 0.00 76.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 1.00 8.00 1.00 50.00 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.03 0.00 0.00 3.13 0.00 96.84
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 34.50 570.00 0.00 0.00 0.00 16.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme1n1 0.00 0.00 0.00 0.00 0.00 0.00 35.50 570.00 0.00 0.00 0.00 16.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
# pidstat -d 2 5 (Average block)
Average: UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
Average: 0 537 0.00 0.40 0.00 0 jbd2/dm-1-8
Average: 0 652 2.40 1.35 0.00 0 zvol_tq-0
Average: 0 653 6.79 55.54 0.00 0 zvol_tq-1
Average: 0 654 0.00 0.40 0.00 0 zvol_tq-2
Average: 0 655 3.20 0.25 0.00 0 zvol_tq-3
Average: 0 656 0.80 6.94 0.00 0 zvol_tq-4
Average: 0 1604 0.00 4.80 0.00 0 pmxcfs
Average: 0 152830 46.55 81.37 0.00 0 kvm
(remaining zvol_tq-* threads each < 6 kB/s)
# swapon --show
NAME TYPE SIZE USED PRIO
/dev/dm-0 partition 8G 0B -2- No tasks in D state: 5 samples of `ps -eo state,pid,wchan:32,comm | awk '$1=="D"'` returned nothing
- dmesg: no IO errors, timeouts or resets
- The qemu iothread is in state S, wchan do_sys_io_uring_enter, i.e. idle, not blocked on disk:
Code:
# ps -eLo state,tid,wchan:32,comm | grep -E 'iothread|kvm' | head -20
S 152830 do_sys_poll kvm
S 152832 do_sys_io_uring_enter kvm
S 152946 kvm_vcpu_block CPU 0/KVM
S 152947 kvm_vcpu_block CPU 1/KVM
S 152952 vhost_task_fn kvm-nx-lpage-re
S 152950 kthread_worker_fn kvm-pit/152830**Isolation tests**
Same VM, only these disk options changed, each time a full qm stop + qm start:
iothread=1, aio=io_uring (default) -> IO pressure ~95%
iothread=1, aio=native -> IO pressure ~95%
iothread=0, aio=native -> IO pressure normal
iothread=0, aio=io_uring -> IO pressure normal
So the trigger is iothread=1; with iothread=0 the pressure disappears even with io_uring.
**What I checked / ruled out**
- Kernel treats sda as SSD, so the SATA HW-RAID is not the cause:
Code:
# lsblk -d -o NAME,ROTA,TRAN,MODEL,SIZE /dev/sda
NAME ROTA TRAN MODEL SIZE
sda 0 sata H/W RAID1 447.1G
# /sys/block/sda/queue/rotational -> 0, scheduler: none [mq-deadline]
ID_BUS=ata, ID_ATA_ROTATION_RATE_RPM=0- `apt changelog pve-qemu-kvm` shows that the PSI fix "fix misleading IO wait and pressure stall information (PSI) accounting with io_uring-based event loop. A drive with IO thread being idle resulted in nearly 100% IO pressure." is included in the installed 11.0.3-4
- SSD emulation (ssd=1): tried, no change
- Versions:
Code:
# pveversion -v | grep -E 'pve-qemu-kvm|proxmox-ve'
proxmox-ve: 9.2.0 (running kernel: 6.8.12-43-pve)
pve-qemu-kvm: 11.0.3-4
# qm status 101 --verbose | grep -E 'running-qemu|running-machine'
running-machine: pc-q35-11.0+pve0
running-qemu: 11.0.3
# qm config 101 (iothread=1 state; all disks, efidisk and tpmstate on vmpool):
bios: ovmf
boot: order=scsi0;net0
cores: 2
cpu: x86-64-v2-AES
efidisk0: vmpool:vm-101-disk-0,efitype=4m,ms-cert=2023k,pre-enrolled-keys=1,size=1M
machine: pc-q35-11.0+pve2
memory: 4096
meta: creation-qemu=11.0.3,ctime=1791394204
name: w11
net0: virtio=BC:24:11:BA:12:65,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: vmpool:vm-101-disk-1,iothread=1,size=52G
scsihw: virtio-scsi-single
smbios1: uuid=8df64374-b4a7-4d49-8e74-436f93d11bba
sockets: 1
tpmstate0: vmpool:vm-101-disk-2,size=4M,version=v2.0
vmgenid: 8e361360-dac4-4a59-affc-583a11f027bb**Why I think this may be a regression**
This matches the idle io_uring accounting issue fixed in pve-qemu-kvm 10.2.1-2 (staff reply in "IO Pressure Stall issue after upgrade to 9.1"), but with 11.0.3-4, which contains that changelog entry, an idle iothread still produces ~95% IO pressure while iothread=0 does not. So the fix may be incomplete or have regressed in QEMU 11.x. chrcoluk reports in this thread a similar jump after a dist-upgrade on 24 Sept 2026 that went away with aio=native, which could be the same root cause seen with a different workaround.
For anyone hitting this: iothread=0 is a workaround for light workloads, at the cost of IO running in QEMU's main loop.
Proxmox "IO pressure stall" CAN have nothing to do with disk IO
The IO Pressure Stall graph in Proxmox VE (node and guest summary) can read close to 100% while the disk does almost nothing. When that happens, the graph is not reporting a slow disk. It is reporting a process that is sleeping in a way the kernel books as "waiting for IO".
Proxmox draws the graph from Linux Pressure Stall Information (PSI): /proc/pressure/io for the node and the cgroup's io.pressure for a guest. PSI does not count disk requests. It counts time during which tasks are marked as waiting for IO. Most of the time that is the same thing. Sometimes it isn't.
An LXC container showed io.pressure "full" at avg10 = 98.5, around the clock. The host showed about 95% because nothing else on it was doing IO. Everything else on the box was quiet:
- load average 0.13
- no task in D state (uninterruptible disk wait)
- the system disk did 7 operations in 5 seconds
- memory.events all zero, so no reclaim or swap thrash either
The whole stall sat in one service's sub-cgroup: mpd (Music Player Daemon) 0.24, observed on a 7.0 Proxmox kernel. mpd reads files through io_uring and always keeps one request open. Its IO thread sleeps in io_cqring_wait until that request completes. The kernel flags that sleep as iowait, and PSI counts every second of it as a stall. The thread is in state S, not D, so the load average stays low, but procs_blocked in /proc/stat reads 1.
Whether an io_uring completion wait counts as iowait has changed between kernel releases. A kernel update alone can therefore make this graph jump or drop with no change to the workload.
A second, spare mpd instance was started in the same container. With io_uring allowed, it opened a ring and its own cgroup's IO pressure rose from 0 to about 15% within three seconds. With io_uring blocked, it opened no ring, its pressure stayed at 0, and it still decoded an mp3 in real time.
A graph that sits at 100% forever is a graph nobody looks at. A real disk problem, such as a dying SSD or a pool scrubbing at the wrong time, would look exactly like this one and go unnoticed.
How to check your own box
1. Check that the disk really is idle. Take two samples of /proc/diskstats a few seconds apart and look at the load average. If both are busy, this article is not your problem.
2. Find the cgroup carrying the pressure. Rank the sub-cgroups by io.pressure avg10 under /sys/fs/cgroup/lxc/<ctid>/ (containers) or /sys/fs/cgroup/qemu.slice/<vmid>.scope/ (VMs).
3. List the io_uring users: ls -l /proc/*/fd 2>/dev/null | grep io_uring.
4. Check the wait channel of their threads: cat /proc/<pid>/task/*/wchan. Threads parked in io_cqring_wait while the disk is idle are the pattern described here.
5. Before blocking io_uring for a service in production, test that the service falls back cleanly, ideally on a spare instance.
mpd is just the case I hit. Any daemon that keeps an io_uring read outstanding while idle can produce the same reading.
The IO Pressure Stall graph in Proxmox VE (node and guest summary) can read close to 100% while the disk does almost nothing. When that happens, the graph is not reporting a slow disk. It is reporting a process that is sleeping in a way the kernel books as "waiting for IO".
Proxmox draws the graph from Linux Pressure Stall Information (PSI): /proc/pressure/io for the node and the cgroup's io.pressure for a guest. PSI does not count disk requests. It counts time during which tasks are marked as waiting for IO. Most of the time that is the same thing. Sometimes it isn't.
An LXC container showed io.pressure "full" at avg10 = 98.5, around the clock. The host showed about 95% because nothing else on it was doing IO. Everything else on the box was quiet:
- load average 0.13
- no task in D state (uninterruptible disk wait)
- the system disk did 7 operations in 5 seconds
- memory.events all zero, so no reclaim or swap thrash either
The whole stall sat in one service's sub-cgroup: mpd (Music Player Daemon) 0.24, observed on a 7.0 Proxmox kernel. mpd reads files through io_uring and always keeps one request open. Its IO thread sleeps in io_cqring_wait until that request completes. The kernel flags that sleep as iowait, and PSI counts every second of it as a stall. The thread is in state S, not D, so the load average stays low, but procs_blocked in /proc/stat reads 1.
Whether an io_uring completion wait counts as iowait has changed between kernel releases. A kernel update alone can therefore make this graph jump or drop with no change to the workload.
A second, spare mpd instance was started in the same container. With io_uring allowed, it opened a ring and its own cgroup's IO pressure rose from 0 to about 15% within three seconds. With io_uring blocked, it opened no ring, its pressure stayed at 0, and it still decoded an mp3 in real time.
A graph that sits at 100% forever is a graph nobody looks at. A real disk problem, such as a dying SSD or a pool scrubbing at the wrong time, would look exactly like this one and go unnoticed.
How to check your own box
1. Check that the disk really is idle. Take two samples of /proc/diskstats a few seconds apart and look at the load average. If both are busy, this article is not your problem.
2. Find the cgroup carrying the pressure. Rank the sub-cgroups by io.pressure avg10 under /sys/fs/cgroup/lxc/<ctid>/ (containers) or /sys/fs/cgroup/qemu.slice/<vmid>.scope/ (VMs).
3. List the io_uring users: ls -l /proc/*/fd 2>/dev/null | grep io_uring.
4. Check the wait channel of their threads: cat /proc/<pid>/task/*/wchan. Threads parked in io_cqring_wait while the disk is idle are the pattern described here.
5. Before blocking io_uring for a service in production, test that the service falls back cleanly, ideally on a spare instance.
mpd is just the case I hit. Any daemon that keeps an io_uring read outstanding while idle can produce the same reading.
