Severe ZFS I/O latency and txg_sync stalls on PVE 9.2 / OpenZFS 2.4.4 – reproduced on two nodes

[gennaris]

Hi everyone,

I'm experiencing extremely high I/O latency and I/O pressure on a Proxmox VE 9.2 cluster running OpenZFS 2.4.4.

The cluster has been using ZFS for a long time and was previously running Proxmox VE 8 before being upgraded to Proxmox VE 9.

Importantly, the upgrade itself did not cause any immediately noticeable performance issues. The cluster continued operating normally after the upgrade, and we had never experienced I/O slowdowns of this magnitude before.

The problem has appeared more recently, without an identified triggering change, and has now been observed on two different nodes after migrating the same Windows 11 VM.

Both nodes are currently running kernel 7.0.14-20-pve and OpenZFS 2.4.4-pve1.

I'm trying to determine whether this behaviour could be related to a kernel/OpenZFS regression, a specific interaction between QEMU and ZFS, or an underlying storage performance issue.

Environment:

• Proxmox VE: 9.2.0
• pve-manager: 9.2.21
• Running kernel: 7.0.14-20-pve
• OpenZFS: 2.4.4-pve1
• QEMU: 11.0.3
• Storage: local ZFS pool, single SATA SSD on the currently affected node
• Guest: Windows 11, 6 vCPUs, 16 GB RAM
• Virtual disk: 256 GB ZFS zvol, VirtIO SCSI Single, IOThread enabled
• ZFS volblocksize: 16K
• ZFS sync: standard
• ZFS compression: on
• ZFS logbias: latency
• ZFS autotrim: off

The problem was initially observed on another cluster node. After migrating the VM, similar symptoms appeared on the second node, despite the change of physical host.

During a recent Windows Update installation, the VM became extremely slow, remaining at 0% for a long time and progressing only a few percentage points over approximately an hour.

However, the VM was not completely frozen. QEMU continued processing read/write requests, and ZFS transactions continued completing, although sometimes with extremely high latency.

Windows Update made the problem particularly noticeable, but similar ZFS synchronization stalls had already been recorded in the kernel logs the previous day, before this update operation.

The most concerning symptoms are:

• Linux I/O pressure (full avg10) reaching approximately 75–80%.
• Very high ZFS write latency, occasionally reaching 25 seconds of total wait time.
• ZFS transaction synchronization (txg_sync) occasionally taking over 100 seconds.
• Kernel reporting txg_sync blocked for more than 122 seconds.
• Very low disk throughput despite extremely high I/O latency.

The following kernel messages were repeatedly recorded:

INFO: task txg_sync blocked for more than 122 seconds

io_schedule_timeout
__cv_timedwait_io
zio_wait
dsl_pool_sync
spa_sync
txg_sync_thread
Some measurements obtained using zpool iostat -l -v:

write total_wait: up to 25s
write disk_wait: approximately 437–949ms
write syncq_wait: up to 25s
write asyncq_wait: up to 25s
During a separate three-minute monitoring session, the following values were observed:

I/O pressure full avg10 peak: 75.63%
ZFS write total_wait: up to 6s
ZFS write disk_wait: up to 933ms
ZFS async queue wait: up to 6s
ZFS sync queue wait: up to 2s
These measurements suggest significant delays both at the block-device level and within ZFS queues.

Some examples from /proc/spl/kstat/zfs/rpool/txgs, using the stime field:

TXG 9066036: approximately 109.9s
TXG 9066053: approximately 61.7s
TXG 9066070: approximately 116.6s
Transactions eventually complete, but their synchronization time sometimes becomes extremely high.

During a roughly three-minute monitoring session, the Windows VM completed approximately:

Reads: 240 MiB
Writes: 638 MiB
Flushes: 114
Therefore, QEMU continues completing disk operations, even while the guest appears almost unresponsive.

The affected ZFS pool reports:

state: ONLINE
READ: 0
WRITE: 0
CKSUM: 0
errors: No known data errors
SMART reports PASSED, with no reallocated sectors, pending sectors, or logged errors.

However, block-device statistics also show elevated write completion latency, so I cannot rule out an underlying SSD performance limitation.

ZFS ARC statistics show:

ARC size: approximately 3.17 GB
ARC maximum: approximately 3.30 GB
memory_throttle_count: 0
arc_need_free: 0
There is no apparent memory pressure or swapping.

What makes this particularly interesting is that the same VM has exhibited similar symptoms on two different cluster nodes running the same kernel and OpenZFS versions.

ZFS has been used successfully in this environment for a long time, including after the upgrade from Proxmox VE 8 to 9. These extreme I/O latency problems are a recent occurrence, and I have not yet identified a specific change that could explain them.

I would appreciate some guidance on the following questions:

1. Are there any known I/O performance regressions involving kernel 7.0.14-20-pve or OpenZFS 2.4.4-pve1?
2. Could the extremely long txg_sync times indicate an issue in ZFS transaction synchronization, or are they more likely a consequence of slow underlying storage?
3. Are there specific ZFS tracing tools or kernel statistics that would help identify why requests remain queued for several seconds?
4. Has anyone experienced similar behaviour with Windows 11 guests using VirtIO SCSI and ZFS zvols?
5. Would testing kernel 7.0.14-24-pve be a useful first troubleshooting step?

I have not modified the ZFS sync settings or performed performance tuning, because the affected VM was still processing Windows updates and I wanted to avoid risking data integrity.

I have collected additional performance metrics, including I/O pressure, ZFS latency statistics, QEMU block statistics, disk-level counters and ZFS transaction data, which I can share in anonymized form if useful.

Any suggestions on how to further investigate or isolate this issue would be greatly appreciated.

Thank you in advance for your help!

 

[chrcoluk]

To rule out kernel, I think an older kernel is available still in repo. You can also downgrade QEMU machine version on the VM. Usually proxmox fixes this on Windows VMs though.

You seem to be reporting an extreme change of write performance of async writes if I understand right, it might be worth examining openzfs change logs, to see if any tunable defaults have changed. ZFS has a separate dirty cache for writes, the ARC is just for reads.