io_uring on kernel 7.0.6-2-pve (PVE 9.2.3): guest disk I/O errors (EIO) + filesystem XFS shutdown

could be an issue on either end, or in the combination of both. please keep us posted if any new information comes up!
 
By the way, I also reproduced the issue on AlmaLinux 10.1 (kernel 6.12.0-124.52.1.el10_1.x86_64) with this command :
Bash: 
fio --name=stress-hdd --ioengine=libaio --iodepth=8 --rw=randwrite \
    --bs=4k --direct=1 --fdatasync=8 --size=20G --numjobs=4 \
    --runtime=1800 --time_based --ramp_time=30 --group_reporting \
    --filename=/tmp/fio-test --output-format=normal,json \
    --output=fio-result.json

However, it seems to last longer / be more complicated to reproduce on this kernel version.

Is anyone on the Proxmox team able to reproduce the problem with a configuration similar to mine?

EDIT

I would like to add that impacted VMs have one less tag on their disk than unaffected VMs :
1781804954945.png

Healthy VMs (until now) :
1781804984402.png

The format=raw may impact something ? I don't know what is the difference.
 
Last edited:
Some news :

I was able to reboot one of the both Proxmox host on Kernel 6.17.13-13-pve and retry the same process / same stress write test (30 minutes of stress).

I confirm that there is no issue on this kernel version.

So it seems to be a regression on the last Proxmox Linux kernel version (7.X or 7.0.6-2-pve).

It would be great if someone from Proxmox staff could try to reproduce the problem or give me more commands / info on debugging.
 
  • Like
Reactions: waltar
This issue does not affect the storage migration (VM disks) between 2 nodes that not shared the same storage.
 
The problem is also triggered on last kernel 7.0.12-1-pve.

Bash: 
[root@patchmon ~]# fio --name=stress-hdd --ioengine=libaio --iodepth=8 --rw=randwrite     --bs=4k --direct=1 --fdatasync=8 --size=20G --numjobs=8
--runtime=1800 --time_based --ramp_time=30 --group_reporting     --filename=/tmp/fio-test --output-format=normal,json     --output=fio-result.json
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=3303440384, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=15728369664, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=12987797504, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=4300693504, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=3548835840, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=781250560, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=1649344512, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=4513239040, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=19759464448, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=3274137600, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=17922351104, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=19376713728, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=93745152, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=14806507520, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=19999735808, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=11025801216, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=14361858048, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=16999051264, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=20181307392, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=16825335808, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=15835607040, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=11907469312, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=20892581888, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=13690458112, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=10695925760, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=10553192448, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=20011687936, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=20895113216, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=17017688064, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=13688827904, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=15695372288, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=13781270528, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=20881649664, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=13658992640, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=13343629312, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=14113169408, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=14118154240, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=14968557568, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=13344370688, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=20108697600, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=18358050816, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=10622705664, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=565350400, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=9192083456, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=18820165632, buflen=4096
[root@patchmon ~]#

It takes more testing time to trigger it.
 
Last edited:
Answer from AlmaLinux support :

Hello,

Thank you for the additional updates.

Based on the latest test results, this looks more like a regression in the Proxmox 7.x host kernel rather than an AlmaLinux 10.2 guest-only issue.

The key point is that the same VM and the same fio stress-write workload did not reproduce the issue when the Proxmox host was booted with 6.17.13-13-pve, while the issue appears with 7.0.x-pve kernels such as 7.0.6-2-pve and 7.0.12-1-pve.

Also, since the problem can now be reproduced with AlmaLinux 10.1 as well, AlmaLinux 10.2 no longer seems to be the only trigger.

From the guest side, XFS appears to be shutting down as a consequence of receiving EIO / Aborted Command from the virtual block device.

So I think it would be useful to focus the debugging on the Proxmox host kernel / QEMU / virtio-scsi / LVM-thin stack.
Click to expand...
 
thanks for the additional information! let's continue over in the BZ entry!
 
By the way, the issue is still present in Linux 7.0.14-2-pve. So my first guess was wrong.

Bash: 
[699923.551209] sd 0:0:0:0: [sdb] tag#142 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[699923.551386] sd 0:0:0:0: [sdb] tag#142 Sense Key : Aborted Command [current]
[699923.551486] sd 0:0:0:0: [sdb] tag#142 Add. Sense: I/O process terminated
[699923.551568] sd 0:0:0:0: [sdb] tag#142 CDB: Write(10) 2a 00 06 80 04 38 00 00 08 00
[699923.551650] I/O error, dev sdb, sector 109052984 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 0
[699923.551731] I/O error, dev sdb, sector 109052984 op 0x1:(WRITE) flags 0x29800 phys_seg 1 prio class 0
[699923.551822] XFS (sdb3): log I/O error -5
[699923.551901] XFS (sdb3): Filesystem has been shut down due to log error (0x2).
[699923.552014] XFS (sdb3): Please unmount the filesystem and rectify the problem(s).
[699923.552670] systemd-journald[563]: /var/log/journal/.../system.journal: I/O error, rotating.
[699923.552896] systemd-journald[563]: Failed to rotate .../system.journal: Input/output error
[699923.553077] systemd-journald[563]: Failed to open ...: Input/output error
[699923.553205] systemd-journald[563]: Failed to vacuum ... ignoring: Input/output error
[699923.553301] systemd-journald[563]: Failed to write entry to .../system.journal (13 items, 418 bytes) despite vacuuming, ignoring: Input/output error
[699923.553501] systemd-journald[563]: Failed to rotate .../system.journal: Input/output error
[699923.553603] systemd-journald[563]: Failed to open ...: Input/output error
[699923.553703] systemd-journald[563]: Failed to vacuum ... ignoring: Input/output error
[699923.553803] systemd-journald[563]: .../system.journal: I/O error, rotating.
[699923.553908] systemd-journald[563]: Suppressing rotation, as we already rotated immediately before write attempt. Giving up.
[699923.554019] systemd-journald[563]: Failed to rotate .../system.journal: Input/output error
[699923.554285] systemd-journald[563]: Failed to open ...: Input/output error
[699923.554705] systemd-journald[563]: Failed to vacuum ... ignoring: Input/output error
[699923.555446] systemd-journald[563]: .../system.journal: I/O error, rotating.
[699923.555193] systemd-journald[563]: Suppressing rotation, as we already rotated immediately before write attempt. Giving up.
[699923.555434] systemd-journald[563]: Suppressing rotation, as we already rotated immediately before write attempt. Giving up.
[699924.221291] coredump: 950(zabbix_agent2): [/usr/lib/systemd/systemd-coredump pipe] failed
[699924.230163] coredump: 1(systemd): [/usr/lib/systemd/systemd-coredump pipe] failed
 
Does anyone able to reproduce this issue ?

My setup is :
  • NVME or HD
  • LUKS
  • MDADM RAID 1
  • LVM-THIN for storage type
  • VM AlmaLinux 10 with XFS
 
We hit the same signature across VMs with different guest kernels and across different nodes of 1 cluster:
- Ubuntu 22.04.5, kernel 5.15.0-181 (XFS data disks fail; ext4 root survives)
- RHEL 8.10, kernel 4.18.0-553 (XFS)

Host:
- Proxmox VE 9.1, kernel 7.0.2-6-pve, pve-qemu-kvm 11.0.0-2
- Storage: dir on ext4 -> LUKS2 -> PERC RAID5 (no BBU) -> INTEL SSDSC2KG019T8
- Disk: scsihw virtio-scsi-single, aio=threads, cache=none, iothread=1
- Host is completely clean at the event - nothing in dmesg / journalctl -k
- SMART clean. Perccli reports no suspicious events.

Workload on vm's: Heavy sustained random-write. Rare/random, only under load.

Symptom: guest EIO, FS goes read-only; recovered by a remount (unless it's the root partition).
On the affected drives QMP query-blockstats shows failed_wr_operations > 0, while the host block layer logs nothing - so we can't find any backend error; the failed_wr looks like in-flight writes cancelled by a guest-initiated virtio-scsi reset.

We're on aio=threads and it does NOT prevent it. Based on this thread we're about to pin a 6.8.x kernel to test whether it resolves it here, and will report back.

Shared with the other reports: kernel 7.0.x, LUKS-encrypted storage, XFS guest, high random-write.
Different here: multiple guest kernels (5.15 and 4.18) and plain local directory/qcow2 (not LVM).
 
  • Like
Reactions: hidigoudi
static302 said:
We hit the same signature across VMs with different guest kernels and across different nodes of 1 cluster:
- Ubuntu 22.04.5, kernel 5.15.0-181 (XFS data disks fail; ext4 root survives)
- RHEL 8.10, kernel 4.18.0-553 (XFS)

Host:
- Proxmox VE 9.1, kernel 7.0.2-6-pve, pve-qemu-kvm 11.0.0-2
- Storage: dir on ext4 -> LUKS2 -> PERC RAID5 (no BBU) -> INTEL SSDSC2KG019T8
- Disk: scsihw virtio-scsi-single, aio=threads, cache=none, iothread=1
- Host is completely clean at the event - nothing in dmesg / journalctl -k
- SMART clean. Perccli reports no suspicious events.

Workload on vm's: Heavy sustained random-write. Rare/random, only under load.

Symptom: guest EIO, FS goes read-only; recovered by a remount (unless it's the root partition).
On the affected drives QMP query-blockstats shows failed_wr_operations > 0, while the host block layer logs nothing - so we can't find any backend error; the failed_wr looks like in-flight writes cancelled by a guest-initiated virtio-scsi reset.

We're on aio=threads and it does NOT prevent it. Based on this thread we're about to pin a 6.8.x kernel to test whether it resolves it here, and will report back.

Shared with the other reports: kernel 7.0.x, LUKS-encrypted storage, XFS guest, high random-write.
Different here: multiple guest kernels (5.15 and 4.18) and plain local directory/qcow2 (not LVM).
Click to expand...
Amazing!

Please let me know if you're still experiencing the issue on a 6.x kernel. I think the problem is related to newer Linux kernels, specifically versions greater than 7.
 
static302 said:
We hit the same signature across VMs with different guest kernels and across different nodes of 1 cluster:
- Ubuntu 22.04.5, kernel 5.15.0-181 (XFS data disks fail; ext4 root survives)
- RHEL 8.10, kernel 4.18.0-553 (XFS)

Host:
- Proxmox VE 9.1, kernel 7.0.2-6-pve, pve-qemu-kvm 11.0.0-2
- Storage: dir on ext4 -> LUKS2 -> PERC RAID5 (no BBU) -> INTEL SSDSC2KG019T8
- Disk: scsihw virtio-scsi-single, aio=threads, cache=none, iothread=1
- Host is completely clean at the event - nothing in dmesg / journalctl -k
- SMART clean. Perccli reports no suspicious events.

Workload on vm's: Heavy sustained random-write. Rare/random, only under load.

Symptom: guest EIO, FS goes read-only; recovered by a remount (unless it's the root partition).
On the affected drives QMP query-blockstats shows failed_wr_operations > 0, while the host block layer logs nothing - so we can't find any backend error; the failed_wr looks like in-flight writes cancelled by a guest-initiated virtio-scsi reset.

We're on aio=threads and it does NOT prevent it. Based on this thread we're about to pin a 6.8.x kernel to test whether it resolves it here, and will report back.

Shared with the other reports: kernel 7.0.x, LUKS-encrypted storage, XFS guest, high random-write.
Different here: multiple guest kernels (5.15 and 4.18) and plain local directory/qcow2 (not LVM).
Click to expand...
Any news about this ?
 
You must log in or register to reply here.
Top Bottom
Back