Additional confirmation: guest disk EIO / MariaDB InnoDB failures with Proxmox 7.x host kernel, mitigated by booting 6.17.13-13-pve

P

PMXOlli

New Member
Jun 24, 2026
2
0
1
Hi all,

I would like to add another data point to the already reported issue regarding guest disk I/O errors with the Proxmox 7.x host kernel series.

This is related to the existing thread:

io_uring on kernel 7.0.6-2-pve / guest disk I/O errors / EIO / filesystem shutdown

In our case, the affected guest was a Debian-based VM running MariaDB 10.11.14. I am intentionally omitting VM names, internal hostnames, storage names, IP addresses, customer-specific details and internal infrastructure layout.

Summary

  • After booting the Proxmox host into the 7.x pve kernel series (also current 7.0.12-1-pve), many VMs started to show hard guest-side disk I/O errors.
  • The first visible application failure was MariaDB/InnoDB refusing to start because ib_logfile0 could not be read.
  • A full sequential read test inside the guest later completed successfully, which made the issue look transient rather than a permanently bad virtual block.
  • During a later database restore attempt, guest-side write EIOs appeared again and MariaDB/InnoDB failed again.
  • Changing the affected VM disk to aio=threads on the Proxmox side did not fully resolve the issue in our environment.
  • After pinning and booting the Proxmox host back to 6.17.13-13-pve, the EIOs have not reappeared so far.
  • The issue was not limited to a single guest OS or guest kernel. We observed the same class of errors on Debian 13 guests with both 6.x and 7.x guest kernels, as well as on Ubuntu- and Arch-based guests.
  • So far, running a 7.x kernel inside the guest did not change the behavior in a meaningful way. The stronger correlation appears to be the Proxmox host kernel 7.x series.

Guest-side symptoms

MariaDB initially failed during startup with:

Code: 
InnoDB: pread("ib_logfile0") returned -1, operating system error 5
InnoDB: Failed to read log at <offset>: I/O error
InnoDB: Log scan aborted at LSN <value>
InnoDB: Plugin initialization aborted with error Generic error
Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Unknown/unsupported storage engine: InnoDB
Aborting

The guest kernel showed disk I/O errors on the virtual disk, for example:

Code: 
I/O error, dev sda, sector <sector> op 0x0:(READ)
sd <x:x:x>: [sda] Sense Key : Aborted Command [current]
sd <x:x:x>: [sda] Add. Sense: I/O process terminated

A full sequential read of the guest disk later completed without errors:

Code: 
dd if=/dev/sda of=/dev/null bs=1M status=progress

This completed successfully, which suggested that the problem was not a stable bad block inside the guest disk image.

However, while trying to restore the database into a freshly initialized MariaDB datadir, the server crashed again and the guest kernel logged write/read EIOs:

Code: 
I/O error, dev sda, sector <sector> op 0x1:(WRITE)
sd <x:x:x>: [sda] Sense Key : Aborted Command [current]
sd <x:x:x>: [sda] Add. Sense: I/O process terminated
I/O error, dev sda, sector <sector> op 0x0:(READ)

MariaDB then again failed at InnoDB recovery/startup:

Code: 
InnoDB: Log scan aborted at LSN <value>
InnoDB: Plugin initialization aborted with error Generic error
Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Unknown/unsupported storage engine: InnoDB
Aborting

Relevant observations

  1. The failure pattern was application-visible as MariaDB/InnoDB corruption/failure, but the underlying problem appears to be below MariaDB.
  2. The guest received real block-level EIOs from its virtual disk. InnoDB was only the workload that exposed the issue quickly because database restore/startup creates a lot of synchronous write and redo-log activity.
  3. A full sequential dd read inside the guest was not sufficient to prove the system was safe. The problem reappeared under write-heavy database restore workload.
  4. Changing the VM disk to aio=threads was not sufficient in our case. The VM was fully stopped and started again after changing the disk configuration.
  5. After booting the Proxmox host back to 6.17.13-13-pve and pinning that kernel, the issue has not reappeared so far.

Sanitized VM disk configuration after mitigation attempt

Code: 
scsihw: virtio-scsi-single
scsi0: <storage>:vm-<id>-disk-0,aio=threads,cache=none,iothread=1,size=<size>

Inside the guest, MariaDB was also configured conservatively during recovery:

Code: 
[mysqld]
innodb_use_native_aio=0
max_allowed_packet=1G
net_read_timeout=600
net_write_timeout=600
wait_timeout=28800
interactive_timeout=28800

[mariadb]
max_allowed_packet=1G
binary-mode

Workaround used

Code: 
proxmox-boot-tool kernel pin 6.17.13-13-pve
proxmox-boot-tool refresh
reboot

After reboot:

Code: 
uname -r

confirmed:

Code: 
6.17.13-13-pve

Operational recommendation based on our experience

  • Do not continue database restores or filesystem repair attempts while the guest is still seeing I/O error, dev sdX from the virtual disk.
  • Treat any database datadir or filesystem that was written to during those EIO events as potentially inconsistent.
  • For stateful workloads, especially databases, consider booting/pinning the last known stable 6.17 pve kernel until this is fully understood or fixed.
  • aio=threads may reduce one possible io_uring-related path, but in our case it did not fully prevent guest EIOs while running the affected 7.x host kernel.
  • Before trusting a restored database again, perform write-heavy testing, not only sequential read testing.

A useful guest-side stress test could be something similar to:

Code: 
fio --name=stress-hdd
--ioengine=libaio
--iodepth=8
--rw=randwrite
--bs=4k
--direct=1
--fdatasync=8
--size=20G
--numjobs=4
--runtime=1800
--time_based
--ramp_time=30
--group_reporting
--filename=/tmp/fio-test
--output-format=normal,json
--output=fio-result.json

Please run this only on a disposable test VM or disposable test disk.

Current status

  • Host kernel pinned to 6.17.13-13-pve.
  • No further guest disk EIOs observed so far after rollback.
  • Database recovery/restoration is being redone from a clean datadir after the host kernel rollback.
  • The datadir created or written during the 7.x-kernel EIO events is considered unsafe and is not being reused.

Additional guest OS / guest kernel observations

We were also able to observe the same class of guest-side disk I/O errors on different Linux guests, not only on one specific distribution or guest kernel.

Affected guest environments included:

  • Debian 13 guests with 6.x guest kernels
  • Debian 13 guests with 7.x guest kernels
  • Ubuntu-based guests
  • Arch-based guests

So far, using a 7.x kernel inside the guest did not make a relevant difference in our environment. The issue still appeared to depend primarily on the Proxmox host running the affected 7.x pve kernel series.

This is why our current working assumption is that the guest kernel version may influence timing or reproducibility, but is unlikely to be the primary root cause. The stronger correlation in our environment is still:

Code: 
Proxmox host kernel 7.x
→ guest receives virtual disk EIO / Aborted Command / I/O process terminated
→ stateful workloads such as MariaDB/InnoDB fail or become inconsistent

After rolling the Proxmox host back to 6.17.13-13-pve, the issue has not reappeared so far, regardless of the tested guest distribution/kernel combination.

I hope this helps with debugging and provides another confirmation that this may not be limited to XFS guests. In our case the most visible failure was MariaDB/InnoDB on a Debian-based guest, with the guest receiving Aborted Command / I/O process terminated / I/O error from the virtual disk.

I am not claiming this is definitively a Proxmox-only bug. However, in our environment the practical mitigation was to move away from the affected 7.x pve host kernel and boot/pin 6.17.13-13-pve.
 
are you also using XFS inside the VM, or another file system?
 
Hi @PMXOlli,
what storage are you using on the host for the virtual disks? Please share the relevant part of cat /etc/pve/storage.cfg.
 
Hi,

Just a +1 from my side, forum post discovered thanks the error messages. Latest no-subscription PVE 9.2.3 with kernel, 7.0.12-1-pve, single node mode, data on an encrypted LUKS partition, VM = latest Ubuntu LTS 24.04.4, mariadb (10.11.14-0ubuntu0.24.04.1) unable to start with similar errors as @PMXOlli is having:

Code: 
Jul 01 23:38:21 host kernel: I/O error, dev sdb, sector 211980800 op 0x0:(READ) flags 0x4000 phys_seg 20 prio class 2
Jul 01 23:38:21 host kernel: sd 1:0:0:1: [sdb] tag#58 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Jul 01 23:38:21 host kernel: sd 1:0:0:1: [sdb] tag#58 Sense Key : Aborted Command [current]
Jul 01 23:38:21 host kernel: sd 1:0:0:1: [sdb] tag#58 Add. Sense: I/O process terminated
Jul 01 23:38:21 host kernel: sd 1:0:0:1: [sdb] tag#58 CDB: Read(10) 28 00 0c a2 9c 00 00 0a 00 00
Jul 01 23:38:21 host kernel: I/O error, dev sdb, sector 211983360 op 0x0:(READ) flags 0x4000 phys_seg 20 prio class 2
Jul 01 23:38:21 host mariadbd[2085]: 2026-07-01 23:38:21 0 [ERROR] InnoDB: pread("ib_logfile0") returned -1, operating system error 5
Jul 01 23:38:21 host mariadbd[2085]: 2026-07-01 23:38:21 0 [ERROR] InnoDB: Failed to read log at 185772032: I/O error
Jul 01 23:38:21 host mariadbd[2085]: 2026-07-01 23:38:21 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error

In 2 cases this temporary "workaround" worked:
Code: 
# cd /var/lib/mysql
# mv ib_logfile0 ib_logfile0_err_orig_20260702_1
# cp -Rp ib_logfile0_err_orig_20260702_1 ib_logfile0
# service mysql start

I will attempt doing a rollback to a 6.x kernel if there is no other solution.

Kind regards,
Olivier
 
omueller said:
I will attempt doing a rollback to a 6.x kernel if there is no other solution.
Click to expand...

A quick confirmation that booting to
Linux host 6.17.13-14-pve #1 SMP PREEMPT_DYNAMIC PMX 6.17.13-14 (2026-06-23T12:24Z) x86_64 GNU/Linux
apparently fixes the issue for now. (manual kernel selection in Grub on boot).

I would still be very interested to know how this could be fixed for 7.x, and what is the root cause of this issue.

Cheers, Olivier
 
Hi @omueller,
could you share more details about your LUKS setup and your VM configuration? How is the filesystem layout within the VM?
 
fiona said:
Hi @omueller,
could you share more details about your LUKS setup and your VM configuration? How is the filesystem layout within the VM?
Click to expand...
Hi @fiona, thanks for your reply !

For the LUKS setup, it is a simple setup where a encryption is mounted manually on boot, and RAW or QCOW2 disk volumes are located there.

Code: 
cryptsetup open /dev/mapper/pve-data volume1
mount /dev/mapper/volume1 /volume1/

VM config is pretty standard, but volumes have been imported from an older virt-manager host:

Code: 
agent: 1
balloon: 16384
boot: order=scsi0;ide2;net0
cores: 4
ide2: none,media=cdrom
memory: 32768
meta: creation-qemu=10.1.2,ctime=1765999311
name: host
net0: virtio=xx:xx:xx:xx:xx:xx,bridge=vmbr1,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: volume1:112/vm-112-disk-0.raw,iothread=1,size=60G
scsi1: volume1:112/vm-112-disk-1.raw,iothread=1,size=60G
scsi2: volume1:112/vm-112-disk-2.raw,iothread=1,size=300G
scsi3: volume1:112/vm-112-disk-3.raw,iothread=1,size=300G
scsihw: virtio-scsi-single
smbios1: xxx
sockets: 4
vmgenid: xxx

Please let me know if you have any question !
Kind regards, Olivier
 
> But nobody is able to determine what causes this problem.

Since various fixes have been made, I wonder if the issue was resolved because it was included in this update.

https://launchpad.net/ubuntu/+source/linux/7.0.0-28.28

- io_uring/waitid: clear waitid info before copying it to userspace
- io_uring/net: punt IORING_OP_BIND async if it needs file create
- io_uring: propagate array_index_nospec opcode into req->opcode
- io_uring/nop: pass all errors to userspace
- io_uring/napi: cap busy_poll_to 10 msec
- io_uring/zcrx: use guards for locking
- io_uring/zcrx: warn on freelist violations
- io_uring/kbuf: support min length left for incremental buffers
- io_uring/tw: serialize ctx->retry_llist with ->uring_lock

https://git.kernel.org/pub/scm/linu.../?id=737adda64f5142dd809c2b3f1d227fe5bb6cfd32

Code: 
But the ->retry_llist is updated while runing, and hence
it could potentially race between normal task_work running and the
task-has-exited shutdown path.

They forgot to use uring_lock, so the retry_llist got corrupted.

*That said, there’s no point in knowing that, though. If it actually got better, that’s all that matters.
If the issue has been resolved, I recommend marking it as solved.
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back