WARNING: Pool 'rpool' has encountered an unrecorrectable I/O failure and has been suspended. after update

[hwm.land]

I updated Proxmox and kernel was updated `6.8.12-23-pve` => `6.8.12-29-pve` After reboot machine did not start fully, it ended up with repeating message each several seconds: WARNING: Pool 'rpool' has encountered an unrecorrectable I/O failure and has been suspended. after update.
I booted again into old kernel and pinned it (so hopefully I'm save for time being)
Note: rpool is on the same NVMe disk I'm booting from.
Any idea how to investigate or (even better) sove this problem? Machine was stable before and is stable again pinning older kernel, so I don't expect real hardware problem (of course cannot be excluded, worked/working is not proof, but...)
I was planning to upgrade (again) to Proxmox 9, but now I'm afraid again. I already upgreded in the past and have to reinstall back to Proxmox 8 because of problem with passthrough PCIe card into VM (issue with Intel N150 CPU). Additionally I'm passing ASM1166 and I saw some articles that there were problems with this as wll in some kernels. Not sure what is state now.
Thanks, hwm

 

[Impact]

To take passthrough out of the equation you can try to disable IOMMU/VT-d in the node's UEFI and see if it boots without such issues then.
For basic troubleshooting check journalctl -r and zpool status -v.

 

[Fluffee]

Exact same issue for me when updating from 6.8.12-24-pve to 6.8.12-29-pve
Strangely, it works fine in my test environment on identical hardware and patch level.
Booting 6.8.12-24-pve also works fine so I've pinned this for now.

There is one one difference between my test box running 6.8.12-29-pve where the affected machine has PCI passthrough configured (two Intel I226-V NICs passed through to an OPNsense VM).

 

[hwm.land]

Hmm, this passthrough seems to be common denominator. Once I have bigger window to put it down I'll try without IOMMU just to know, but this is not solution for me. Makes me even more scarry about upgrade to Proxmox 9