Intermittent I/O errors on SAN-backed VM disks (Proxmox VE 9.2)

[Matheus Alexandre]

Hello everyone,


I'm facing an intermittent issue in a Proxmox VE 9.2 cluster and I'd like some help understanding where to continue the investigation.

Environment​

• Proxmox VE 9.2
• Shared Fibre Channel SAN storage
• LVM over SAN (no local storage for VM disks)
• VM disks are stored as qcow2 images
• VirtIO SCSI controller
• Async I/O = threads
• IO Thread enabled
• Cache = Default (No cache)
• Multipath configured
• ALUA enabled on the storage array

Problem​

Approximately every Friday, one or more VMs suddenly enter an I/O Error state.


The VMs become unresponsive until they are restarted.


The issue is intermittent and difficult to reproduce manually.

What I've investigated​

I checked the kernel logs using:

journalctl -k --since "24 hours ago"

The only recurring storage-related messages I can find are:
sd 16:0:7:1: Asymmetric access state changed
sd 16:0:8:1: Asymmetric access state changed

These messages appear periodically throughout the day, not only when the issue happens.

I could not find any of the following:

• blk_update_request I/O errors
• SCSI timeouts
• Device resets
• Multipath failures
• Path failures
• Filesystem errors

multipath -ll reports all paths healthy.

No failed paths are reported during or after the incident.

Additional information​

One thing that caught my attention is that the issue seems to happen almost every Friday.


Our backup infrastructure uses Bacula, and backups are executed on Fridays, so I'm wondering if increased I/O load could somehow expose an underlying issue.


However, I have not been able to correlate this with any storage errors.

VM configuration​

Current disk configuration:

• VirtIO SCSI Single
• qcow2 image
• IO Thread enabled
• Async IO = threads
• Cache = Default (No cache)

My question​

Has anyone experienced something similar with:

• Fibre Channel SAN
• LVM over SAN
• qcow2 images
• Proxmox VE 9.x

Is there anything else I should investigate on the Proxmox side?

Could this simply be normal ALUA path optimization messages, or should I focus my investigation on the storage array despite the absence of multipath or kernel I/O errors?

Any suggestions would be greatly appreciated.

Thanks!

 

[longer]

My first thought was if there's backup running during that time, and I kept on reading until spotting it:
"Our backup infrastructure uses Bacula, and backups are executed on Fridays, so I'm wondering if increased I/O load could somehow expose an underlying issue."
But to be honest I didn't saw such issue with Bacula (I saw it with low speed storage in combination of PBS, where using backup fleecing usually helps), unless you also uses fsfreeze with Bacula. Anyway my 2 cent would be, if its possible for you to exclude some affected VM from the backup, or move it to a different day if possible and observe.

 

[bbgeek17]

I am not familiar with how Bacula integrates with PVE. If it operates through the QEMU layer, similar to PBS and Veeam, it is possible that the I/O error originates at a higher layer than FC, multipath, or LVM - that is, within the QEMU layer itself.

The suggestion to manipulate the backup schedule is a good one. Rather than excluding the affected VMs altogether (which may not be acceptable from a business continuity perspective), you could simply shift the backup window to Saturday. If the issue moves along with the backup schedule, that would be a strong indication that the two events are related.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Matheus Alexandre]

My first thought was if there's backup running during that time, and I kept on reading until spotting it:
"Our backup infrastructure uses Bacula, and backups are executed on Fridays, so I'm wondering if increased I/O load could somehow expose an underlying issue."
But to be honest I didn't saw such issue with Bacula (I saw it with low speed storage in combination of PBS, where using backup fleecing usually helps), unless you also uses fsfreeze with Bacula. Anyway my 2 cent would be, if its possible for you to exclude some affected VM from the backup, or move it to a different day if possible and observe.

Thank you for your suggestion!

That's a good point. I'll check with our backup team to see whether it's possible to move one of the affected VMs to a different backup day so we can observe whether the issue still occurs. I'll gather more information about our backup configuration as well and will update the thread once I have more details.

Thanks again for your input!

 

[Matheus Alexandre]

I am not familiar with how Bacula integrates with PVE. If it operates through the QEMU layer, similar to PBS and Veeam, it is possible that the I/O error originates at a higher layer than FC, multipath, or LVM - that is, within the QEMU layer itself.

The suggestion to manipulate the backup schedule is a good one. Rather than excluding the affected VMs altogether (which may not be acceptable from a business continuity perspective), you could simply shift the backup window to Saturday. If the issue moves along with the backup schedule, that would be a strong indication that the two events are related.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thank you for your insights!

That's a very good point regarding the possibility that the I/O error could be occurring at the QEMU layer rather than in the FC, multipath, or LVM stack.

I also agree that moving the backup window to Saturday would be a better approach than excluding the affected VMs, as it allows us to keep the backups running while testing the correlation. I'll discuss this with our backup team to see if we can temporarily adjust the schedule and gather more information about how Bacula is interacting with Proxmox in our environment.

I'll report back with the results once we've had a chance to test it.

Thanks again!

 

[bbgeek17]

That's a very good point regarding the possibility that the I/O error could be occurring at the QEMU layer rather than in the FC, multipath, or LVM stack.

Given your report that you have not found any IO related messages in "dmesg" and "journalctl" that is a pretty strong indication that the error is coming from a higher application layer.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox