Input/Output Error During Large VM Backup with PBS on Proxmox 8.2.4

[tigless]

Hello everyone,

I've been gathering a lot of information about Proxmox from this forum, and I need some help with a persistent issue I'm facing.

I'm running PBS (Proxmox Backup Server) in a virtual machine and using TrueNAS NFS mounts for snapshot backups. While smaller VMs of 50GB and 35GB back up without any issues, I'm encountering an Input/Output error when trying to back up a larger 200GB VM. The backup process fails around the 111GB mark with the following log:

INFO: 54% (108.1 GiB of 200.0 GiB) in 5m 43s, read: 384.8 MiB/s, write: 384.8 MiB/s
INFO: 55% (110.3 GiB of 200.0 GiB) in 5m 49s, read: 378.7 MiB/s, write: 378.7 MiB/s
INFO: 55% (111.9 GiB of 200.0 GiB) in 5m 54s, read: 324.8 MiB/s, write: 324.8 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - job failed with err -5 - Input/output error
INFO: Failed at 2024-07-19 11:49:34
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

To troubleshoot, I tried mounting a local disk on PBS and performing the backup there, but I encountered the same error around the same size. Has anyone else experienced similar issues with backing up VMs larger than 110GB?

Additional information:

• PBS Version: 3.2-2
• Proxmox Version: 8.2.4

Thank you for any insights or suggestions!

 

[tcabernoch]

This could mean a couple very different things. "TrueNAS NFS mounts for snapshot backups"

It could mean you've created an NFS mount in your PBS server, and then used that as a backup datastore.
I've seen testing that this is the very worst way to mount a PBS datastore. Worse by orders of magnitude.

Or, it could mean that your NAS is mapped to your cluster as storage,
and the VM has its 'data' qcows virtual disk on the NAS datastore.
Apparently this method (or direct iSCSI) give you somewhat reasonable performance.

 

[tigless]

I prefer running from local disks and performing periodic backups over using iSCSI. In a forum post, it was pointed out that a file system corruption issue was causing problems. After investigating, it seems that the issue was related to the Proxmox XFS file system corruption.

Here is the reference to the forum post: [Input/output error](https://forum.proxmox.com/threads/input-output-error.97224/).

 

[tcabernoch]

I don't entirely understand the last thing you posted, but the proxmox staff member commenting on link you provided indicated that that user had likely experienced a disk issue. (Those logs in the post are for EXT4, not XFS.) We can't draw too many conclusions, but it was likely either a bad board on the drive or bad sectors.

In your case, if you are trying to write to an NFS share that is mounted in your PBS and then set as a datastore ... testing indicates that's the very worst way to do it. Again, we can't draw too many conclusions, but it could definitely cause the i/o error you report.