Issue with VM Backup Leading to Read-Only Filesystem in the VM after fsfreeze

[BenHofer]

Hello Proxmox community,

We have a recurring issue encountered with the backup process for a VM running with the guest agent installed and agent enabled in Proxmox.
During the backup operation, after the execution of the fsfreeze command, the subsequent fs-thaw command doesn't seem to be effectively executed.
This failure results in the VM's filesystem becoming read-only.
We have noticed this behavior when there is a brief connection outage between the Proxmox VE node and the Proxmox Backup Server due to a firewall or network issue. This disconnection seems to interrupt the backup process, preventing the filesystem from being properly thawed after the freeze command.

Do you have any experience or recommendations?

I've got the impression that this issue only occurs when network between PVE nodes and PBS appears to be unstable. Of course, this must be (and is) addressed in the first place. Still, I don't get the root cause of the freezed filesystem here.

I already know this forum post: https://forum.proxmox.com/threads/workaround-to-use-qemu-user-agent-without-guest-fsfreeze.120798/ We're running different system with newer kernel. Please see some logs attached.

Best
Ben

 

[VictorSTS]

Check this thread [1]. There's a but with IO threads that might be affecting you in this case. Anyway, as Proxmox don't yet use a "fleecing" backup for QEMU, conectivity issues with the backup server may cause the problems you have [2].

[1] https://forum.proxmox.com/threads/vms-hung-after-backup.137286/
[2] https://forum.proxmox.com/threads/vms-freezing-and-unreachable-when-backup-server-is-slow.96521/

 

[fiona]

Hi,

This failure results in the VM's filesystem becoming read-only.
We have noticed this behavior when there is a brief connection outage between the Proxmox VE node and the Proxmox Backup Server due to a firewall or network issue. This disconnection seems to interrupt the backup process, preventing the filesystem from being properly thawed after the freeze command.

your backup task log shows that the thaw was already completed. It is done right after starting the backup. The issue here is most likely that the filesystem cannot complete a write and therefore goes read-only because of that error. A guest write to a not yet backed-up sector needs to wait until the old data in that sector can be written to the backup target. If the connection to the backup target is gone, the guest write fails. See also https://bugzilla.proxmox.com/show_bug.cgi?id=3631

Backup fleecing would rather fail the backup than the guest write in such a scenario, but it is not yet implemented: https://bugzilla.proxmox.com/show_bug.cgi?id=4136

I suggest you do not use snapshot or suspend mode backup to PBS if you cannot ensure a reliable connection.

EDIT: For future readers stumbling across this, backup fleecing has been implemented in Proxmox VE 8.2: https://pve.proxmox.com/pve-docs/chapter-vzdump.html#_vm_backup_fleecing

 

[BenHofer]

Hi,
thank you very much for the links and the clarifications. I wasn't aware that the read-only FS can be caused by the unavailable backup server alone. The fsfreeze was the only obvious thing I saw as a potential cause.

I suggest you do not use snapshot or suspend mode backup to PBS if you cannot ensure a reliable connection.

Do you have any recommendations for alternatives? Is there an oficially supported way to take snapshots locally and ship them to the backup server after that? I can ensure that I have enough disk space for local snapshots but at this moment, I can not ensure 100% stable connectivity.

 

[VictorSTS]

Do you have any recommendations for alternatives? Is there an oficially supported way to take snapshots locally and ship them to the backup server after that? I can ensure that I have enough disk space for local snapshots but at this moment, I can not ensure 100% stable connectivity.

If you have some spare server locally to the cluster, install PBS in it and get local backups. Then on the remote PBS server, setup a sync task to copy those backups. You can have different retention policies on local and remote, so you could have say 3 backups locally and n in the remote PBS.

Other option could be to install PBS alongside PVE to get local backups and then sync to the remote PBS.