VM Crashes during backups when using qemu-guest-agent

[adoII]

Hi there,

when running backups on vms which have qemu-guest-agent enabled the vms will regularly crash. This seems to be a well known problem with qemu guest agent which is described like this:

When VM backup is invoked, Qemu agent freezes the file systems, so no single change will be made during the backup. But Qemu agent does not respect the loop* devices in freezing order (we have checked its sources), which leads to the next situation:
1) freeze loopback fs
              ---> send async reqs to loopback thread
2) freeze main fs
3) loopback thread wakes up and trying to write data to the main fs, which is still frozen, and this finally leads to the hung task and kernel crash.

I believe this is the culprit:

/dev/loop0 /tmp ext3 rw,nosuid,noexec,relatime,data=ordered 0 0
/dev/loop0 /var/tmp ext3 rw,nosuid,noexec,relatime,data=ordered 0 0

The only workaround I found is disabling qemu-guest-agent entirely. If i do this i never have a crash.

Couldn't we either fix the bug or add an option to proxmox to not use fs freeze when running backup and have guest agent enabled ?
Or are there better workarounds ?

 

[Cookiefamily]

:+1: for an option at the VM leveln in PVE to disable fsfreeze during snapshots

 

[mctaguma]

I'm seeing this too, mostly on CloudLinux 7 guests, but also some CentOS 7 guests.
It's causing significant disruption to customers, really need a fix for this.

 

[davlaw]

+1, but have to be honest, my servers were running v.0.9.x and PBS was on 1.0-1. But I have started to see more often.

So, I just updated libproxmox-backup-qemu0 on each server to match (without a full update to other proxmox components

I understand it might be best to get the whole thing updated, but was wondering if this might be something

Also noticed qemu-agent was updated on a few spins I have installed

Just FYI