Since 6.0 backup hang vms

[mgiammarco]

Hello,
I have upgraded two proxmox ceph cluster to 6.0 and then 6.1
I had backups of type "snapshot" and when the backup was running the virtual machine backupped was slow but usable.
Now the virtual machine hangs without responding until the end of backup and then starts working again.
If the backup for some reason has a problem the virtual machine continue to stay in "not responding" state forever.
It happens only to me?
Do I need to change some options? Agent?
Thanks,
Mario

 

[spirit]

Are you sure that backup is not faster than before, and use more iops than before ?

when backup is running, do you have hang only on the currently backuped vms or also others vms on this ceph storage ?

 

[mgiammarco]

Only the backupped vm.
And if the backup does not terminate correctly but is for example aborted at 90% due to a timeout the vm stays not responding (in this case is obvious that no more i/o is present)

 

[spirit]

What is your target backup storage ? (local, remote nfs, .... ?)

When backup is running, if a new write try to overwrite a not yet backuped block, the block is first copied to backup storage, then the new write is able to overwrite the block.
If the backup storage is hanging, this will hang the new write to the vm.

when you have a timeout, does it also hang the read ?

 

[mgiammarco]

I get this error (the qemu guest agent and qemu vss are started I have just checked):
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 100 qmp command 'guest-fsfreeze-thaw' failed - got timeout

 

[spirit]

can you try to disable qemu agent feature option in the vm options ?
I would like to see if it's a bug with backup , or a bug the agent freeze feature.

 

[sg90]

Also make sure your running the latest qemu agent in the VM. Either apt / yum if you installed it from a repo.

 

[pace-IT]

Same "problem" here since version 6.0.

Sporadically VMs during the backup freeze, probably when the backup storage is too slow to respond. We use a provider shared NFS storage and unfortunately have no influence on performance. We did not have these problems with 5.4. Starting with version 6 we had to disable several backups, because our monitoring regularly generates a lot of messages during the night. I have this feeling, preferably with Linux VMs without guest agent (agent is also disabled... e.g. an OPNSense firewall cluster).

 

[mgiammarco]

Hello,
in my case I have a windows server but I am happy to know that someone else has similar problems.
I had to disable proxmox backup and now I am using eve4pve.
First I would like to clarify how snapshot backups work: In my mind a snapshot of machine is taken then the snapshot is copied to backup while the VM can continue working reading and writing on disk. Is is true? Until 5.4 it seemed so because the VM never stop responding even when backup is started. I can see that with slow hardware the backup slowdown the VM but slowdown is different from "completely not responding" as in 6.0
I see now the guest agent can invoke fs sync in VM I need to try to disable the agent but the agent is useful for other things.

 

[sander93]

Hello,

I exactly have the same problem, with vm's from i backup from local storage and from ceph storage.

Both backups going to NFS, limiting the backup speed wit bwlimit in vzdump.conf prevents this from happn but now my backups are to slow and cannot finish during night.

in 6.0 i had no problem, problem exist since update to 6.1


EDIT
Also with bwlimit VM's still hanging from time to time..

 

[spirit]

First I would like to clarify how snapshot backups work: In my mind a snapshot of machine is taken then the snapshot is copied to backup while the VM can continue working reading and writing on disk. Is is true?

No, vm backup don't use storage snapshot. (it's called snapshot, because your backup will have the state at the begin of backup time)

When the backup is running, if a new write is coming to the storage, and this block will be copied first to the backup, and overwrite.

That mean, that if you have a fast ssd storage for your production, and a mega slow nfs server, the backup job will slowdown new writes coming to not yet backuped block. (only on the first write of the block).