Almost 0 disk IO during backup

[kenneth_vkd]

Hi
We have experienced an issue for some time now where a VM is running extremely slow and access to disk IO is almost impossible during the backup.
Usually it is not a problem as backups to PBS finish within a few hours. It is however a problem when backups take longer than usual and run ov er into business hours.
Affected PVE hosts run 100% nVME local storage in a mix of ZFS mirrors and Ceph.
When the PBS server performs as normally, we run with about 80-120MB/s on average, but due to the backup server being migrated to a different host, it runs much slower and runs about 5MB/s on average. During backup of a VM it is almost impossible to perform any operations that require disk IO. To fix it we have to either stop the backup job or wait for it finish on that VM, after which everything returns back to normal.
Coming from a vmware ESXi + vCenter + Veeam solution, we know that some performance degradation is to be expected, but a VM should not appear frozen for multiple hours.
Is this expected behavior or is there a way to prevent VMs from "locking up" during backups?

 

[dietmar]

but due to the backup server being migrated to a different host, it runs much slower and runs about 5MB/s on average.

Sorry, but why is the backup server that slow? This is simply too slow for production use.

 

[kenneth_vkd]

The data on the PBS host is currently being migrated to another host with faster and larger disks. The sync job eats disk IO as if it was candy, so data is being transferred as fast as possible, but since backups still have to run to the "old" PBS host. Once migration is done, performance will be back to normal levels. But the question is still relevant as a number of things could happen, such as increased latency on the network, a disk failing, which requires a resilver of ZFS etc.
But we have previously observed similar case of VMs "locking up" when running just vzdump, which was the way prior to PBS existing as a product

 

[dietmar]

But we have previously observed similar case of VMs "locking up" when running just vzdump, which was the way prior to PBS existing as a product

Yes, a slow backup connection can badly influence the VM (like a slow storage).

 

[kenneth_vkd]

So although the storage on the PVE host is fast enough that it should not starve the VM of IO, it will seem so until the backup is done?

Is there any way around it?

 

[spirit]

The problem is that, when backup is running, if a new write is coming on a block not yet backuped, it must be writen first to the backup storage, before being overwriten.

So, this is really depend of the workload.

if you have a lot of new write on the same block, it's not a problem, you just need to wait that the first time the block is backuped.
if you have a lot of new write everywhere on the disk, you'll need to wait for each block to be backuped.

(and not sure about parallelism impact too)

so , if you really need big speed on the vm (nvme...), and you have a pretty slow backup storage (7200k), and writes everywhere during the backup, yes, you can have big slow down.