Backup Server too slow, IO on VM stalled during Backup

[adoII]

Hi,
I have a setup at a Hoster where the Proxmox VM Server is damned fast using NVMEs and the Proxmox Backup Server is terrible slow, for example network Bandwidth sometimes is only 30 MB/s

What happens now: The Backup ov VM starts really fast with let's say 500 MB/s reading speed.
The Proxmox Backup Server and/or the Network can only write at around 30MB/s
As soon as some Cache gets full all IO gets halted on the VM that is backed up and the VM is not usable anymore

What can I do about it ? My idea is to limit the reading speed for backups to a pbs server to something like 30MB/s, would that be a good workaround ?

 

[Dunuin]

First I would try to find out why it is so slow.

You could use iperf3 to test the performance of the network connection between the PVE and PBS.
Then you could run a PBS benchmark: proxmox-backup-client benchmark --repository yourbackupstore
And you can use fio to benchmark the read performance of the PVE host and write/IOPS performance of the PBS server.

PBS needs a good IOPS + random IO performance, so best you use only SSDs or atleast HDDs + SSDs for caching / to store the metadata.
A suddenly terrible performance drop indeed sounds like a cache thats gets full. Make sure that server isn't using SMR HDDs or QLC SSDs as there this drop will be especially horrible.

 

[adoII]

Well, in the Hoster environment the main Problem is the shared 1 GBit/s uplink to the proxmox server which is used by several vms+Backup
This 1GBit/s Uplink is the main Bottleneck. I cannot change that within budget so i somehow have to live with it and work around.

 

[fabian]

Well, in the Hoster environment the main Problem is the shared 1 GBit/s uplink to the proxmox server which is used by several vms+Backup
This 1GBit/s Uplink is the main Bottleneck. I cannot change that within budget so i somehow have to live with it and work around.

if your local storage is fast, you might want to investigate setting up a local PBS instance, doing the backup there (no network bottleneck), and then syncing the backup snapshots to your big main PBS instance (sync does not slow down guest IO like regular backups do).