Backup Server too slow, IO on VM stalled during Backup

adoII

Active Member
Jan 28, 2010
166
15
38
Hi,
I have a setup at a Hoster where the Proxmox VM Server is damned fast using NVMEs and the Proxmox Backup Server is terrible slow, for example network Bandwidth sometimes is only 30 MB/s

What happens now: The Backup ov VM starts really fast with let's say 500 MB/s reading speed.
The Proxmox Backup Server and/or the Network can only write at around 30MB/s
As soon as some Cache gets full all IO gets halted on the VM that is backed up and the VM is not usable anymore

What can I do about it ? My idea is to limit the reading speed for backups to a pbs server to something like 30MB/s, would that be a good workaround ?
 

Dunuin

Famous Member
Jun 30, 2020
9,039
2,321
156
Germany
First I would try to find out why it is so slow.

You could use iperf3 to test the performance of the network connection between the PVE and PBS.
Then you could run a PBS benchmark: proxmox-backup-client benchmark --repository yourbackupstore
And you can use fio to benchmark the read performance of the PVE host and write/IOPS performance of the PBS server.

PBS needs a good IOPS + random IO performance, so best you use only SSDs or atleast HDDs + SSDs for caching / to store the metadata.
A suddenly terrible performance drop indeed sounds like a cache thats gets full. Make sure that server isn't using SMR HDDs or QLC SSDs as there this drop will be especially horrible.
 
Last edited:

adoII

Active Member
Jan 28, 2010
166
15
38
Well, in the Hoster environment the main Problem is the shared 1 GBit/s uplink to the proxmox server which is used by several vms+Backup
This 1GBit/s Uplink is the main Bottleneck. I cannot change that within budget so i somehow have to live with it and work around.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,424
1,679
174
Well, in the Hoster environment the main Problem is the shared 1 GBit/s uplink to the proxmox server which is used by several vms+Backup
This 1GBit/s Uplink is the main Bottleneck. I cannot change that within budget so i somehow have to live with it and work around.

if your local storage is fast, you might want to investigate setting up a local PBS instance, doing the backup there (no network bottleneck), and then syncing the backup snapshots to your big main PBS instance (sync does not slow down guest IO like regular backups do).
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!