Multiple backups occur simultaneously

Dec 10, 2022
36
1
8
Good morning, everyone,

I have multiple Proxmox VE (PVE) instances that are being saved on the same Proxmox Backup Server (PBS). When multiple backups occur simultaneously, some guests are affected with processes going into uninterruptible sleep. How can I remedy this? I have already tried reducing the maximum number of workers and limiting the bandwidth, but I haven't seen any noticeable improvements.

Thank you.
 
Last edited:
This is by design. If you have several nodes it will start the backup job on each node at the same time. Only way around that is to create separate backup jobs to balance the load.

Mine is set by pools to make it easier to manage the backup jobs.
 
This is by design. If you have several nodes it will start the backup job on each node at the same time. Only way around that is to create separate backup jobs to balance the load.

Mine is set by pools to make it easier to manage the backup jobs.
Until now, I have been doing the same, but it's not enough. I have initial vzdump backups of 15/20 TB that take days, and when they become incremental, it's difficult to find time windows. I was thinking of scheduling a script to execute them sequentially, hoping that the guests of the 6 Proxmox VE nodes can at least perform one backup during the day. Is it possible to have multiple instances of proxmox-backup-proxy? My problem is that the I/O is below the server's capacity, but the throughput is low. Thank you.
 
Maybe you could work with the PBS API and VZDump hook scripts? I for example don't want backups jobs to run within the first 20 minutes after booting a PVE node, so a VZDump hook script will hold a backup job until the uptime exeeds 20 minutes. You probably could do something similar, but instead of checking the uptime you could use the PBS API to check if there wasn't a backup job running for 10 minutes or something similar and hold those backup jobs if a backup job was running recently.
 
  • Like
Reactions: Leprelnx
Maybe you could work with the PBS API and VZDump hook scripts? I for example don't want backups jobs to run within the first 20 minutes after booting a PVE node, so a VZDump hook script will hold a backup job until the uptime exeeds 20 minutes. You probably could do something similar, but instead of checking the uptime you could use the PBS API to check if there wasn't a backup job running for 10 minutes or something similar and hold those backup jobs if a backup job was running recently.
Thank you for the insight,

I'm new, and I will search for APIs. As a heads up, do they also exist for Python? I have recently started learning Rust and I'm having difficulties with object ownership.

Is it possible to do something in bash also?
 
Sure. You could use curl to send the API requests and then use jq to parse the json output.
The example hook script uses perl, but you could write your own bash hook script which works fine here.
So would be possible with bash.
 
Last edited:
  • Like
Reactions: Leprelnx
What do you think if I sequentially launch all the vzdump commands via SSH so that the next one starts once the previous one finishes?

For example:
#!/bin/bash
ssh hostx "vzdump nnn --compress zstd ...."
ssh hosty "vzdump --all --compress zstd ...."
...
 
What do you think if I sequentially launch all the vzdump commands via SSH so that the next one starts once the previous one finishes?

For example:
#!/bin/bash
ssh hostx "vzdump nnn --compress zstd ...."
ssh hosty "vzdump --all --compress zstd ...."
...
Anyway, this is just a temporary fix. The backup mechanism needs to be reconsidered for both the qemu blockcopy part and the backup-proxy functionality of PBS. I have noticed errors on XFS filesystems of guests caused by the I/O wait during backups.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!