Multiple backups occur simultaneously

[Leprelnx]

Good morning, everyone,

I have multiple Proxmox VE (PVE) instances that are being saved on the same Proxmox Backup Server (PBS). When multiple backups occur simultaneously, some guests are affected with processes going into uninterruptible sleep. How can I remedy this? I have already tried reducing the maximum number of workers and limiting the bandwidth, but I haven't seen any noticeable improvements.

Thank you.

 

[Darkk]

This is by design. If you have several nodes it will start the backup job on each node at the same time. Only way around that is to create separate backup jobs to balance the load.

Mine is set by pools to make it easier to manage the backup jobs.

 

[Neobin]

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=3086

 

[Leprelnx]

This is by design. If you have several nodes it will start the backup job on each node at the same time. Only way around that is to create separate backup jobs to balance the load.

Mine is set by pools to make it easier to manage the backup jobs.

Until now, I have been doing the same, but it's not enough. I have initial vzdump backups of 15/20 TB that take days, and when they become incremental, it's difficult to find time windows. I was thinking of scheduling a script to execute them sequentially, hoping that the guests of the 6 Proxmox VE nodes can at least perform one backup during the day. Is it possible to have multiple instances of proxmox-backup-proxy? My problem is that the I/O is below the server's capacity, but the throughput is low. Thank you.

 

[Dunuin]

Maybe you could work with the PBS API and VZDump hook scripts? I for example don't want backups jobs to run within the first 20 minutes after booting a PVE node, so a VZDump hook script will hold a backup job until the uptime exeeds 20 minutes. You probably could do something similar, but instead of checking the uptime you could use the PBS API to check if there wasn't a backup job running for 10 minutes or something similar and hold those backup jobs if a backup job was running recently.

 

[Leprelnx]

Maybe you could work with the PBS API and VZDump hook scripts? I for example don't want backups jobs to run within the first 20 minutes after booting a PVE node, so a VZDump hook script will hold a backup job until the uptime exeeds 20 minutes. You probably could do something similar, but instead of checking the uptime you could use the PBS API to check if there wasn't a backup job running for 10 minutes or something similar and hold those backup jobs if a backup job was running recently.

Thank you for the insight,

I'm new, and I will search for APIs. As a heads up, do they also exist for Python? I have recently started learning Rust and I'm having difficulties with object ownership.

Is it possible to do something in bash also?

 

[Dunuin]

Sure. You could use curl to send the API requests and then use jq to parse the json output.
The example hook script uses perl, but you could write your own bash hook script which works fine here.
So would be possible with bash.

 

[Leprelnx]

Sure. You could use curl to send the API requests and then use whatever you want to parse the json output.
The example hook script uses perl, but you could write your own bash hook script which works fine here.

Great, I'll try. Thx very much

 

[Leprelnx]

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=3086

I think the problem lies not in the lack of backup simultaneity, but in the management of qemu blockcopy and the throughput of PBS.

 

[Leprelnx]

What do you think if I sequentially launch all the vzdump commands via SSH so that the next one starts once the previous one finishes?

For example:
#!/bin/bash
ssh hostx "vzdump nnn --compress zstd ...."
ssh hosty "vzdump --all --compress zstd ...."
...

 

[Leprelnx]

What do you think if I sequentially launch all the vzdump commands via SSH so that the next one starts once the previous one finishes?

For example:
#!/bin/bash
ssh hostx "vzdump nnn --compress zstd ...."
ssh hosty "vzdump --all --compress zstd ...."
...

Anyway, this is just a temporary fix. The backup mechanism needs to be reconsidered for both the qemu blockcopy part and the backup-proxy functionality of PBS. I have noticed errors on XFS filesystems of guests caused by the I/O wait during backups.