Clustering PBS for scaling possibilities

[floh]

Hello Proxmox-Community!

I've read the Wiki of PBS as well as the Roadmap and the Documentation available online - But sadly I wasn't able to figure that out.
There is one major question which comes in my mind.
If I have 10+ PVE Servers with 300+ VMs (incl. LXCs) is there a way to scale the Proxmox Backup Server?
Can I just have a second PVS-Server so I theoretically get twice as much throughput/performance?

So is there a way to improve the performance when adding more Servers (horizontal scaling)?
Or is this feature on your todo-list/roadmap?

Best wishes,
Floh

 

[t.lamprecht]

Hi,

If I have 10+ PVE Servers with 300+ VMs (incl. LXCs) is there a way to scale the Proxmox Backup Server?

I do not think that you'd need to have multiple PBS instances for that node and guest count.

Note that backups are incremental and deduplicated. So, while initial backup may need a good chunk of performance or require a bit more of time, succeeding backups should be rather fast and much less demanding on the backup server than the first ones.
At least, if your VMs do not constantly rewrite all available disk data between backups.

Our architecture is not only able to do things parallel, it is also designed to run asynchronous. This means that one can handle multiple backups even when having just one available core, as soon as one waits for IO transactions to complete we yield with almost zero overhead to another task (similar to co-routines).

Even if one really want to be sure to not run into issues here, and calculates about one CPU thread per parallel backup they expect to happen, one would use a 24/48 CPU for about as many nodes (there runs only a single backup per node at any time).
As said, this is doing calculation with a lot of performance reserve available.

Further, the Server CPU is seldom the bottleneck, a few fast cores can handle a lot of storage IO.

Can I just have a second PVS-Server so I theoretically get twice as much throughput/performance?

So is there a way to improve the performance when adding more Servers (horizontal scaling)?

Yes and no. You can add more PBS instances, but you cannot add them to Proxmox VE as a single unified storage, they'd be separate storage, where you load balance between them by splitting up the backup jobs into two, for example.

Or is this feature on your todo-list/roadmap?

No, the feature that one can put multiple Proxmox Backup Server, and talk to that "cluster" as it'd be one is not on the roadmap yet.

Clustering is no simple task, and would add a lot of complexity to the backup system. But, in our opinion, backup systems should be as easy as possible, to reduce the amount of different failure points, in the runtime and code-bug wise.

 

[t.lamprecht]

Oh, and theoretically it would be possible to put a http 1.1/2 load balancer in front of two PBS instances.
When they'd provide the same credentials (could be rather easily synced, it's "just" two config files) and have a periodic remote sync job setup (this functionality is provided by PBS efficiently and natively), one could do round-robin scheduling between new requests, and so spread the load.

If one does daily backups it would be enough to have a remote sync job between the two PBS instances once a day, could be improved by using a fast direct connection (10Gbps or more) between the two.

So, it is theoretically possible, but was not tried much in practice, AFAIK. I'd still argue that one has less complexity if just a single, powerful, PBS instance is used.