Practical limits for large VM backups (up to 16TB) in Proxmox Backup Server

[Pasiu]

Hi everyone,

I’m planning to use Proxmox Backup Server (PBS) with PBS client only - no Proxmox VE. I want to back up very large VMs, up to 16TB disks.
I searched online but couldn’t find any official limits, so I’d like to confirm if the only constraints are hardware-related.

Specifically:

• Any practical differences between file-based and block-based backups for very large VMs?
• Any tips or limitations regarding network, storage, or I/O for such large backups?

Thanks in advance for sharing your experience!

 

[kosnar]

Hi,

I don't have much experiences with PBC...so take this with some reserve. For better response I need more precise description of the very large VM - which OS, FS, data type, amount of changed bytes per day. But if I understand correctly you are in HW design phase.

Generally there is physical limits:
A) path between VM and PBS
1) LAN - the slowest device in lan chain (f.e. - good 1Gbps link mean limit about 80MB/s)

B) PBS
1) disks IOps
2) disk (pool) absolute write and read speed
3) CPU - for validation actually not so expensive

C) VM
1) disks IOps (count of files)
2) disk (pool) absolute read speed
3) CPU - file splitting and hashing of chunks

PBC backups delta, but PBS have to check whole backup (every backup stored in PBS is full backup, data is deduplicated). Another word If your VM is 14TB of data and you change 1GB per day so daily backup transfer approximately 1GB to PBS but for PBS it is 14TB backup to check consistency and this must be done before next backup time gets up.

I don't think there is software limits.

Asked limits needs to be calculated and verified from this HW limits with enough reserve.
I recommend to make tests before implementing - try it with 140GB system similar to final one.

-----------

File based mean file change -> transfer whole file
Block based -> transfer part of the file ... because single file can be splited to many chunks

-----------

Please consider this backup process twice because PBC and generaly file system based backups is good for files but can broke databases.

R.

 

[IsThisThingOn]

I’m planning to use Proxmox Backup Server (PBS) with PBS client only - no Proxmox VE. I want to back up very large VMs, up to 16TB disks.

There is not technical limitation AFAIK, but it is just not very smart.

The root issue is as already mentioned that you have data in blockstorage instead of files.
(at least that is my guess with the limited information you provided).
This comes with all the drawbacks of blockstorage.

These are just wild guesses, but lets compare these two scenarios:

Pasiu:
Pasiu is using a virtual Windows Servers that has a 16TB disk. These Windows servers share some files to other Windows clients.
To backup them with PBS, the whole 16TB has to be read, put into a 4MB chunk, calculate the checksum for this, ask PBS if it already has that checksum. Even assuming that you can read at 5GB/s, this will take almost an hour.

TrueNAS setup:
Files are shared from a ZFS TrueNAS. Maybe also in AD, maybe just local user managment.
TrueNAS has a 16TB of files in a dataset share.
Backing that up is almost instant in most cases (ZFS snapshot + ZFS send is mostly limited by (changed data / bandwidth)).
Or alternatively with rsync, which adds metadata performance constraints, but still way better.

And that is ignoring the many other potentially drawbacks block storage has.

 

[Pasiu]

Many thanks for your prompt answers, and sorry for not providing more details earlier.

My main concern was whether PBS can reliably handle backups of around ~16 TB, and from your feedback, it seems this shouldn’t be a problem.

I’ll mainly be working with Linux (Debian/Ubuntu) QEMU virtual machines. My plan had been to take an LVM snapshot of each running VM and then run the Proxmox Backup Client against that snapshot. My initial reasoning was to use a block-based backup approach so that I could easily restore the entire VM if needed.

@IsThisThingOn – you raised a very interesting point about the filesystem aspect, which I hadn’t considered earlier. I now realize this can have a big impact on processing and backup performance, so I’ll definitely take it into account.

Thanks again to everyone for your help and insights! I’ll run some performance tests on my setup, but your answers already clarified a lot for me.