Extremely Slow PBS Speeds

[endlessnesting]

Uploaded 66 chunks in 5 seconds.
Time per request: 82194 microseconds.
TLS speed: 51.03 MB/s
SHA256 speed: 273.01 MB/s
Compression speed: 552.43 MB/s
Decompress speed: 972.39 MB/s
AES256/GCM speed: 82.09 MB/s
Verify speed: 136.66 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 51.03 MB/s (4%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 273.01 MB/s (14%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 552.43 MB/s (73%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 972.39 MB/s (81%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed │ 136.66 MB/s (18%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 82.09 MB/s (2%) │
└───────────────────────────────────┴───────────────────┘


I have Proxmox Backup Server 2.1-2 running on proxmox VE 7.1-8 on a Dell R630 (2x xeon 2637v4) and a Truenas VM running on the same host. The Truenas VM has a PERC in HBA mode with 8 Seagate Nytro SSDs.
PBS VM: 4 cores, 8GB ram
TrueNas VM: 4 cores, 12GB ram

I have a SMB share mapped from the PBS VM to the TrueNAS VM with these options:
//IP/PrimaryBackup /mnt/PrimaryBackup cifs credentials=/root/.smbcreds,iocharset=utf8,file_mode=0660,dir_mode=0770,uid=34,noforceuid,gid=34,noforcegid,rw 0 0

As you can see above, the performance benchmark shows terrible TLS speeds to a local network share.

I have just switched from ESXi + Veeam using the SAME truenas VM and was frequently able to write data to the TrueNAS VM at over 330MB/s. I was excited to switch to a more open virtualization platform, but this behavior is very concerning. My backups would take days to complete per job at these speeds.

Is there some performance tuning I am missing? I have qemu guest agent installed on the PBS VM, is there anything else I can do?

 

[Dunuin]

PBS was designed with only local SSDs in mind. Network shares will work but don't expect great performance that way.

 

[endlessnesting]

PBS was designed with only local SSDs in mind. Network shares will work but don't expect great performance that way.

That is extremely disappointing, considering Veeam had full support for network shares and at high speeds.

I have done plenty of SMB data transfers from other debian VMs and can easily saturate the disk write speeds, why is PBS (which is just debian) limited here?

 

[Dunuin]

Keep in mind that PBS really needs low latencies. Veeam will use big incremental image files. PBS chops everything in small (some KB and up to a maximum of 4 MiB) chunks. So if you backup a 4TB disk it will need to hash and write atleast 1 million small files. NFS/SMB isn't that great at handling millions of small files.

 

[endlessnesting]

Ah, that sucks.

I think I just confirmed that, I tried transfering a 10GB file full of random data from PBS through the SMB share to the Truenas VM and was able to sustain 330-450MB/s.

Is there anyway to force backup jobs to pull large images instead?

Any idea why proxmox would stray from using large image files? Seems like a mistake to me.
Who keeps all their VM backups on local disks anyways? I'd rather push all that important data to an actual NAS.

 

[Dunuin]

Is there anyway to force backup jobs to pull large images instead?

Any idea why proxmox would stray from using large image files? Seems like a mistake to me.
Who keeps all their VM backups on local disks anyways? I'd rather push all that important data to an actual NAS.

Like I already said, PBS was designed with local Enterprise SSDs in mind. Local SSDs got no problem with alot of small files and only HDDs or network shares struggle with the IOPS. Everything PBS stores will be hashed, compressed and deduplicated and for that everything has to be chopped in small chunks. If the chunksize would be bigger, the data would be way less deduplicatable. If your data got 100 time the same 4MB block of data, it will only need 4MB (+ some kb metadata) to store it. Using big image files like other backup tools doing it, this would consume the full 400MB instead of just 4MB. So you can really save space.
My PBS for example stores 8.732 TiB of VMs and just needs 115.522 GiB to store that. Compare that to other backup tools that are storing stuff as big images.

 

[endlessnesting]

Well, I guess I'll have to move back to cron job for vzdump. That at least transfers at full speed.

All I'm saying is that veeam does a decent job at incremental backups, and doesn't use this method.

In order for PBS to work for me, I'd have to go and buy a ton of local storage for PBS and I have other things that need to use truenas as a regular NAS, so having it act as a vm backup location is very convenient. This just creates unnessesary cost and complexity for my setup.

 

[Dunuin]

Transfer speed isn't everything. With PBS deduplication will save you TBs of space and it allows incremental backups. Lets say you got a 1 TB VM that changes only 10GB per day and you want to keep 7 daily backups of it. Lets say PBS can only transfer with 100 MB/s and vzdump with 500 MB/s.

	Transfer time (Vzdump):	Backup size (Vzdump):	Transfer time (PBS):	Backup size (PBS):
Backup 1:	1600 s	800 GB	7000 s	700 GB
Backup 2:	1600 s	800 GB	70 s	7 GB
Backup 3:	1600 s	800 GB	70 s	7 GB
Backup 4:	1600 s	800 GB	70 s	7 GB
Backup 5:	1600 s	800 GB	70 s	7 GB
Backup 6:	1600 s	800 GB	70 s	7 GB
Backup 7:	1600 s	800 GB	70 s	7 GB
Total:	11200 s	5600 GB	7420 s	742 GB

So even if Vzdump would transfer 5 times faster PBS would be way faster because it needs to transfer way less data because of the deduplication. And saving nearly 5TB of storage is a nice benefit too.

Thats just an example, but you really should test PBS for some weeks to see how it works in your use case.

 

[endlessnesting]

Transfer speed isn't everything. With PBS deduplication will save you TBs of space and it allows incremental backups. Lets say you got a 1 TB VM that changes only 10GB per day and you want to keep 7 daily backups of it. Lets say PBS can only transfer with 100 MB/s and vzdump with 500 MB/s.

	Transfer time (Vzdump):	Backup size (Vzdump):	Transfer time (PBS):	Backup size (PBS):
Backup 1:	1600 s	800 GB	7000 s	700 GB
Backup 2:	1600 s	800 GB	70 s	7 GB
Backup 3:	1600 s	800 GB	70 s	7 GB
Backup 4:	1600 s	800 GB	70 s	7 GB
Backup 5:	1600 s	800 GB	70 s	7 GB
Backup 6:	1600 s	800 GB	70 s	7 GB
Backup 7:	1600 s	800 GB	70 s	7 GB
Total:	11200 s	5600 GB	7420 s	742 GB

So even if Vzdump would transfer 5 times faster PBS would be way faster because it needs to transfer way less data because of the deduplication. And saving nearly 5TB of storage is a nice benefit too.

Thats just an example, but you really should test PBS for some weeks to see how it works in your use case.

I see the benefits there. And if I really cared about the fine-grained recovery options I'd probably consider it. But to be honest, I really only care about something big breaking and having a couple restore points. Any important application data on the VMs is all stored directly on a NAS through a share and replicated offsite. I think vzdump suites me as I will only be needing 3-5 backups MAX per vm. Couple that with the biggest VM I have is only ~20GB.

I do appreciate what proxmox is trying to do with PBS. All I can ask is if this method is so good, why doesn't the entire industry use it? Especially Veeam? Don't get me wrong, I don't like veeam that much, but it is kinda the standard for virtualization backup.

 

[Dunuin]

I do appreciate what proxmox is trying to do with PBS. All I can ask is if this method is so good, why doesn't the entire industry use it? Especially Veeam? Don't get me wrong, I don't like veeam that much, but it is kinda the standard for virtualization backup.

The big players are all that big, because the are so old and therefore everything was designed with slow HDDs in mind because SSDs were way to expensive back then. But nowadays SSD prices became reasonable enough to be used for backups and if you look at the trend in some years noone wants to buy HDDs anymore if you get way faster SSDs for the same price per TB or even cheaper. So its not the dumbest idea, if you have to design a new backup utility from stratch, to design it with SSDs in mind and make use of all the SSD benefits, if you want to be futureproof.

 

[tom]

AES256 GCM encryption speed │ 82.09 MB/s (2%) │

This is way to low for your hardware. Are you sure that you have AES on your hardware enabled (BIOS setting)?

Type the following command to make sure that the processor has the AES instruction set and enabled in the BIOS:

> grep -m1 -o aes /proc/cpuinfo

 

[endlessnesting]

This is way to low for your hardware. Are you sure that you have AES on your hardware enabled (BIOS setting)?

Type the following command to make sure that the processor has the AES instruction set and enabled in the BIOS:

> grep -m1 -o aes /proc/cpuinfo

# grep -m1 -o aes /proc/cpuinfo
aes

 

[endlessnesting]

The big players are all that big, because the are so old and therefore everything was designed with slow HDDs in mind because SSDs were way to expensive back then. But nowadays SSD prices became reasonable enough to be used for backups and if you look at the trend in some years noone wants to buy HDDs anymore if you get way faster SSDs for the same price per TB or even cheaper. So its not the dumbest idea, if you have to design a new backup utility from stratch, to design it with SSDs in mind and make use of all the SSD benefits, if you want to be futureproof.

Seems reasonable. I just don't know why there isn't a more compatible solution to these older backup architectures. Why not include an additional feature in PBS to conduct backups similar to veeam? It would give people the flexibility to do network based backups, and make PBS easier to implement in a variety of environments.

 

[fabian]

that benchmark doesn't even benchmark your storage (like it says on the tin - it benchmarks uploading via TLS/HTTP2, since that is one of the potential bottle necks) - something is very wrong with your machine (as @tom already said, the low AES performance might be the culprit)! is PBS running directly on the PVE host, or in a VM?

for reference, here's output from a slightly more modern, workstation-grade hardware (Cascade Lake i9):

Uploaded 1567 chunks in 5 seconds.
Time per request: 3193 microseconds.
TLS speed: 1313.30 MB/s
SHA256 speed: 573.20 MB/s
Compression speed: 862.16 MB/s
Decompress speed: 1508.35 MB/s
AES256/GCM speed: 2867.33 MB/s
Verify speed: 412.48 MB/s

 

[Dunuin]

@fabian:
As far as I understand PBS and TrueNAS (serving the datastore over SMB) are both running inside VMs on a PVE.

@ehammer:
Did you set the cpu type of these VMs to "host" or are they running as the default "kvm64"? If its the latter try changing it to host and also set the AES flag for better AES speeds.

 

[endlessnesting]

@fabian:
As far as I understand PBS and TrueNAS (serving the datastore over SMB) are both running inside VMs on a PVE.

@ehammer:
Did you set the cpu type of these VMs to "host" or are they running as the default "kvm64"? If its the latter try changing it to host and also set the AES flag for better AES speeds.

The cpu was set to default kvm64. Is there any downside to running the host option? Why is it not the default?

 

[Dunuin]

Kvm64 will only use a limited amount of instruction sets to be compatible with most systems. So with kvm64 the VM most likely can't make use of all features of the physical CPU and the VM will be slower but the benefit is that this VM can be migrated between nodes of a unhomogenoius cluster.
So if you are not running a cluster its always better to to use "host" so the VM can make use of the full potential of the CPU.

 

[endlessnesting]

Kvm64 will only use a limited amount of instruction sets to be compatible with most systems. So with kvm64 the VM most likely can't make use of all features of the physical CPU and the VM will be slower but the benefit is that this VM can be migrated between nodes of a unhomogenoius cluster.
So if you are not running a cluster its always better to to use "host" so the VM can make use of the full potential of the CPU.

Ah that makes sense, I will move everything over to that. I am not running a cluster.

 

[deepcloud]

how does one get these values

Uploaded 1567 chunks in 5 seconds.
Time per request: 3193 microseconds.
TLS speed: 1313.30 MB/s
SHA256 speed: 573.20 MB/s
Compression speed: 862.16 MB/s
Decompress speed: 1508.35 MB/s
AES256/GCM speed: 2867.33 MB/s
Verify speed: 412.48 MB/s

 

[Neobin]

how does one get these values

https://forum.proxmox.com/threads/how-fast-is-your-backup-datastore-benchmark-tool.72750