PBS Performance improvements (enterprise all-flash)

[Ramalama]

thanks for the additional numbers! that does indeed look like there is some severe bottle neck happening that we should get to the bottom of..

I suspect that there is something in the compression pipeline.
Because like i mentioned in the Bug Report, even if the way the Backup works with PBS compared to Local-Storage Backup, the limitations are all exactly the same.

Or the speeds are identical between Local and PBS as soon as Compression is enabled (no matter which compression or how many zstd multitasking threads).
With disabling Compression, it feels like every Limitation is removed, the Backup-Speeds go up to 5GB/s here.
With enabled Compression (no matter which/multitasking etc), no matter if Local or PBS, i can't pass 1GB/s.

And in my opinion it has nothing todo with Clockspeeds either because i Monitored the CPU-Utilization during Backuping (Local and PBS), and i don't see any Cores reaching 100%.
But im not entirely sure, since on other side, the Clockspeeds are definitively important, i get on Servers that can reach higher Clockspeeds faster Backup Speeds.

I took me 3 days of trying to find out what the Bottleneck is, as i created my Bugreport...
And i tryed to make PBS-VM Instances and even reinstalled one of both Genoa-Servers with PBS, to have the fastest possible PVE+PBS on the planet, tryed with Local Storage and all tuning options possible on Compression, etc...
Monitored CPU and IO etc...
And couldn't find out what the Bottleneck is and gaved up.

On the returning side of things, it seems like im still Archieving the fastest PBS Backup-Speeds with 1GB/s even with SAS (HDD) Drives on PBS, while others are using NVME Drives and have troubles to reach 500mb/s.

Cheers

 

[fabian]

Just as bad, or worse.

Upload image '/dev/mapper/pve-vm--100--disk--0' to 'root@pam@10.226.10.10:8007:pbs-primary' as tomtest.img.fidx
tomtest.img: had to backup 62.832 GiB of 80 GiB (compressed 42.404 GiB) in 673.35s
tomtest.img: average backup speed: 95.552 MiB/s
tomtest.img: backup was done incrementally, reused 17.168 GiB (21.5%)
Duration: 676.53s
End Time: Tue Jul  9 12:59:22 2024

FWIW, I very likely found the issue with this particular invocation, and might have some test package (if you are willing!) to try for image backups using proxmox-backup-client.

 

[tomstephens89]

FWIW, I very likely found the issue with this particular invocation, and might have some test package (if you are willing!) to try for image backups using proxmox-backup-client.

Please share.

 

[fabian]

https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=deb237a28883bba0584766129b01997ccd63c4fe

since there is another patch pending with performance implications, I'll wait for that one before doing a test build:

https://lists.proxmox.com/pipermail/pbs-devel/2024-July/010330.html