host backup performance

[mnicholls]

Hello,

I've been using PBS to backup my VMs for some time and performance has been fantastic. I'm now testing it out as a file backup target and using the backup client 3.0.2 against a rather large directory (family pictures, around 120,000 files) the client seems to slow significantly after some time, before getting through much of the directory. I'm wondering if there are any optimizations I should be making? So far the only thing I've tried is reducing the chunk size, but I can't really see much difference.

cpu/ram/disk on the host doing the backup seems minimal so it doesn't appear to be a bottleneck there or on the network.

 

[Chris]

Hi,
can you share the output of a proxmox-backup-client benchmark --repository <target-repository>? This might indicate what the bottleneck is.

Note that the current implementation of the file level backup client requires to read and chunk all of the data before upload and de-duplication. There are efforts to improve this in the future, see https://lists.proxmox.com/pipermail/pbs-devel/2023-November/007059.html

 

[mnicholls]

Thanks Chris,

Uploaded 178 chunks in 5 seconds.
Time per request: 28383 microseconds.
TLS speed: 147.77 MB/s
SHA256 speed: 85.28 MB/s
Compression speed: 95.57 MB/s
Decompress speed: 115.61 MB/s
AES256/GCM speed: 277.58 MB/s
Verify speed: 47.90 MB/s


I think the bottleneck is just the size of the directory as I have other directories being backed up by the same host that saturate my uplink to the PBS server. I'm still doing the initial backup of this one large directory so I'll just wait it out, but I thought I'd ask if I was missing any tuning as there are times it slows to a crawl and I think you've explained why sufficiently.

thanks again

 

[deanjarnold]

Hello @Chris, Like the OP I am attempting to perform large file backups of RAW/ProRes Video files and many smaller MP4s and photos; this is ~24TB of data and takes over 19hrs to complete. I have monitored the backup and the client is sending all the file data to PBS and then PBS is deduping & dropping non-changes etc. For this backup, very few files actually change week to week.

Previously I was using Rclone to a SMB file share, but as we recently move from VMWare to PVE I'd like to standardize on PBS and the PBS Backup Client.

The patch details you posted above looks very promising. Do you know if there is a beta available; I'd be happy to test & provide feedback. I can also apply patches, as I actually build a Redhat/Rocky PBS Client from source.

-------

Just found https://bugzilla.proxmox.com/show_bug.cgi?id=3174 & figured out the thread contains the patches.
Is the Nov V5 or the March v3 the most current?

 

[Chris]

The patch details you posted above looks very promising. Do you know if there is a beta available; I'd be happy to test & provide feedback. I can also apply patches, as I actually build a Redhat/Rocky PBS Client from source.

Hi,
no there is currently (at the time of writing) no packaged version available, as the patches are still undergoing some changes and also the final file format is not fixed yet. So this is not ready to be used with production data just yet.

Just found https://bugzilla.proxmox.com/show_bug.cgi?id=3174 & figured out the thread contains the patches.
Is the Nov V5 or the March v3 the most current?

There are 2 different approaches which were prototyped in order to find the more viable solution. The patch series from November used an appendix approach, re-indexing the reusable chunks for a pxar archive at the end of the archive. This was however discontinued in favor of the much more promising second approach with splitting pxar archives in a metadata and payload data part, which currently has version 3 of the patch series on the mailing list and is liked in the bug report. Note that (as described also in the bug report) there where already some further adaptions and bugfixes, so do not use that version for productive use as well. Once there is a packaged version available, we will let you know and you can get a notification by subscribing to the issue in the bugtracker.

Hope this answers your questions.

 

[deanjarnold]

@Chris Thank you for your detailed response.

I will subscribe and follow the bugtracker issue with great interest, and will be happy to patch and test as needed.