Can Backup Fleecing Reduce PBS CPU Usage?

[sdet00]

Greetings! I currently have a Proxmox environment backing up to a Proxmox Backup Server that appears to be CPU limited. The specs of this system are as follows:

Intel Celeron J3060 Dual Core @ 2.48Ghz
8GB RAM
128GB SSD Boot Drive
3x 4TB HDDs in ZFS RAIDz1

During backups it is not uncommon for CPU usage to sit at 100% for hours on end, and I only get around 40MB/s throughput. Whilst it works just fine, I accepted that performance wasn't great and figured it is not a big issue for my environment at the time. That said backups are starting to get larger, and it's getting to the point where processing times could be a problem.

I have a couple of nodes in my Proxmox cluster that have much more powerful CPUs, but I don't have much SSD storage available - only about 200GB free, with around 1.5TB of used storage. Before I go out and purchase another SSD, does anyone know if the new backup fleecing option could help reduce CPU usage on my Proxmox Backup Server and allow for higher throughput? I did some quick searches and couldn't find anything conclusive. The server will easily move data at 110MB/s when it's not doing checksumming and compression on the fly. If not - I might have to make the effort to move things to a more powerful backup server, or I might just turn it back into an SMB server which is what I had it setup previously for backups. Advice appreciated.

 

[Chris]

Hi,

you can try and see with proxmox-backup-client benchmark --repository <your-repository> where possible bottle necks are.

does anyone know if the new backup fleecing option could help reduce CPU usage on my Proxmox Backup Server and allow for higher throughput?

Fleecing is acting on the client side. So if your Proxmox Backup Server is the bottleneck, than fleecing will not help with throughput, it will however help with the VM IO not being blocked on the PVE side by the slow transfers to the PBS.

Intel Celeron J3060 Dual Core @ 2.48Ghz

Probably not the fastest CPU, but I would suggest to first check the performance of you datastore backing storage. Also, how high is the IO delay during the backups on the Proxmox Backup Sever side?

3x 4TB HDDs in ZFS RAIDz1

Running a RAIDZ1, especially on spinning disks also will drastically impact performance. For Proxmox Backup Server, a fast local storage is recommended for most of the use-cases, adding a special device mirror can help to speed up operations, see [0].

Did you already try to see what random read/write performance you will get from the storage itself? You can use tools such as fio to benchmark your storage.

[0] https://pbs.proxmox.com/docs/sysadmin.html#local-zfs-special-device

 

[sdet00]

Yeah, it's not fast. I don't think my storage is a bottleneck. IO Delay is peaking at 0.8% during backups which barely registers. My CPU usage on the other hand...

 

[AmateurProgrammer]

Yeah, it's not fast. I don't think my storage is a bottleneck. IO Delay is peaking at 0.8% during backups which barely registers. My CPU usage on the other hand...


View attachment 72947

Those numbers are REALLY poor, it clearly indicates that you are in urgent need for a better cpu. Incremental backup jobs are quite cpu intensive on the pbs it seems. I have a 1x Xeon E5-2603 v4 (6 core 1.70GHz) on my system and cpu usage frequently rises above 60% while backing up single host at a time even when there is very little data to actually upload from the host.

Here are my benchmark results for system with the before mentioned Xeon cpu for comparison:

 

[sdet00]

Yep my CPU is pretty pathetic. I would like to upgrade it but getting out there to get the job done seems like a big challenge for now.

It may not be such a big deal considering that it is possible to do live-restores nowadays... but I still worry about restore times.