PBS local sync slow

[lifi]

Hello,

I got a little problem with syncing my backups to a bigger local datastore.
Source is a 3-disk ZFS, target is a 5-disk ZFS - both are on the same machine.

"proxmox-backup-client benchmark" delivers the following numbers:

SHA256 speed: 1787.13 MB/s   
Compression speed: 485.86 MB/s   
Decompress speed: 617.96 MB/s   
AES256/GCM speed: 3968.64 MB/s   
Verify speed: 460.15 MB/s   
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ not tested          │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 1787.13 MB/s (88%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 485.86 MB/s (65%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 617.96 MB/s (52%)   │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 460.15 MB/s (61%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 3968.64 MB/s (109%) │
└───────────────────────────────────┴─────────────────────┘

However, the sync job is almost idle at around 5MiB/s.
Thing is, I want to recycle the 3 disks from the smaller datastore for an offsite PBS on Thursday and need to remove them in like 2 days.
Is there an option to speed up the sync?

 

[Chris]

Can you specify in more detail what disks you are using for the pools the PBS datastores resides on? I suspect that you are severely limited by IO, how high is your IO delay? Note that PBS stores data in a lot of small (typically around 2MiB in size) chunk files, which for syncing it needs to read from the source pool and write to the target pool.

Can you also show the sync job configuration? Are you using a local sync job?

 

[lifi]

All the disks are Seagate Exos 24TB, IO-Delay is between 12% and 17%.
The PBS is hosted inside a PVE, and has 8 cores of an Epyc 9124 as well as 96GiB RAM assigned.
The storages are grouped as RAIDZ1 in PVE and forwarded as SCSI disks with iothread=1 as recommended by a colleague.

There are a few other VMs hosted on that machine, but those have their own SSDs,

The sync job is a local pull job with "Max depth = 1"

 

[Johannes S]

All the disks are Seagate Exos 24TB, IO-Delay is between 12% and 17%.

Then you should really consider to add SSDs as a special device mirror to your RAIDZ. It basically works as a cache for your files metadata and small file (not larger than a few kB) storage. It should be noted however that in case of a broken special device (due to a hardware failure or whatever) the data on the hdds will be lost too. For that reason the redundancy of the special device SSDs must match that of the HDDs, so you will need three or five SSDs for your setup. You will also need to rewrite the data with zfs send/receive.
This won't do much for sync performance, but will do a lot for operations who need to read and write a lot of metadata of the files (e.g. garbage collection). For operations which actually need to read or write the acutual data the speedup will be not so much (since you still need to deal with HDDs), but should help at least a little bit.
https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_special_device

The SSDs don't need to be large, I read that it can be from 0,02 to 2 % of the HDDs capacity. So to be safe, 2% is a good rule of thumb.
Like for vms it's highly recommended that they have power-loss-protection aka that they are datacenter grade.

Another thing I'm wondering about is that you run PBS as a VM. Did you try to do a parallel install to your PVE? Although (for good reasons!) this is not recommended (but running as a VM isn't either) it might be worth a shot whether the virtualization hurts your performance.