How fast is your backup datastore? Benchmark tool

[tom]

We provide a small benchmark tool providing some metrics about your datastore on your Proxmox Backup Server, here is an example:

root@pbs-20:~# proxmox-backup-client benchmark --repository mybackupstore

Uploaded 703 chunks in 5 seconds.
Time per request: 7131 microseconds.
TLS speed: 588.11 MB/s
SHA256 speed: 1629.45 MB/s
Compression speed: 1855.22 MB/s
Decompress speed: 8156.74 MB/s
AES256/GCM speed: 2949.58 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 588.11 MB/s (100%)  │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum comptation speed  │ 1629.45 MB/s (77%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 1855.22 MB/s (86%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 8156.74 MB/s (101%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 2949.58 MB/s (78%)  │
└───────────────────────────────────┴─────────────────────┘
root@pbs-20:~#

This is an AMD EPYC 7302P 16-Core Processor (1 Socket), the datastore is a ZFS mirror on 2 x 6 TB HDDs.

 

[msproact]

Not as fast, but still nice what a Intel NUC NUC6i5SYB with 1,5 TB WD Green USB harddisk can do:

Uploaded 631 chunks in 5 seconds.
Time per request: 7947 microseconds.
TLS speed: 527.77 MB/s
SHA256 speed: 366.95 MB/s
Compression speed: 1067.84 MB/s
Decompress speed: 4218.49 MB/s
AES256/GCM speed: 2794.41 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 527.77 MB/s (89%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum comptation speed  │ 366.95 MB/s (17%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1067.84 MB/s (49%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 4218.49 MB/s (52%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2794.41 MB/s (73%) │
└───────────────────────────────────┴────────────────────┘

 

[czechsys]

DL380P G8, Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, hypertreading enabled, 2x10Gbps
PBS is VM, cpu type host , 2GB RAM, 8xHDD R10 P430 as qemu-disk

2x core

Uploaded 350 chunks in 5 seconds.
Time per request: 14517 microseconds.
TLS speed: 288.91 MB/s
SHA256 speed: 249.73 MB/s
Compression speed: 892.48 MB/s
Decompress speed: 2989.36 MB/s
AES256/GCM speed: 1143.31 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 288.91 MB/s (49%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum comptation speed  │ 249.73 MB/s (12%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 892.48 MB/s (41%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 2989.36 MB/s (37%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1143.31 MB/s (30%) │
└───────────────────────────────────┴────────────────────┘

4x core

Uploaded 194 chunks in 5 seconds.
Time per request: 26037 microseconds.
TLS speed: 161.09 MB/s
SHA256 speed: 249.19 MB/s
Compression speed: 932.68 MB/s
Decompress speed: 3302.35 MB/s
AES256/GCM speed: 1101.85 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 161.09 MB/s (27%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum comptation speed  │ 249.19 MB/s (12%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 932.68 MB/s (43%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 3302.35 MB/s (41%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1101.85 MB/s (29%) │
└───────────────────────────────────┴────────────────────┘

 

[t.lamprecht]

PBS is VM,

Be sure to use the most specific CPU model you can, ideally "host". Else it may not be able to recognize some hardware acceleration support used for AES, SHA256, ...

 

[czechsys]

Be sure to use the most specific CPU model you can, ideally "host". Else it may not be able to recognize some hardware acceleration support used for AES, SHA256, ...

I am using host. Updated and edited. Interesting thing - raising from 2 to 4 core dropped TLS speed.

 

[Ansy]

AMD Ryzen 5 1600X Six-Core Processor

        NAME                                                STATE     READ WRITE CKSUM
        rpool                                               ONLINE       0     0     0
          mirror-0                                          ONLINE       0     0     0
            ata-TOSHIBA_DT01ACA100_X4CNGG2NS-part3          ONLINE       0     0     0
            ata-WDC_WD10EZEX-60WN4A0_WD-WCC6Y6PT33JY-part3  ONLINE       0     0     0

root@ryzen:~# proxmox-backup-client benchmark --repository backup
Uploaded 623 chunks in 5 seconds.
Time per request: 8045 microseconds.
TLS speed: 521.30 MB/s
SHA256 speed: 1820.74 MB/s
Compression speed: 1807.70 MB/s
Decompress speed: 6678.33 MB/s
AES256/GCM speed: 3125.94 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 521.30 MB/s (88%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum comptation speed  │ 1820.74 MB/s (86%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1807.70 MB/s (84%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 6678.33 MB/s (83%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 3125.94 MB/s (82%) │
└───────────────────────────────────┴────────────────────┘

 

[philwo]

root@pve:~ # proxmox-backup-client benchmark --repository proxve@pbs@pbs.fritz.box:backup
Uploaded 165 chunks in 5 seconds.
Time per request: 31610 microseconds.
TLS speed: 132.69 MB/s
SHA256 speed: 631.15 MB/s
Compression speed: 1986.44 MB/s
Decompress speed: 7689.46 MB/s
AES256/GCM speed: 4154.22 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 132.69 MB/s (22%)   │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum comptation speed  │ 631.15 MB/s (30%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 1986.44 MB/s (92%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 7689.46 MB/s (95%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 4154.22 MB/s (109%) │
└───────────────────────────────────┴─────────────────────┘

Backup client is an HPE ML30 Gen10 with a Xeon E-2288G and 2x10GbE (Intel X550-T2), the local storage is 4x Micron 5300 Pro 7.68TB in ZFS RAID10.

Backup server is an HPE MicroServer Gen10Plus with a Xeon E-2246G CPU and 1x5GbE (via a QNA-UC5G1T USB adapter). Backup datastore is 4x WD Gold 12TB in ZFS RAIDZ1.

The servers are connected via a Netgear XS712Tv2 switch, the backup client to two native 10GBase-T ports (configured as LACP LAG), the backup server to an SFP+ port with a Mikrotik S+RJ10 module, because the switch can't do 5GbE on its RJ45 ports. Bandwidth between both servers measured via iperf3 is 3.3 Gbit/sec - not great, not terrible.

Any idea why the TLS speed is so low?

Also, the measured SHA256 speed seems to be using a single thread (I get the same results when benchmarking with "openssl speed sha256", which uses just one thread according to top). Would it be possible to parallelize this in proxmox-backup-client? It looks like my container backups are currently bottle-necked by sha256 verification, as proxmox-backup-client stays at 200% CPU reading ~800MB/s according to iotop for a long time.

 

[iTweek]

Uploaded 386 chunks in 5 seconds.
Time per request: 13014 microseconds.
TLS speed: 322.29 MB/s
SHA256 speed: 347.54 MB/s
Compression speed: 1551.26 MB/s
Decompress speed: 5258.47 MB/s
AES256/GCM speed: 153.82 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 322.29 MB/s (55%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum comptation speed │ 347.54 MB/s (16%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 1551.26 MB/s (72%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 5258.47 MB/s (65%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 153.82 MB/s (4%) │
└───────────────────────────────────┴────────────────────┘

 

[H4R0]

Uploaded 997 chunks in 5 seconds.
Time per request: 5020 microseconds.
TLS speed: 835.43 MB/s
SHA256 speed: 579.44 MB/s
Compression speed: 1213.22 MB/s
Decompress speed: 3158.22 MB/s
AES256/GCM speed: 4226.95 MB/s

Proxmox testlab running inside a libvirt kvm on a i9-9900k

 

[SirUffsALot]

Whitebox Server, Intel Xeon E3-1230 v6, 32GB RAM, 6x WD Red 10TB in 3 mirror-vdevs

Uploaded 884 chunks in 5 seconds.
Time per request: 5669 microseconds.
TLS speed: 739.81 MB/s
SHA256 speed: 476.44 MB/s
Compression speed: 1414.64 MB/s
Decompress speed: 6721.47 MB/s
AES256/GCM speed: 3575.03 MB/s
+===================================+====================+
| Name                              | Value              |
+===================================+====================+
| TLS (maximal backup upload speed) | 739.81 MB/s (125%) |
+-----------------------------------+--------------------+
| SHA256 checksum comptation speed  | 476.44 MB/s (22%)  |
+-----------------------------------+--------------------+
| ZStd level 1 compression speed    | 1414.64 MB/s (66%) |
+-----------------------------------+--------------------+
| ZStd level 1 decompression speed  | 6721.47 MB/s (83%) |
+-----------------------------------+--------------------+
| AES256 GCM encryption speed       | 3575.03 MB/s (94%) |
+===================================+====================+

 

[oversite]

backup server is a vm on a 5-node ceph cluster
backupuser@backupserver:~# proxmox-backup-client benchmark --repository zfs1
Uploaded 1035 chunks in 5 seconds.
Time per request: 4838 microseconds.
TLS speed: 866.87 MB/s
SHA256 speed: 2074.04 MB/s
Compression speed: 1769.71 MB/s
Decompress speed: 11135.67 MB/s
AES256/GCM speed: 4098.41 MB/s
┌───────────────────────────────────┬──────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪══════════════════════╡
│ TLS (maximal backup upload speed) │ 866.87 MB/s (147%) │
├───────────────────────────────────┼──────────────────────┤
│ SHA256 checksum comptation speed │ 2074.04 MB/s (98%) │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 compression speed │ 1769.71 MB/s (82%) │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 decompression speed │ 11135.67 MB/s (138%) │
├───────────────────────────────────┼──────────────────────┤
│ AES256 GCM encryption speed │ 4098.41 MB/s (108%) │
└───────────────────────────────────┴──────────────────────┘

 

[Dark26]

PBS server in a vm in pve. 1 Go ram 2 vcpu. backend store Store is a disk image on a cifs share on a freebox ( speed max around 25 mo /s)

Uploaded 101 chunks in 5 seconds.
Time per request: 52430 microseconds.
TLS speed: 80.00 MB/s
SHA256 speed: 107.57 MB/s
Compression speed: 295.80 MB/s
Decompress speed: 957.99 MB/s
AES256/GCM speed: 45.41 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 80.00 MB/s (14%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum comptation speed │ 107.57 MB/s (5%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 295.80 MB/s (14%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 957.99 MB/s (12%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 45.41 MB/s (1%) │
└───────────────────────────────────┴───────────────────┘


with cpu=host :
Uploaded 174 chunks in 5 seconds.
Time per request: 29190 microseconds.
TLS speed: 143.69 MB/s
SHA256 speed: 469.64 MB/s
Compression speed: 342.56 MB/s
Decompress speed: 773.49 MB/s
AES256/GCM speed: 454.77 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 143.69 MB/s (24%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum comptation speed │ 469.64 MB/s (22%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed │ 342.56 MB/s (16%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed │ 773.49 MB/s (10%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed │ 454.77 MB/s (12%) │
└───────────────────────────────────┴───────────────────┘


my other Vm on a xeon 1246v3 / 2 Go / SSD nvme for the storage

Uploaded 756 chunks in 5 seconds.
Time per request: 6634 microseconds.
TLS speed: 632.22 MB/s
SHA256 speed: 481.40 MB/s
Compression speed: 1232.06 MB/s
Decompress speed: 4029.75 MB/s
AES256/GCM speed: 2533.91 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 632.22 MB/s (107%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum comptation speed │ 481.40 MB/s (23%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 1232.06 MB/s (57%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 4029.75 MB/s (50%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 2533.91 MB/s (67%) │
└───────────────────────────────────┴────────────────────┘

 

[Andros]

vm 4 cores of Xeon Gold 6140 2.3Ghz (cpu=skylake-server)
Uploaded 831 chunks in 5 seconds.
Time per request: 6035 microseconds.
TLS speed: 694.90 MB/s
SHA256 speed: 474.73 MB/s
Compression speed: 1302.23 MB/s
Decompress speed: 4416.81 MB/s
AES256/GCM speed: 3395.26 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 694.90 MB/s (118%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum comptation speed │ 474.73 MB/s (22%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 1302.23 MB/s (60%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 4416.81 MB/s (55%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 3395.26 MB/s (89%) │
└───────────────────────────────────┴────────────────────┘

 

[mbosma]

vm 2 cores xeon E2236 (cpu=host)

Uploaded 1136 chunks in 5 seconds.
Time per request: 4418 microseconds.
TLS speed: 949.27 MB/s
SHA256 speed: 606.75 MB/s
Compression speed: 1797.42 MB/s
Decompress speed: 8328.39 MB/s
AES256/GCM speed: 4592.29 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 949.27 MB/s (161%)  │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum comptation speed  │ 606.75 MB/s (29%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 1797.42 MB/s (83%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 8328.39 MB/s (103%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 4592.29 MB/s (121%) │
└───────────────────────────────────┴─────────────────────┘

 

[Egner]

E3-1230v2
on a virtual machine


proxmox-backup-client benchmark --repository backup2
Password for "root@pam": **********
Uploaded 341 chunks in 5 seconds.
Time per request: 14735 microseconds.
TLS speed: 284.65 MB/s
SHA256 speed: 315.07 MB/s
Compression speed: 651.47 MB/s
Decompress speed: 1501.77 MB/s
AES256/GCM speed: 885.17 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 284.65 MB/s (48%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 315.07 MB/s (15%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 651.47 MB/s (30%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 1501.77 MB/s (19%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 885.17 MB/s (23%) │
└───────────────────────────────────┴────────────────────┘

 

[fireon]

proxmox-backup-client benchmark
Password for "xxxxxxxxx@pbs": ******************
Uploaded 147 chunks in 5 seconds.
Time per request: 36374 microseconds.
TLS speed: 115.31 MB/s
SHA256 speed: 1382.31 MB/s
Compression speed: 1525.76 MB/s
Decompress speed: 7045.81 MB/s
AES256/GCM speed: 2523.92 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 115.31 MB/s (20%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1382.31 MB/s (65%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1525.76 MB/s (71%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 7045.81 MB/s (87%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2523.92 MB/s (66%) │
└───────────────────────────────────┴────────────────────┘
proxmox-backup-client benchmark  7,42s user 0,44s system 30% cpu 25,781 total

 

[sukerman]

Locally I get :

┌───────────────────────────────────┬──────────────────────┐
│ Name                              │ Value                │
╞═══════════════════════════════════╪══════════════════════╡
│ TLS (maximal backup upload speed) │ 707.85 MB/s (120%)   │
├───────────────────────────────────┼──────────────────────┤
│ SHA256 checksum computation speed │ 2028.31 MB/s (96%)   │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 compression speed    │ 2298.23 MB/s (106%)  │
├───────────────────────────────────┼──────────────────────┤
│ ZStd level 1 decompression speed  │ 10448.70 MB/s (130%) │
├───────────────────────────────────┼──────────────────────┤
│ AES256 GCM encryption speed       │ 3685.61 MB/s (97%)   │
└───────────────────────────────────┴──────────────────────┘

Over the internet, from one server to another, ping 1.72ms 1Gbps connection

┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 34.98 MB/s (6%)    │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1428.84 MB/s (67%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1132.24 MB/s (52%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 4115.81 MB/s (51%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2168.28 MB/s (57%) │
└───────────────────────────────────┴────────────────────┘

Over the internet, from one server to another, ping 30ms 1Gpbs connection

┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 2.39 MB/s (0%)     │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1428.68 MB/s (67%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1137.23 MB/s (53%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 4142.02 MB/s (51%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2267.90 MB/s (60%) │
└───────────────────────────────────┴────────────────────┘

Is this normal? any tips to increase speed? is latency the limiting factor? or should backups always be local and then transferred via remote_sync?

Thanks !

 

[dcsapak]

Is this normal?

see this bug: https://bugzilla.proxmox.com/show_bug.cgi?id=2983

 

[carsten2]

Is this normal? any tips to increase speed? is latency the limiting factor? or should backups always be local and then transferred via remote_sync?

Remote sync is extremely slow also. I get 500KB/s over a 5MByte/s line.

 

[sukerman]

I'm doing a remote sync now (about 30ms latency 1 Gbps -> 1 Gbps) and getting 18 Mbps, both servers are HDD backed, and there's 20% iowait on the receiving server which to be fair is running a web server, mail server, & database server but the cpu is idling at about 5%.

Tried again from backup server -> new host with Nvme disks, iowait on both boxes is nothing and it's still about 18 Mbps, there's never any load on the sending server, so it seems at this latency this is about max for this type of transfer.

So with these very limited results, with this latency, IMO, it's a lot better than the 2.5 Mbps if you backup/restore to remote servers and worth using remote sync to copy the backups to a remote box. If you have your backup server very close to your pve server with very low latency it's probably a different story.