Ability to Disable TLS transport

[tt2468]

Hi all,

I'm in a situation where I'm being heavily bottlenecked by my PBS's TLS speed. I'm running PBS as a VM on a Synology RS1221RP+, and when I run a benchmark against the repository, the TLS speed is the clear bottleneck. Additionally, the CPU usage of the PBS vm gets pinned to 100% during backups. I can directly affect TLS speed by adding vCPUs to the PBS VM.

Is it possible to disable the TLS transport for PBS, like you can do for example with migrations in PVE? We're running this in a datacenter environment. If someone were to gain the ability to intercept our internal network traffic, I would have far bigger worries than whether or not they're siphoning our backups data.

Benchmark test results:

root@prox-cpu1:~# proxmox-backup-client benchmark --repository root@pam@[pbsip]:storage
Uploaded 436 chunks in 5 seconds.
Time per request: 11601 microseconds.
TLS speed: 361.53 MB/s
SHA256 speed: 1914.23 MB/s
Compression speed: 752.73 MB/s
Decompress speed: 992.74 MB/s
AES256/GCM speed: 2304.00 MB/s
Verify speed: 648.16 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 361.53 MB/s (29%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1914.23 MB/s (95%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 752.73 MB/s (100%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 992.74 MB/s (83%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 648.16 MB/s (86%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2304.00 MB/s (63%) │
└───────────────────────────────────┴────────────────────┘

 

[ggoller]

Hi!
this is not currently possible.

 

[lucius_the]

+1 to this... TLS is a huge bottleneck for me as well.

TLS speed: 169.14 MB/s
SHA256 speed: 429.66 MB/s
Compression speed: 418.15 MB/s
Decompress speed: 619.55 MB/s
AES256/GCM speed: 1164.14 MB/s
Verify speed: 257.23 MB/s

Although I'm not sure why that is. TLS per se should not be this slow. What is actually measured under "TLS speed" test ?

 

[_gabriel]

don't forget PBS datastore is millions of small files so HDD will slowdown before TLS limit.
if already SSD, you can try installing PBS alongside PVE keeping NAS as NFS or cifs datastore.
it's not recommended but if you know what you do, it's works great.

 

[fabian]

the TLS benchmark tests both TLS and HTTP/2.0 performance - it basically tests uploading chunks to the server (which immediately discards them, so no storage involved on either end). HTTP/2.0 can be slower than expected if your link has higher latency.

 

[lucius_the]

Hm... 10 Gbps link, I believe latency should be great. But... still surprised that TLS would slow it down this much. Oh well.

 

[tomstephens89]

I would also like to see if TLS transport could be disabled for PBS connections.

I am using fast enterprise hardware with all flash storage, PBS on 2 x 40Gb NIC, hosts on 2 x 10Gb, recent Xeon gold & platinum CPU's but these are the benchmark numbers.

Be nice to see if I can speed this up. Using SCP I can hit 500MB/s sustained, obviously there is TLS overhead there, iPerf will max out the single 10Gbit NIC's at a time.

 

[lucius_the]

I agree, it would be really nice if we had an option to turn off TLS for backup transport. Can't really see any good reason not to offer that as an option at least. My backups run over a dedicated VLAN in a controlled environment... not sure why TLS is obligatory... authentication could still use TLS (get some single-operation ticket through that) but the whole backup transfer slows things down and it should be possible to skip TLS if we want to speed things up, in controlled environments, understanding risks, etc.