10gb ethernet to 10gb ethernet slow

yjjoe

Member
Dec 27, 2021
34
2
13
24
Hi,

For some reason my transfer speed seems to be limited to 1gb even tho both my pbs server and my pve node are only using 10gb sfp+ ethernet card.

How can I debug why I end up with slow restore speed from my pve node? I've tried several stuff with no success of understanding what's the culprit.

Thx
 
First I would do a iperf3 test to see if the connection is slow in general or if PVE/PBS, CPU or even the storage is the bottleneck.
 
I'm trying to restore a 300gb VM but it takes more than 40min to be done. I'll try running 'iperf3 -s -f K' after the actual restore is done.

BTW, my CPU is a '56 x Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz (2 Sockets)'

Thx for your insight ;)
 
So a quite slow CPU. Is your PBS using local SSDs? And is your PVE using enterprise SSDs or at least TLC consumer SSD? If not it is probably a slow storage.
To benchmark PBS performance you could run a proxmox-backup-client benchmark --repository YourDatastore from your PBS and a proxmox-backup-client benchmark --repository root@pam@YOUR.PBS.IP.ADDRESS:8007:YourDatastore from your PVE.
 
  • Like
Reactions: Fathi
My PBS pool is a zpool2 from 12 'HGST 4tb' Sas drives. PBS OS is installed on a sata SSD.

I have to wait for my actual restore that is on-going and than I'll be running proxmox-backup-client benchmark
 
Last edited:
proxmox-backup-client benchmark --repository PBS-Pool
Password for "root@pam": *************
Uploaded 840 chunks in 5 seconds.
Time per request: 5969 microseconds.
TLS speed: 702.68 MB/s
SHA256 speed: 385.38 MB/s
Compression speed: 366.84 MB/s
Decompress speed: 543.33 MB/s
AES256/GCM speed: 1079.69 MB/s
Verify speed: 223.10 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 702.68 MB/s (57%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 385.38 MB/s (19%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 366.84 MB/s (49%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 543.33 MB/s (45%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 223.10 MB/s (29%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1079.69 MB/s (30%) │
 
And from PVE:

proxmox-backup-client benchmark --repository root@pam@10.255.10.196:8007:pBS-Pool
Password for "root@pam": *************
fingerprint: 71:ff:eb:f9:95:7f:eb:53:f8:58:87:6e:f8:61:53:fb:78:1e:09:91:a8:87:0e:0c:a0:5f:5d:9e:16:18:a3:50
Are you sure you want to continue connecting? (y/n):
Are you sure you want to continue connecting? (y/n): y
fingerprint: 71:ff:eb:f9:95:7f:eb:53:f8:58:87:6e:f8:61:53:fb:78:1e:09:91:a8:87:0e:0c:a0:5f:5d:9e:16:18:a3:50
Are you sure you want to continue connecting? (y/n): y
Uploaded 148 chunks in 5 seconds.
Time per request: 35657 microseconds.
TLS speed: 117.63 MB/s
SHA256 speed: 419.42 MB/s
Compression speed: 404.47 MB/s
Decompress speed: 609.28 MB/s
AES256/GCM speed: 1110.88 MB/s
Verify speed: 246.77 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 117.63 MB/s (10%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 419.42 MB/s (21%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 404.47 MB/s (54%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 609.28 MB/s (51%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 246.77 MB/s (33%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1110.88 MB/s (30%) │
 
Then it looks like you are really limited to 1Gbit by the network. Your PBS storage can handle roughly 2Gbit.
Next I would check if both NIC have auto negotiated 1Gbit speed instead of 10Gbit.

And I understand you correctly that no 1Gbit Ethernet NIC is in use at all? Not even for management?

Did you verify that you are actually using SFP+ DAC or tranceivers and not just SFP ones by googling part numbers?
 
Last edited:
  • Like
Reactions: Fathi
Hi, Thanks you so much for your help!

All my 10g NIC are hooked up using SFP+ DAC cables (connected to my 16 10g ports switch). I have to figure out how I can look at the auto-negotiation in the context of proxmox.

BTW, any hint about that sweet benchmark command in proxmox pve? ('proxmox-backup-client benchmark' like but for proxmox?)

Cheerios
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!