10gb ethernet to 10gb ethernet slow

[yjjoe]

Hi,

For some reason my transfer speed seems to be limited to 1gb even tho both my pbs server and my pve node are only using 10gb sfp+ ethernet card.

How can I debug why I end up with slow restore speed from my pve node? I've tried several stuff with no success of understanding what's the culprit.

Thx

 

[Dunuin]

First I would do a iperf3 test to see if the connection is slow in general or if PVE/PBS, CPU or even the storage is the bottleneck.

 

[yjjoe]

I'm trying to restore a 300gb VM but it takes more than 40min to be done. I'll try running 'iperf3 -s -f K' after the actual restore is done.

BTW, my CPU is a '56 x Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz (2 Sockets)'

Thx for your insight

 

[Dunuin]

So a quite slow CPU. Is your PBS using local SSDs? And is your PVE using enterprise SSDs or at least TLC consumer SSD? If not it is probably a slow storage.
To benchmark PBS performance you could run a proxmox-backup-client benchmark --repository YourDatastore from your PBS and a proxmox-backup-client benchmark --repository root@pam@YOUR.PBS.IP.ADDRESS:8007:YourDatastore from your PVE.

 

[yjjoe]

My PBS pool is a zpool2 from 12 'HGST 4tb' Sas drives. PBS OS is installed on a sata SSD.

I have to wait for my actual restore that is on-going and than I'll be running proxmox-backup-client benchmark

 

[yjjoe]

proxmox-backup-client benchmark --repository PBS-Pool
Password for "root@pam": *************
Uploaded 840 chunks in 5 seconds.
Time per request: 5969 microseconds.
TLS speed: 702.68 MB/s
SHA256 speed: 385.38 MB/s
Compression speed: 366.84 MB/s
Decompress speed: 543.33 MB/s
AES256/GCM speed: 1079.69 MB/s
Verify speed: 223.10 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 702.68 MB/s (57%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 385.38 MB/s (19%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 366.84 MB/s (49%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 543.33 MB/s (45%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 223.10 MB/s (29%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1079.69 MB/s (30%) │

 

[yjjoe]

And from PVE:

proxmox-backup-client benchmark --repository root@pam@10.255.10.196:8007BS-Pool
Password for "root@pam": *************
fingerprint: 71:ff:eb:f9:95:7f:eb:53:f8:58:87:6e:f8:61:53:fb:78:1e:09:91:a8:87:0e:0c:a0:5f:5d:9e:16:18:a3:50
Are you sure you want to continue connecting? (y/n):
Are you sure you want to continue connecting? (y/n): y
fingerprint: 71:ff:eb:f9:95:7f:eb:53:f8:58:87:6e:f8:61:53:fb:78:1e:09:91:a8:87:0e:0c:a0:5f:5d:9e:16:18:a3:50
Are you sure you want to continue connecting? (y/n): y
Uploaded 148 chunks in 5 seconds.
Time per request: 35657 microseconds.
TLS speed: 117.63 MB/s
SHA256 speed: 419.42 MB/s
Compression speed: 404.47 MB/s
Decompress speed: 609.28 MB/s
AES256/GCM speed: 1110.88 MB/s
Verify speed: 246.77 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 117.63 MB/s (10%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 419.42 MB/s (21%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 404.47 MB/s (54%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 609.28 MB/s (51%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 246.77 MB/s (33%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1110.88 MB/s (30%) │

 

[Dunuin]

Then it looks like you are really limited to 1Gbit by the network. Your PBS storage can handle roughly 2Gbit.
Next I would check if both NIC have auto negotiated 1Gbit speed instead of 10Gbit.

And I understand you correctly that no 1Gbit Ethernet NIC is in use at all? Not even for management?

Did you verify that you are actually using SFP+ DAC or tranceivers and not just SFP ones by googling part numbers?

 

[yjjoe]

Hi, Thanks you so much for your help!

All my 10g NIC are hooked up using SFP+ DAC cables (connected to my 16 10g ports switch). I have to figure out how I can look at the auto-negotiation in the context of proxmox.

BTW, any hint about that sweet benchmark command in proxmox pve? ('proxmox-backup-client benchmark' like but for proxmox?)

Cheerios

 

[Dunuin]

PVE got a pveperf /path/to/mounted/filesystem for some basic benchmarks.
But if you really want to do proper storage benchmarks you should install fio: https://fio.readthedocs.io/en/latest/
And iperf3 for network benchmarks: https://iperf.fr/iperf-doc.php