10G SFP+ Nodes running slow

[bluepr0]

Hi guys!

I've created a cluster with 2 nodes, both are connected at 10Gb through SFP+ using a Unifi 10G Aggregation switch. Both machines are capable enough to sustain a 10G network at full speed, both CPU and SSD wise.

However, when migrating an LXC, I'm getting around 380MB/s, which IMHO is quite slow from the theoretical speed.

1. iperf3 tests show around 9.5G speed
2. fio SSD tests show around 2500-2700MiB/s on both machines for sequential write/read

2024-06-18 20:59:45   Logical volume "vm-104-disk-0" created.
2024-06-18 20:59:49 1165754368 bytes (1.2 GB, 1.1 GiB) copied, 3 s, 389 MB/s
2024-06-18 20:59:52 2329608192 bytes (2.3 GB, 2.2 GiB) copied, 6 s, 388 MB/s
2024-06-18 20:59:55 3478388736 bytes (3.5 GB, 3.2 GiB) copied, 9 s, 386 MB/s
2024-06-18 20:59:58 4681826304 bytes (4.7 GB, 4.4 GiB) copied, 12 s, 390 MB/s
2024-06-18 21:00:01 5826740224 bytes (5.8 GB, 5.4 GiB) copied, 15 s, 388 MB/s
2024-06-18 21:00:04 6851067904 bytes (6.9 GB, 6.4 GiB) copied, 18 s, 381 MB/s
2024-06-18 21:00:07 8022589440 bytes (8.0 GB, 7.5 GiB) copied, 21 s, 382 MB/s
2024-06-18 21:00:10 9207611392 bytes (9.2 GB, 8.6 GiB) copied, 24 s, 384 MB/s
2024-06-18 21:00:13 10349379584 bytes (10 GB, 9.6 GiB) copied, 27 s, 383 MB/s
2024-06-18 21:00:16 11518935040 bytes (12 GB, 11 GiB) copied, 30 s, 384 MB/s
2024-06-18 21:00:19 12711100416 bytes (13 GB, 12 GiB) copied, 33 s, 385 MB/s
2024-06-18 21:00:22 13837467648 bytes (14 GB, 13 GiB) copied, 36 s, 384 MB/s
2024-06-18 21:00:25 14986772480 bytes (15 GB, 14 GiB) copied, 39 s, 384 MB/s
2024-06-18 21:00:28 16193290240 bytes (16 GB, 15 GiB) copied, 42 s, 386 MB/s
2024-06-18 21:00:31 17318739968 bytes (17 GB, 16 GiB) copied, 45 s, 385 MB/s
2024-06-18 21:00:34 18477613056 bytes (18 GB, 17 GiB) copied, 48 s, 385 MB/s
2024-06-18 21:00:37 19629277184 bytes (20 GB, 18 GiB) copied, 51 s, 385 MB/s
2024-06-18 21:00:40 20828651520 bytes (21 GB, 19 GiB) copied, 54 s, 386 MB/s
2024-06-18 21:00:43 21962686464 bytes (22 GB, 20 GiB) copied, 57 s, 385 MB/s
2024-06-18 21:00:46 23162716160 bytes (23 GB, 22 GiB) copied, 60 s, 386 MB/s
2024-06-18 21:00:50 376832+0 records in
2024-06-18 21:00:50 376832+0 records out

Any ideas on what I could be doing wrong, unless this is the expected speed. Both machines are also on the latest Proxmox version as of today.

 

[bluepr0]

MTU set to 9000 is a bit better, but still far away from 10G speeds!

2024-06-18 21:34:22 10442833920 bytes (10 GB, 9.7 GiB) copied, 27 s, 387 MB/s
2024-06-18 21:34:25 11662983168 bytes (12 GB, 11 GiB) copied, 30 s, 389 MB/s
2024-06-18 21:34:28 12918063104 bytes (13 GB, 12 GiB) copied, 33 s, 391 MB/s
2024-06-18 21:34:31 14130479104 bytes (14 GB, 13 GiB) copied, 36 s, 393 MB/s
2024-06-18 21:34:34 15340666880 bytes (15 GB, 14 GiB) copied, 39 s, 393 MB/s
2024-06-18 21:34:37 16575823872 bytes (17 GB, 15 GiB) copied, 42 s, 395 MB/s
2024-06-18 21:34:40 17809997824 bytes (18 GB, 17 GiB) copied, 45 s, 396 MB/s
2024-06-18 21:34:43 19024314368 bytes (19 GB, 18 GiB) copied, 48 s, 396 MB/s
2024-06-18 21:34:46 20210253824 bytes (20 GB, 19 GiB) copied, 51 s, 396 MB/s
2024-06-18 21:34:49 21437743104 bytes (21 GB, 20 GiB) copied, 54 s, 397 MB/s
2024-06-18 21:34:52 22643081216 bytes (23 GB, 21 GiB) copied, 57 s, 397 MB/s
2024-06-18 21:34:55 23891935232 bytes (24 GB, 22 GiB) copied, 60 s, 398 MB/s

 

[Falk R.]

Can you set the Migration Networ to insecure. The Encryption for migration runs only on one core.

 

[bluepr0]

Can you set the Migration Networ to insecure. The Encryption for migration runs only on one core.

umm I could try but I'm not sure how to set that up?

 

[Falk R.]

Go to Datacenter >> Options >> Migration Settings - define your Migration Network here.
Then edit /etc/pve/datacenter.cfg and on the line Migration change secure to insecure.

Then try again.

 

[Maximiliano]

Hello,

• fio SSD tests show around 2500-2700MiB/s on both machines for sequential write/read

For how long did you run the fio benchmarks?

 

[bluepr0]

Go to Datacenter >> Options >> Migration Settings - define your Migration Network here.
Then edit /etc/pve/datacenter.cfg and on the line Migration change secure to insecure.

Then try again.

This actually is doing something! I went from 400MB to 700-800MB. What are the implications of having an insecure migration network?

Hello,



For how long did you run the fio benchmarks?

I'm using these

sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4M --size=4G --readwrite=write --ramp_time=4

sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4M --size=4G --readwrite=read --ramp_time=4

 

[Falk R.]

This actually is doing something! I went from 400MB to 700-800MB. What are the implications of having an insecure migration network?

If an attacker is in this LAN segment, he can theoretically read the complete migration traffic in plain text.
If this is a separate isolated network, it has no effect.

 

[bluepr0]

If an attacker is in this LAN segment, he can theoretically read the complete migration traffic in plain text.
If this is a separate isolated network, it has no effect.

Gotcha! I guess then this is the max speed I would be able to get for the migration?

 

[Falk R.]

Gotcha! I guess then this is the max speed I would be able to get for the migration?

Yes