Dual 10GB, weird behaviour

[kosta88]

Hello,
i just upgraded my homelab to 2x 10GBit links, and yet I am getting inconsistent results.
It's like this:
I have Protectli VP6630 with OPNsense, which has two SFP+, connected to the Aruba Instant On 1930, which has 4 SFP+ 10GB, and then Proxmox with its new X710 dual 25Gbit card (running at 10GBit, automatically, I guess).

I am using l3+l4 hashing everywhere, firewall, switch and proxmox.

The issue I am seeing: no matter what I do, it won't go beyond around 14GBit. That is kinda hard to accept... but what is more weird:
I am running iperf3 on 6 separate VMs/LXCs. Some are VMs, some are LXC. But they are in 3 separate VLANs, because I want to go over the firewall.
I also run iperf3 with -P 100.
The weird thing is:
1) VLAN30 and VLAN40 -> max 8GBit solo
2) VLAN40 and VLAN50 -> 14GBit solo
3) VLAN50 to VLAN30 -> max 6GBit solo
If I run simult, they will drop to a max 14GBit all together, but still link 2) will steal the most of bandwidth.

So I have two issues:
- unbalanced behaviour
- max of 14GBit

Do you have any recommendations that I could try?

Thanks

 

[weehooey-bh]

Some suggestions:

• You can use bmon on your PVE host to determine which interface is handling the traffic.
• When you run iperf3, use port numbers other than the default.
• Disconnect one of the links and test again. Do you get the speeds you are expecting with just one link?

Based on what you are describing, it would seem one or more devices are ignoring the layer 4.