Half network speed one direction between VM's different hosts

[etnicor]

Hello,
I have 2 proxmox hosts with 10gigabit nw cards.

If i run iperf3 --bidir between the proxmox hosts i get full line speed both ways.


However if I setup one vm on both hosts and run iperf3 inside vm's on a VLAN I get full line speed in one direction but half the speed in the other direction. Same result if I do not use --bidir but test one way at the time.

VM's use host cpu and virtio and no firewall enabled.

Since it's on same VLAN my opnsense box should not be involved so that should not be the culprit.


I can't figure this out. Would make more sense if speed was bad both ways.



EDIT: I guess next step would be to setup a physical client on same VLAN and test against each vm on each proxmox host.

 

[sb-jw]

You could check whether the problem also exists when both VMs are on the same node.

 

[etnicor]

You could check whether the problem also exists when both VMs are on the same node.

Hello, have tested that and then it was no issue. Got ~40gbps between vm's then, guess the traffic never leaves the bridge.

 

[sb-jw]

Have you compared the configuration of your two nodes and switch ports in detail? Maybe you have set something different for the bridge.

Sometimes a reboot helps, have you just tried that to see if it works afterwards? Maybe something got stuck or packages weren't used correctly, etc.

 

[etnicor]

Tested adding a physical client on same VLAN and got line speed to both vm's. Was abit lower when receiving.

interface config is pretty much the same on both hosts, only thing which I have added to the bridge is to make it VLAN aware.


Will reboot stuff this evening, including switch.

 

[etnicor]

Update: Tested Reboot proxmox hosts and switch. No difference. =/

Output from a bidir run between 2 vm's.

[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-1.00   sec   311 MBytes  2.60 Gbits/sec  184    639 KBytes
[  7][RX-C]   0.00-1.00   sec  1.05 GBytes  9.05 Gbits/sec
[  5][TX-C]   1.00-2.00   sec   301 MBytes  2.53 Gbits/sec   23    779 KBytes
[  7][RX-C]   1.00-2.00   sec  1.08 GBytes  9.28 Gbits/sec
[  5][TX-C]   2.00-3.00   sec   398 MBytes  3.33 Gbits/sec    0   1.07 MBytes
[  7][RX-C]   2.00-3.00   sec  1.09 GBytes  9.32 Gbits/sec
[  5][TX-C]   3.00-4.00   sec   499 MBytes  4.18 Gbits/sec    0   1.38 MBytes
[  7][RX-C]   3.00-4.00   sec  1.09 GBytes  9.34 Gbits/sec
[  5][TX-C]   4.00-5.00   sec   620 MBytes  5.20 Gbits/sec    2   1.67 MBytes
[  7][RX-C]   4.00-5.00   sec  1.08 GBytes  9.32 Gbits/sec
[  5][TX-C]   5.00-6.00   sec   886 MBytes  7.43 Gbits/sec   17   2.00 MBytes
[  7][RX-C]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec
[  5][TX-C]   6.00-7.00   sec   918 MBytes  7.70 Gbits/sec    1   2.31 MBytes
[  7][RX-C]   6.00-7.00   sec  1.09 GBytes  9.37 Gbits/sec
[  5][TX-C]   7.00-8.00   sec   872 MBytes  7.32 Gbits/sec  673   1.27 MBytes
[  7][RX-C]   7.00-8.00   sec  1.08 GBytes  9.30 Gbits/sec
[  5][TX-C]   8.00-9.00   sec   459 MBytes  3.85 Gbits/sec   64   1.12 MBytes
[  7][RX-C]   8.00-9.00   sec  1.09 GBytes  9.36 Gbits/sec
[  5][TX-C]   9.00-10.00  sec   494 MBytes  4.14 Gbits/sec    0   1.42 MBytes
[  7][RX-C]   9.00-10.00  sec  1.09 GBytes  9.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  5.62 GBytes  4.83 Gbits/sec  964             sender
[  5][TX-C]   0.00-10.00  sec  5.62 GBytes  4.83 Gbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  10.8 GBytes  9.31 Gbits/sec  25699             sender
[  7][RX-C]   0.00-10.00  sec  10.8 GBytes  9.30 Gbits/sec                  receiver

Output if I just run directly between proxmox hosts.

[  5][TX-C]   0.00-1.00   sec  1.09 GBytes  9.33 Gbits/sec  135   1.56 MBytes
[  7][RX-C]   0.00-1.00   sec  1.06 GBytes  9.14 Gbits/sec
[  5][TX-C]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec    4   1.66 MBytes
[  7][RX-C]   1.00-2.00   sec  1.09 GBytes  9.35 Gbits/sec
[  5][TX-C]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.71 MBytes
[  7][RX-C]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec
[  5][TX-C]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec    1   1.83 MBytes
[  7][RX-C]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5][TX-C]   4.00-5.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.83 MBytes
[  7][RX-C]   4.00-5.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5][TX-C]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.83 MBytes
[  7][RX-C]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5][TX-C]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec    3   1.95 MBytes
[  7][RX-C]   6.00-7.00   sec  1.09 GBytes  9.38 Gbits/sec
[  5][TX-C]   7.00-8.00   sec  1.09 GBytes  9.41 Gbits/sec    0   2.05 MBytes
[  7][RX-C]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5][TX-C]   8.00-9.00   sec  1.09 GBytes  9.39 Gbits/sec    1   2.05 MBytes
[  7][RX-C]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5][TX-C]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec    0   2.08 MBytes
[  7][RX-C]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec  144             sender
[  5][TX-C]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  10.9 GBytes  9.37 Gbits/sec  1150             sender
[  7][RX-C]   0.00-10.00  sec  10.9 GBytes  9.37 Gbits/sec                  receiver

 

[etnicor]

I think I have found that the issue only arises when I have a VM which I have moved between VLAN's. I guess something remembers the path/mac and something go bonkers..

 

[sb-jw]

That seems somehow illogical to me. Did you route the VLANs differently? Do you only have one link at a time or do you have LACP active?

 

[etnicor]

That seems somehow illogical to me. Did you route the VLANs differently? Do you only have one link at a time or do you have LACP active?

Don't have LACP. I think my VLAN assumption was wrong after some more testing.

Have also tested setting multiqueue on virtio interface, but that just made it worse.

 

[etnicor]

Tested changing to another switch(mikrotik) and disconnected my router/firewall just to be sure it did not interfere.

Have tested to explicity disable hardware vlan offload on both nics(x520-da2, x710-da2).

Onlything left is linux bridge or virtio nic's.