Edit: SOLVED! - One of Three Hosts Slow Network

C

codix

New Member
Jun 29, 2024
6
1
3
Hello,

I have three identical hosts (PVE 8.2.2), hardware wise clustered. First two (alpha, and beta) behave just fine. Delta, the third takes a long time to pull down ISOs, and this also affects the VM running on it. I did some IPERF tests and it shows that something is weird in Delta. These are trunked with subinterface for the host @ 1gb. I have tried to find the culprit to no effect, this is a new install and I really dont want to have to pull Delta out of the cluster and reinstall if I can avoid it.

ALPHA and BETA are on the same switch. DELTA is on its own switch.

IPERF ALPHA (10.0.99.20) to Beta (10.0.99.21) and Delta (10.0.99.22)

Code: 
root@alpha:~# iperf3 -c 10.0.99.21
Connecting to host 10.0.99.21, port 5201
[  5] local 10.0.99.20 port 43356 connected to 10.0.99.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  95.7 MBytes   802 Mbits/sec    0    321 KBytes
[  5]   1.00-2.00   sec  95.0 MBytes   796 Mbits/sec    0    321 KBytes
[  5]   2.00-3.00   sec  94.0 MBytes   788 Mbits/sec    0    321 KBytes
[  5]   3.00-4.00   sec  94.1 MBytes   790 Mbits/sec    0    351 KBytes
[  5]   4.00-5.00   sec  94.3 MBytes   791 Mbits/sec    0    351 KBytes
[  5]   5.00-6.00   sec  93.5 MBytes   784 Mbits/sec    0    351 KBytes
[  5]   6.00-7.00   sec  93.6 MBytes   785 Mbits/sec    0    351 KBytes
[  5]   7.00-8.00   sec  93.7 MBytes   786 Mbits/sec    0    351 KBytes
[  5]   8.00-9.00   sec  93.5 MBytes   784 Mbits/sec    0    351 KBytes
[  5]   9.00-10.00  sec  94.3 MBytes   791 Mbits/sec    0    351 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   942 MBytes   790 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   940 MBytes   788 Mbits/sec                  receiver

iperf Done.
root@alpha:~# iperf3 -c 10.0.99.22
Connecting to host 10.0.99.22, port 5201
[  5] local 10.0.99.20 port 54470 connected to 10.0.99.22 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.92 MBytes  24.5 Mbits/sec  127   1.41 KBytes
[  5]   1.00-2.00   sec  1.86 MBytes  15.6 Mbits/sec  101   2.83 KBytes
[  5]   2.00-3.00   sec   764 KBytes  6.25 Mbits/sec   52   2.83 KBytes
[  5]   3.00-4.00   sec  2.24 MBytes  18.8 Mbits/sec  104   1.41 KBytes
[  5]   4.00-5.00   sec   891 KBytes  7.30 Mbits/sec   55   1.41 KBytes
[  5]   5.00-6.00   sec  1.62 MBytes  13.6 Mbits/sec   76   1.41 KBytes
[  5]   6.00-7.00   sec  1018 KBytes  8.34 Mbits/sec   61   1.41 KBytes
[  5]   7.00-8.00   sec  2.24 MBytes  18.8 Mbits/sec  123   1.41 KBytes
[  5]   8.00-9.00   sec  4.60 MBytes  38.6 Mbits/sec  235   2.83 KBytes
[  5]   9.00-10.00  sec  1.49 MBytes  12.5 Mbits/sec   79   2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  19.6 MBytes  16.4 Mbits/sec  1013             sender
[  5]   0.00-10.00  sec  19.4 MBytes  16.2 Mbits/sec                  receiver

IPERF BETA (10.0.99.21) to Alpha (10.0.99.20) and Delta (10.0.99.22)

Code: 
root@beta:~# iperf3 -c 10.0.99.20
Connecting to host 10.0.99.20, port 5201
[  5] local 10.0.99.21 port 53656 connected to 10.0.99.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  97.3 MBytes   817 Mbits/sec    0    331 KBytes
[  5]   1.00-2.00   sec  96.0 MBytes   805 Mbits/sec    0    331 KBytes
[  5]   2.00-3.00   sec  95.6 MBytes   802 Mbits/sec    0    331 KBytes
[  5]   3.00-4.00   sec  96.3 MBytes   807 Mbits/sec    0    331 KBytes
[  5]   4.00-5.00   sec  95.5 MBytes   801 Mbits/sec    0    331 KBytes
[  5]   5.00-6.00   sec  95.2 MBytes   799 Mbits/sec    0    331 KBytes
[  5]   6.00-7.00   sec  95.8 MBytes   803 Mbits/sec    0    348 KBytes
[  5]   7.00-8.00   sec  96.0 MBytes   805 Mbits/sec    0    348 KBytes
[  5]   8.00-9.00   sec  95.8 MBytes   804 Mbits/sec    0    348 KBytes
[  5]   9.00-10.00  sec  95.4 MBytes   801 Mbits/sec    0    348 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   959 MBytes   804 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   958 MBytes   803 Mbits/sec                  receiver

iperf Done.
root@beta:~# iperf3 -c 10.0.99.22
Connecting to host 10.0.99.22, port 5201
[  5] local 10.0.99.21 port 35022 connected to 10.0.99.22 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.98 MBytes  16.6 Mbits/sec   96   1.41 KBytes
[  5]   1.00-2.00   sec  2.86 MBytes  24.0 Mbits/sec  158   2.83 KBytes
[  5]   2.00-3.00   sec   700 KBytes  5.73 Mbits/sec   38   1.41 KBytes
[  5]   3.00-4.00   sec  63.6 KBytes   521 Kbits/sec    7   1.41 KBytes
[  5]   4.00-5.00   sec   954 KBytes  7.82 Mbits/sec   48   1.41 KBytes
[  5]   5.00-6.00   sec  1.12 MBytes  9.38 Mbits/sec   64   1.41 KBytes
[  5]   6.00-7.00   sec  4.04 MBytes  33.9 Mbits/sec  194   1.41 KBytes
[  5]   7.00-8.00   sec  4.10 MBytes  34.4 Mbits/sec  200   1.41 KBytes
[  5]   8.00-9.00   sec  1.37 MBytes  11.5 Mbits/sec   74   1.41 KBytes
[  5]   9.00-10.00  sec  3.23 MBytes  27.1 Mbits/sec  165   2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  20.4 MBytes  17.1 Mbits/sec  1044             sender
[  5]   0.00-10.00  sec  20.2 MBytes  16.9 Mbits/sec                  receiver

IPERF DELTA (10.0.99.22) to Alpha (10.0.99.20) and Beta (10.0.99.21)

Code: 
root@delta:~# iperf3 -c 10.0.99.20
Connecting to host 10.0.99.20, port 5201
[  5] local 10.0.99.22 port 49588 connected to 10.0.99.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  97.6 MBytes   819 Mbits/sec    4    291 KBytes
[  5]   1.00-2.00   sec  96.1 MBytes   806 Mbits/sec    1    288 KBytes
[  5]   2.00-3.00   sec  95.7 MBytes   803 Mbits/sec    3    211 KBytes
[  5]   3.00-4.00   sec  96.3 MBytes   807 Mbits/sec    7    157 KBytes
[  5]   4.00-5.00   sec  94.8 MBytes   795 Mbits/sec    8    245 KBytes
[  5]   5.00-6.00   sec  96.8 MBytes   812 Mbits/sec    0    321 KBytes
[  5]   6.00-7.00   sec  96.3 MBytes   808 Mbits/sec    4    263 KBytes
[  5]   7.00-8.00   sec  96.1 MBytes   806 Mbits/sec    4    161 KBytes
[  5]   8.00-9.00   sec  95.8 MBytes   804 Mbits/sec    2    185 KBytes
[  5]   9.00-10.00  sec  96.1 MBytes   806 Mbits/sec    4    216 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   962 MBytes   807 Mbits/sec   37             sender
[  5]   0.00-10.00  sec   960 MBytes   805 Mbits/sec                  receiver

iperf Done.
root@delta:~# iperf3 -c 10.0.99.21
Connecting to host 10.0.99.21, port 5201
[  5] local 10.0.99.22 port 36866 connected to 10.0.99.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  97.7 MBytes   819 Mbits/sec    1    300 KBytes
[  5]   1.00-2.00   sec  96.1 MBytes   806 Mbits/sec    1    225 KBytes
[  5]   2.00-3.00   sec  96.0 MBytes   805 Mbits/sec    2    276 KBytes
[  5]   3.00-4.00   sec  95.5 MBytes   801 Mbits/sec    5    195 KBytes
[  5]   4.00-5.00   sec  95.6 MBytes   802 Mbits/sec    5    215 KBytes
[  5]   5.00-6.00   sec  96.2 MBytes   807 Mbits/sec    4    266 KBytes
[  5]   6.00-7.00   sec  99.4 MBytes   834 Mbits/sec    3    298 KBytes
[  5]   7.00-8.00   sec  96.6 MBytes   811 Mbits/sec    7    182 KBytes
[  5]   8.00-9.00   sec  98.0 MBytes   822 Mbits/sec    4    123 KBytes
[  5]   9.00-10.00  sec  97.6 MBytes   818 Mbits/sec    2    221 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   969 MBytes   813 Mbits/sec   34             sender
[  5]   0.00-10.00  sec   967 MBytes   811 Mbits/sec                  receiver

ALPHA ethtool and interface config:

Code: 
root@alpha:~# ethtool enp0s31f6
Settings for enp0s31f6:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes


INTERFACES

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp0s31f6
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.99
iface vmbr0.99 inet static
        address 10.0.99.20/24
        gateway 10.0.99.254

source /etc/network/interfaces.d/*

BETA ethtool and interface config:

Code: 
root@beta:~# ethtool enp0s31f6
Settings for enp0s31f6:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

INTERFACES

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp0s31f6
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.99
iface vmbr0.99 inet static
        address 10.0.99.21/24
        gateway 10.0.99.254

source /etc/network/interfaces.d/*

DELTA ethtool and interface config:

Code: 
root@delta:~# ethtool enp0s31f6
Settings for enp0s31f6:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: off (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes


INTERFACES

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

auto vmbr0
iface vmbr0 inet static
        bridge-ports enp0s31f6
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.99
iface vmbr0.99 inet static
        address 10.0.99.22/24
        gateway 10.0.99.254

source /etc/network/interfaces.d/*

I know what youre going to say, its something in the switch fabric. Except theres an ESXI host on the same switch as DELTA and iperf between the guest on the ESXI host and guest on ALPHA and BETA works fine.

Guest on ESXI host (same switch as DELTA) to guest on ALPHA (different switch)
Code: 
root@plexbox-l:~$ iperf3 -c 10.0.10.10
Connecting to host 10.0.10.10, port 5201
[  5] local 10.0.10.200 port 59720 connected to 10.0.10.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.07 GBytes   919 Mbits/sec   39             sender
[  5]   0.00-10.00  sec  1.07 GBytes   917 Mbits/sec                  receiver

Guest on ESXI host (same switch as DELTA) to guest on BETA (different switch)

Code: 
root@plexbox-l:~$ iperf3 -c 10.0.10.11
Connecting to host 10.0.10.11, port 5201
[  5] local 10.0.10.200 port 53216 connected to 10.0.10.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.06 GBytes   915 Mbits/sec   40             sender
[  5]   0.00-10.00  sec  1.06 GBytes   913 Mbits/sec                  receiver
 
Last edited:
Also, a download of a Rocky ISO image (1.7g) on Delta:

Connecting to download.rockylinux.org (download.rockylinux.org)|151.101.150.132|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1829634048 (1.7G) [application/octet-stream]
Saving to: '/var/lib/vz/template/iso/Rocky-9.4-x86_64-minimal.iso.tmp_dwnl.775547'
0K ........ ........ ........ ........ 1% 69.2K 7h2m
 
One thing I noticed is that the interfaces in ALPHA and BETA are set to manual, but DELTA is set to static. Not sure why it was set differently and cant find a reason why it would be different but will try to change it manually and reboot.
 
That apparently was it. After host came back online speed tests are working normally,


Code: 
Connecting to host 10.0.99.20, port 5201
[  5] local 10.0.99.22 port 40452 connected to 10.0.99.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   101 MBytes   844 Mbits/sec    0    346 KBytes       
[  5]   1.00-2.00   sec  98.3 MBytes   825 Mbits/sec    1    273 KBytes       
[  5]   2.00-3.00   sec  98.6 MBytes   827 Mbits/sec    0    318 KBytes       
[  5]   3.00-4.00   sec  98.7 MBytes   828 Mbits/sec    3    301 KBytes       
[  5]   4.00-5.00   sec  97.5 MBytes   818 Mbits/sec    0    321 KBytes       
[  5]   5.00-6.00   sec  98.4 MBytes   825 Mbits/sec    3    173 KBytes       
[  5]   6.00-7.00   sec  97.9 MBytes   822 Mbits/sec    2    301 KBytes       
[  5]   7.00-8.00   sec  99.0 MBytes   830 Mbits/sec    2    298 KBytes       
[  5]   8.00-9.00   sec  98.5 MBytes   826 Mbits/sec    0    320 KBytes       
[  5]   9.00-10.00  sec  98.7 MBytes   828 Mbits/sec    0    320 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   986 MBytes   827 Mbits/sec   11             sender
[  5]   0.00-10.00  sec   985 MBytes   826 Mbits/sec                  receiver

iperf Done.
root@delta:~# iperf3 -c 10.0.99.21
Connecting to host 10.0.99.21, port 5201
[  5] local 10.0.99.22 port 46170 connected to 10.0.99.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  99.6 MBytes   836 Mbits/sec    0    344 KBytes       
[  5]   1.00-2.00   sec  97.3 MBytes   816 Mbits/sec    0    344 KBytes       
[  5]   2.00-3.00   sec  98.4 MBytes   826 Mbits/sec    1    283 KBytes       
[  5]   3.00-4.00   sec  98.5 MBytes   826 Mbits/sec    3    280 KBytes       
[  5]   4.00-5.00   sec  98.9 MBytes   830 Mbits/sec    2    281 KBytes       
[  5]   5.00-6.00   sec  98.4 MBytes   825 Mbits/sec    1    304 KBytes       
[  5]   6.00-7.00   sec  99.1 MBytes   831 Mbits/sec    2    247 KBytes       
[  5]   7.00-8.00   sec  97.8 MBytes   821 Mbits/sec    2    290 KBytes       
[  5]   8.00-9.00   sec  98.1 MBytes   823 Mbits/sec    3    260 KBytes       
[  5]   9.00-10.00  sec  98.8 MBytes   829 Mbits/sec    4    221 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   985 MBytes   826 Mbits/sec   18             sender
[  5]   0.00-10.00  sec   984 MBytes   825 Mbits/sec                  receiver

I wasnt sure what the difference was so I found this on the internet on the difference between inet static and inet manual. Not sure how it was working, or how it got set that way but that appears to have been the culprit. I apparently just needed to write everything out to think through and compare the different configs and tests to catch the discrepancy.

inet static – Defines a static IP address.

inet manual – Does not define an IP address for an interface. Generally used by interfaces that are bridge or aggregation members, interfaces that need to operate in promiscuous mode (e.g. port mirroring or network TAPs), or have a VLAN device configured on them. It's a way to keep the interface up without an IP address.
Click to expand...
 
You must log in or register to reply here.
Top Bottom