Linux bridge/bond/nic(s) intermittent behavior (?)

D

dainjaz

New Member
May 20, 2024
4
1
3
I'm having an issue with a network 'linux bridge' or 'bond' in PVE/Debian. I have recently migrated from ESXi 7.0u3 to Proxmox 8.4. I am NEW to Proxmox bonds/bridges but I 'think' I have set it up correctly. (?) I setup PVE 8.4 on a temporary old desktop with (1x 1G NICs and 2x 2.5G NICs) in a similar setup to the faster/newer one output below and all of that seemed to work on the older one. I brought up the guests one at a time in the temporary PVE to test each VM as it really only has enough RAM to run 1-2 VMs simultaneously. I then installed Proxmox 8.4 on my workstation-server that has a lot more hardware resources including the a similar (1x 1G NICs and 2x 2.5G NICs - output of config below), followed the 'adding-full-disk-encryption-to-proxmox.137051' to encrypt and that seemed to mostly work. The issue I'm having is getting network to the guests on the newer hardware. The vmbr0 works. I can attach (I think) any guest to it, they all connect. The issue is with the bonded 2x 2.5G NICs (enp10s0, enp11s0) attached to vmbr1. My only windows (w11) guest successfully connects to network on vmbr1 using VMware vmxnet3 so I know that the vmbr1 bridge partially is working. My linux guests, however, only seem to work on vmbr0, which is the slower 1G interface and I need them to work on vmbr1. I tried a number of different virt NICs on two different Linux guests (HAOS 15.2 and an ubuntu24.04 workstation guest) neither get network via vmbr1, but DO get network via vmbr0, so I know the guest's network configs are good, it's either the NIC, bond, or bridge that's the issue. I'd have thought maybe hardware incompatibility or something, but the windows guest connects... (?). All output in this post is from the faster/permanent PVE host.

Ideas:
1. Note that the MAC for enp10s0 enp11s0 are the same... is that because it's bonded? Is this expected behavior?
2. All the VMs were migrated from VMware (via mounting) --> old desktop PVE 8.4 (via backup/restore) --> faster/permanent PVE 8.4.
2. Could this be a compatibility or driver issue? (output below)
3. I could attempt to swap the 2x 2.5G NIC in the temp PVE and faster/permanent PVE (although both are the same hardware brand/model/card/driver)?
4. I attempted to delete the bond and vmbr1 and recreate them, and I saw something work, but then didnt work... I didnt have the wherewithal to document exactly the behavior. I could repeat that experiment if we need more data. Maybe it's intermittent, or cuts out the vmbr1 after some minutes? Not certain on this.
5. I don't know how to test the bridge/bond itself without using a guest to do it, if anyone knows a good method?

Code: 
# ls -l /sys/class/net/*/device/driver/module | cut -d/ -f5,13 | sed 's?/? -> ?'
enp0s25 -> e1000e
enp10s0 -> r8169
enp11s0 -> r8169

Code: 
# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp0s25
iface enp0s25 inet manual
#NIC - Management

auto enp10s0
iface enp10s0 inet manual
#NIC - Production

auto enp11s0
iface enp11s0 inet manual
#NIC - Production

auto bond0
iface bond0 inet manual
        bond-slaves enp10s0 enp11s0
        bond-miimon 100
        bond-mode balance-rr
#Bond - Production 2x 2.5G

auto vmbr0
iface vmbr0 inet static
        address 10.0.0.10/24
        gateway 10.0.0.1
        bridge-ports enp0s25
        bridge-stp off
        bridge-fd 0
#Sw Management Network

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
#Sw Production Network

source /etc/network/interfaces.d/*

Code: 
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp10s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 98:b7:85:20:d6:74 brd ff:ff:ff:ff:ff:ff
3: enp11s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 98:b7:85:20:d6:74 brd ff:ff:ff:ff:ff:ff permaddr 98:b7:85:20:d6:75
4: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 48:4d:7e:de:07:06 brd ff:ff:ff:ff:ff:ff
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 48:4d:7e:de:07:06 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::4a4d:7eff:fede:706/64 scope link
       valid_lft forever preferred_lft forever
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether 98:b7:85:20:d6:74 brd ff:ff:ff:ff:ff:ff
7: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 98:b7:85:20:d6:74 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9ab7:85ff:fe20:d674/64 scope link
       valid_lft forever preferred_lft forever
24: tap1140i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 56:33:05:ab:6f:79 brd ff:ff:ff:ff:ff:ff
32: tap1100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 2e:c4:ae:df:3e:08 brd ff:ff:ff:ff:ff:ff
35: tap1015i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether de:f7:4f:37:ff:0b brd ff:ff:ff:ff:ff:ff
 
Last edited:
Code: 
# lspci
00:00.0 Host bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 (rev 02)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:01.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
00:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)
00:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 02)
00:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)
00:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 02)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation C610/X99 series chipset HD Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.1 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #2 (rev d5)
00:1c.2 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #3 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 171X (rev 03)
03:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1)
03:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 172Xa/172Xb (rev 01)
06:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
08:00.0 PCI bridge: ASMedia Technology Inc. ASM1806 4-Port PCIe x2 Gen2 Packet Switch (rev 01)
09:00.0 PCI bridge: ASMedia Technology Inc. ASM1806 4-Port PCIe x2 Gen2 Packet Switch (rev 01)
09:02.0 PCI bridge: ASMedia Technology Inc. ASM1806 4-Port PCIe x2 Gen2 Packet Switch (rev 01)
09:06.0 PCI bridge: ASMedia Technology Inc. ASM1806 4-Port PCIe x2 Gen2 Packet Switch (rev 01)
09:0e.0 PCI bridge: ASMedia Technology Inc. ASM1806 4-Port PCIe x2 Gen2 Packet Switch (rev 01)
0a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0b:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
df:08.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
df:08.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
df:08.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
df:09.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
df:09.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
df:09.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
df:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
df:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
df:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
df:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
[...removed some because too many character for post...]
df:0f.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
df:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
df:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
df:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
df:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
df:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
df:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
df:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
df:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
df:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
df:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
df:12.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
df:12.5 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
df:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
df:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
df:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
df:13.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
df:13.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02)
df:13.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
df:14.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02)
df:14.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02)
df:14.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02)
df:14.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02)
df:14.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
df:14.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
df:14.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
df:14.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
df:16.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
df:16.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
df:16.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)
df:16.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)
df:16.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02)
df:16.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
df:17.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02)
df:17.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control (rev 02)
df:17.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers (rev 02)
df:17.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers (rev 02)
df:17.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
df:17.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
df:17.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
df:17.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
df:1e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
df:1e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
df:1e.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
df:1e.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
df:1e.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
df:1f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)
df:1f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)
e0:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)
e0:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 02)
e0:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)
e0:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 02)
ff:08.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
ff:08.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
ff:08.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
ff:09.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
ff:09.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
ff:09.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
ff:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
[...removed some because too many character for post...]
ff:0d.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:12.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
ff:12.5 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
ff:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
....
 
dainjaz said:
1. Note that the MAC for enp10s0 enp11s0 are the same... is that because it's bonded? Is this expected behavior?
Click to expand...
It's the default for balance-rr. It's also part of the reason why you need to setup your switch accordingly (usually something called static LAG), in order for balance-rr to work properly - do you have any special settings on your switch to support this bond mode? We generally only recommend using active-backup or 802.3ad bond modes (if your switch supports it). Other bond modes oftentimes cause problems, particularly when used in conjunction with bridges. So if your switch supports 802.3ad then I'd strongly recommend that bonding mode.

dainjaz said:
5. I don't know how to test the bridge/bond itself without using a guest to do it, if anyone knows a good method?
Click to expand...
You could try removing the bond from the equation altogether and only put on network interface on the bridge for now. If that works, then we can be certain that the bond is at the heart of the issue.


You can learn more about the different bonding modes here:
https://www.kernel.org/doc/Documentation/networking/bonding.txt
 
I've now have deleted the bond0 and have one interface, enp10s0 attached to the bridge vmbr1 and everything seems to work using vmbr1. Though this is not an ideal solution as I'm only using 1of 2 ports.

I do not have any managed switches. I've considering implementing managed switch(es) for VLANs, however, that's for another day/another thread. My 'master' switch is a Trendenet 9-port 2.5G TEG-S5091 connected to my Google Wifi AC-1304, with a few other unmanaged switches in the mix. I can upgrade if needed but... we'll see...

I don't like the idea of active-backup, I want to increase throughput, rather than redundancy, considering I'm running 6+ VGs (one of which is a TrueNAS instance) in a homelab/selfhost environment; I don't 'need' production level redundancy.

Follow up questions:
1. Is it good practice to add two interfaces to one bridge (assuming they use the same device driver)? Can I do this, will it increase throughput at all or to 2.5G + 2.5G = ~5G? I read somewhere that to bond or bridge two adapters they needed to be the same device driver.
2. Is there any bond or structure (like WMware vSwitch equivalent) that will work with unmanaged switches? Should I not use a bond at all considering no managed switch in the mix (seems that way maybe)? I was using a standard vSwitch in VMware and that seemed to be bonding/bridging (not sure correct verbage here, please advise) the 4x 1G adapters and that seemed to be working fine in VMware ESXi. I 'upgraded' to 2x 2.5G for this setup, Is there no equivalent of a VMware standard vSwitch? Is that called a 'bridge'? or Port Groups?
3. Other recommendations on how to best manage networking with current setup, or way to improve for best practices?
4. Can I add VLANs using proxmox internally that never need to go outside proxmox to a physical Sw? Or test and see how the unmanaged Switches handle traffic assuming assigned by proxmox?

Updated:
Code: 
# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp0s25
iface enp0s25 inet manual
#NIC - Management

auto enp10s0
iface enp10s0 inet manual
#NIC - Production

auto enp11s0
iface enp11s0 inet manual
#NIC - Production

auto vmbr0
iface vmbr0 inet static
        address 10.0.0.10/24
        gateway 10.0.0.1
        bridge-ports enp0s25
        bridge-stp off
        bridge-fd 0
#Sw Management Network

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp10s0
        bridge-stp off
        bridge-fd 0
#Sw Production Network

source /etc/network/interfaces.d/*
 
Last edited:
dainjaz said:
1. Is it good practice to add two interfaces to one bridge (assuming they use the same device driver)? Can I do this, will it increase throughput at all or to 2.5G + 2.5G = ~5G? I read somewhere that to bond or bridge two adapters they needed to be the same device driver.
Click to expand...
If you do that unbonded, then you have created a network loop, which will cause issues.


dainjaz said:
2. Is there any bond or structure (like WMware vSwitch equivalent) that will work with unmanaged switches? Should I not use a bond at all considering no managed switch in the mix (seems that way maybe)? I was using a standard vSwitch in VMware and that seemed to be bonding/bridging (not sure correct verbage here, please advise) the 4x 1G adapters and that seemed to be working fine in VMware ESXi. I 'upgraded' to 2x 2.5G for this setup, Is there no equivalent of a VMware standard vSwitch? Is that called a 'bridge'? or Port Groups?
Click to expand...
Afaik VMWare does something that is in practice very similar to the balance-slb mode, which is available in OVS [1], but not with the standard linux bonding driver. A bridge is the Linux network stack equivalent of a vSwitch.


dainjaz said:
4. Can I add VLANs using proxmox internally that never need to go outside proxmox to a physical Sw? Or test and see how the unmanaged Switches handle traffic assuming assigned by proxmox?
Click to expand...
Not sure I understand 100%. You want to see how the switch handles tagged traffic? What would be the purpose of adding additional VLANs on Proxmox? Isolation? But yes, you can just add VLANs to a bridge and then use them internally.


[1] https://docs.openvswitch.org/en/latest/topics/bonding/#slb-bonding
 
You must log in or register to reply here.
Top Bottom