Trouble Adding 2 Port 10G NIC - Only One Port Working At A Time!

[santacruzskim]

I recently got a 10G NIC (Intel X540-T2). The plan was to run one port directly to my workstation and the other directly to my Windows server. I configured each port to have their own Linux Bridge and static IP on a separate ip range from the rest of my home just for this 10G, computer to computer traffic.

So the problem is this: vmbr1 (port 1) works great. Pings, iperf3, shared folders... everything works exactly as expected. But, vmbr2 (port 2) doesn't work at all. I mean, it lights up when I plug in a cable, but nothing goes in or out.

I did all the normal troubleshooting, swapping cables, trying different configs on the other machines, etc., but no change. **HOWEVER** when I swap the configuration to put port 2 on vmbr1 and port 1 on vmbr 2 (with no other change and w/o (and with) a host reboot), I get the inverse result - port 2 works perfectly! To put it another way, vmbr1 works no matter which physical port is assigned to it, and similarly, vmbr2 doesn't work at all regardless of which port is assigned. I also just noticed that when I am connected over the working port (whichever is assigned to vmbr1), I can successfully ping Proxmox at the IP assigned to vmbr1 AND the IP assigned vmbr2!

Here's a quick overview and some reports (the ones I'm aware of).

10.147.X.X ZeroTier network
192.168.1.X Home network (255.255.255.0)
10.10.10.X 10G network (255.255.0.0)

10.10.10.3 - Workstation​

10.10.10.9 - Proxmox 10G port 1 (named ens4f0, assigned to vmbr1)​

10.10.10.10 - Proxmox 10G port 2 (named ens4f1, assigned to vmbr2)​

10.10.10.124 - Windows Server​

The Proxmox machine network looks like this

vmbr0 - 1G NIC - 192.168.1.190​

vmbr1 - 10G Port1 - 10.10.10.9​

vmbr2 - 10G Port2 - 10.10.10.10​

​

/etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual
        post-up /usr/sbin/ethtool -K $IFACE tso off 2> /dev/null

iface ens4f0 inet manual
        mtu 9000

iface ens4f1 inet manual
        mtu 9000

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.190/24
        gateway 192.168.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        post-up /usr/sbin/ethtool -K $IFACE tso off 2> /dev/null

auto vmbr1
iface vmbr1 inet static
        address 10.10.10.9/24
        netmask 255.255.0.0
        bridge-ports ens4f0
        bridge-stp off
        bridge-fd 0
        mtu 9000
#port0 on 10G card

auto vmbr2
iface vmbr2 inet static
        address 10.10.10.10/24
        netmask 255.255.0.0
        bridge-ports ens4f1
        bridge-stp off
        bridge-fd 0
        mtu 9000
#port1 on 10G card

source /etc/network/interfaces.d/*

brctl show

bridge name     bridge id               STP enabled     interfaces
fwbr213i0       8000.627a23dc1909       no              fwln213i0
                                                        veth213i0
vmbr0           8000.480fcf4a7ddc       no              eno1
                                                        fwpr213p0
vmbr1           8000.98b78500f5c4       no              ens4f0
vmbr2           8000.98b78500f5c5       no              ens4f1

ip route

default via 192.168.1.1 dev vmbr0 proto kernel onlink
10.10.10.0/24 dev vmbr1 proto kernel scope link src 10.10.10.9
10.10.10.0/24 dev vmbr2 proto kernel scope link src 10.10.10.10
10.147.18.0/24 dev ztrfyn4bpf proto kernel scope link src 10.147.18.164
192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.190

ifconfig (shortened to relevant entries)

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 48:0f:cf:4a:7d:dc  txqueuelen 1000  (Ethernet)
        RX packets 26010  bytes 6389386 (6.0 MiB)
        RX errors 0  dropped 55  overruns 0  frame 0
        TX packets 16652  bytes 4963054 (4.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xfbd00000-fbd20000

ens4f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        ether 98:b7:85:00:f5:c4  txqueuelen 1000  (Ethernet)
        RX packets 6390  bytes 1609897 (1.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5547  bytes 1184617 (1.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens4f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        ether 98:b7:85:00:f5:c5  txqueuelen 1000  (Ethernet)
        RX packets 1930  bytes 252079 (246.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1766  bytes 192747 (188.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 3794  bytes 700187 (683.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3794  bytes 700187 (683.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.190  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::4a0f:cfff:fe4a:7ddc  prefixlen 64  scopeid 0x20<link>
        ether 48:0f:cf:4a:7d:dc  txqueuelen 1000  (Ethernet)
        RX packets 28058  bytes 7149154 (6.8 MiB)
        RX errors 0  dropped 539  overruns 0  frame 0
        TX packets 19620  bytes 5655276 (5.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 10.10.10.9  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::9ab7:85ff:fe00:f5c4  prefixlen 64  scopeid 0x20<link>
        ether 98:b7:85:00:f5:c4  txqueuelen 1000  (Ethernet)
        RX packets 6390  bytes 1520437 (1.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5547  bytes 1184617 (1.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmbr2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 10.10.10.10  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::9ab7:85ff:fe00:f5c5  prefixlen 64  scopeid 0x20<link>
        ether 98:b7:85:00:f5:c5  txqueuelen 1000  (Ethernet)
        RX packets 1930  bytes 225059 (219.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1766  bytes 192747 (188.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ztrfyn4bpf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2800
        inet 10.147.18.164  netmask 255.255.255.0  broadcast 10.147.18.255
        inet6 fe80::6c8e:d5ff:fee3:3644  prefixlen 64  scopeid 0x20<link>
        inet6 fd3e:fa5c:b78a:82b2:6f99:933c:5769:8118  prefixlen 88  scopeid 0x0<global>
        ether 6e:8e:d5:e3:36:44  txqueuelen 1000  (Ethernet)
        RX packets 1464  bytes 102174 (99.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1541  bytes 103567 (101.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

​

This is a brand new build so I have no special firewall or networking rules applied to Proxmox. I am technically minded, but I'm still new to Proxmox and networking beyond the basics has never clicked with me so I'm really hoping I'm overlooking something obvious! I couldn't find anything out there that matched my predicament either, hence this post

I've hit a wall in my troubleshooting and hoping for ideas.!

 

[mfed]

Working as expected to me. You have both interfaces in the same 10.10.10.x subnet, so it chooses whatever interface it likes more, in your case the first one defined vmbr1...
Just use the different IP subnets between your interfaces and enable routing... Or I think you might be able to configure a single bridge adding both 10G interfaces, just make sure that you never connect them to a switch...

 

[santacruzskim]

Ah, ok - here comes my lack of networking knowledge to bite me. My 192.168.1.X network has about 100 devices on it that all talk to each other just fine so I thought this 10.10.10.X network would behave the same and the three machines would play together and be more interchangeable if they were all on the same subnet. Maybe because they are directly connected there is a difference there vs passing through a router/switch as my 192. network is?

I made the following changes and getting the expected results. vmbr1 and the windows server are on the same subnet and vmbr2 and the workstation are together on a different subnet:

vmbr1 - 10G NIC port 1 - 10.10.10.9​

vmbr2 - 10G NIC port 2 - 10.10.2.10​

Windows server ----- 10.10.10.124​

Workstation ---------- 10.10.2.3​

I also removed lines specifying a subnet mask from the "interfaces" config file. (I had added netmask 255.255.0.0 to both bridges when I first configured them to match the other two machines already on the 10G network). However, now from both the workstation and the Windows server, I can ping both NIC ports successfully. How is that possible?! Clearly, I'm still missing something.

I also noticed that when I added vmbr1 as a second NIC to a LXC I use for samba file sharing, it made the container non-functional. I can't access my shares at any IP (192... or 10... networks), or the web admin page from any device on the network. Thinking back on the issue that started this, would each container and VM need to be on a separate subnet, or is Proxmox acting as a sort of router in that case and letting everyone speak to each other? I thought that was the point of using bridges vs the nic directly. I tried assigning various IP's to vmbr1 in teh LXC config but no luck yet.

Thanks @mfed for pointing me in the right direction. I read a lot of networking documentation and maybe it's over my head, but I didn't see these things addressed. If there's a resource I have yet to find that explains this, I'd be grateful.

 

[santacruzskim]

To wrap this up, I ended up just keeping each 10G port on their own subnet and working around the limitations that creates. Maybe down the road, my 10G network will expand and a completely new approach with a switch managing it will make sense. There's clearly still a lot I need to research before trying to expand beyond this setup.

For the LXC's it's been trial and error, figuring out various configurations that worked. Seems like each one is a little different, depending on the template you're using. For the Linux Turnkey fileserver template, I had to remove all IP assignments for the LXC in the Proxmox GUI (set to static and leave blank), then manually assign IPs and configure them through a combination of the LXC's /etc/network/interfaces and the fileserver's built-in Web UI. It broke access to anything other than the console a few times, but I got it to work once I figured out an order of operations it liked. Other LXCs have been similarly finicky, but not quite as terrible to troubleshoot since it's just you and the console without another configuration layer trying to make everything "easy" (which, in its defense, did make setting up a samba share very quick and easy before trying to add the 10G NIC).

VM's, on the other hand (Win11, TrueNAS Core, TrueNAS Scale, OMV), were *incredibly* easy. I just added them in the GUI, they appeared in the VM, and I configured them there, just as one would expect.

Gonna keep my fingers crossed I don't run into a need to touch this since it's all working now.