[SOLVED] 10Gb Nic - Unexpected Behaviour

R

rbutler81

New Member
Apr 12, 2024
2
1
1
I've added a dual 10Gb nic to my server, which I'm trying to get setup but experiencing some strangeness.
I have a working 1Gb connection already @ 192.168.1.120 (en1 on vmbr0) - great, works.
I added ens5f0 (one of the ports from the 10Gb nic) to vmbr1, gave it IP: 192.168.1.137

cat /etc/networks/interfaces

Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface enx0a94ef07cc27 inet manual

iface ens5f0 inet manual

iface ens5f1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.120/24
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

auto vmbr1
iface vmbr1 inet static
    address 192.168.1.137/24
    gateway 192.168.1.1
    bridge-ports ens5f0
    bridge-stp off
    bridge-fd 0

I checked ens5f0 to make sure it was what I thought it was (the 10Gb nic):

ip link

Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 08:94:ef:07:cc:20 brd ff:ff:ff:ff:ff:ff
    altname enp27s0f0
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 08:94:ef:07:cc:21 brd ff:ff:ff:ff:ff:ff
    altname enp27s0f1
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 08:94:ef:07:cc:22 brd ff:ff:ff:ff:ff:ff
    altname enp27s0f2
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 08:94:ef:07:cc:23 brd ff:ff:ff:ff:ff:ff
    altname enp27s0f3
6: enx0a94ef07cc27: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 0a:94:ef:07:cc:27 brd ff:ff:ff:ff:ff:ff
7: ens5f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:8f:a2:c0 brd ff:ff:ff:ff:ff:ff
    altname enp129s0f0
8: ens5f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:8f:a2:c1 brd ff:ff:ff:ff:ff:ff
    altname enp129s0f1
9: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 08:94:ef:07:cc:20 brd ff:ff:ff:ff:ff:ff
10: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:8f:a2:c0 brd ff:ff:ff:ff:ff:ff
11: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether f6:de:ff:dc:cf:e6 brd ff:ff:ff:ff:ff:ff
12: veth200i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:b5:1e:63:81:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0
13: veth201i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:24:a2:5e:74:0c brd ff:ff:ff:ff:ff:ff link-netnsid 1
14: veth202i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:ef:cf:35:e0:42 brd ff:ff:ff:ff:ff:ff link-netnsid 2

Note that ens5f0 state changes between UP and DOWN if I disconnect / reconnect the DAC cable in that port.

lspci -k | sed -n '/Ethernet/,/driver in use/p'

Code: 
1b:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
    DeviceName: Function 0
    Subsystem: Lenovo NetXtreme BCM5719 Gigabit Ethernet PCIe
    Kernel driver in use: tg3
1b:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
    DeviceName: Function 1
    Subsystem: Lenovo NetXtreme BCM5719 Gigabit Ethernet PCIe
    Kernel driver in use: tg3
1b:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
    DeviceName: Function 2
    Subsystem: Lenovo NetXtreme BCM5719 Gigabit Ethernet PCIe
    Kernel driver in use: tg3
1b:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
    DeviceName: Function 3
    Subsystem: Lenovo NetXtreme BCM5719 Gigabit Ethernet PCIe
    Kernel driver in use: tg3
81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
    Kernel driver in use: ixgbe
81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
    Kernel driver in use: ixgbe

Note that 81:00.0 is using Kernel driver ixgbe

dmesg | grep ixgbe

Code: 
[    2.934801] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    2.934804] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    2.934885] ixgbe 0000:81:00.0: enabling device (0140 -> 0142)
[    4.121694] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 XDP Queue count = 0
[    4.121986] ixgbe 0000:81:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    4.122308] ixgbe 0000:81:00.0: MAC: 2, PHY: 1, PBA No: G43260-007
[    4.122310] ixgbe 0000:81:00.0: 38:ea:a7:8f:a2:c0
[    4.123430] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection
[    4.123574] ixgbe 0000:81:00.1: enabling device (0140 -> 0142)
[    5.309643] ixgbe 0000:81:00.1: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 XDP Queue count = 0
[    5.309936] ixgbe 0000:81:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    5.310258] ixgbe 0000:81:00.1: MAC: 2, PHY: 1, PBA No: G43260-007
[    5.310259] ixgbe 0000:81:00.1: 38:ea:a7:8f:a2:c1
[    5.311436] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection
[    5.341056] ixgbe 0000:81:00.1 ens5f1: renamed from eth1
[    5.381028] ixgbe 0000:81:00.0 ens5f0: renamed from eth0
[   13.195171] ixgbe 0000:81:00.0 ens5f0: entered allmulticast mode
[   13.195212] ixgbe 0000:81:00.0 ens5f0: entered promiscuous mode
[   13.347258] ixgbe 0000:81:00.0: registered PHC device on ens5f0
[  223.266986] ixgbe 0000:81:00.0 ens5f0: detected SFP+: 4
[  223.341410] ixgbe 0000:81:00.0 ens5f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 1257.790350] ixgbe 0000:81:00.0 ens5f0: NIC Link is Down
[ 1271.116020] ixgbe 0000:81:00.0 ens5f0: detected SFP+: 4
[ 1271.239667] ixgbe 0000:81:00.0 ens5f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 1468.153608] ixgbe 0000:81:00.0 ens5f0: NIC Link is Down
[ 1472.119518] ixgbe 0000:81:00.0 ens5f0: detected SFP+: 4
[ 1472.379046] ixgbe 0000:81:00.0 ens5f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 2450.242985] ixgbe 0000:81:00.0 ens5f0: NIC Link is Down
[ 2533.603966] ixgbe 0000:81:00.0 ens5f0: detected SFP+: 4
[ 2533.678010] ixgbe 0000:81:00.0 ens5f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Note that the Gigabit network ports are called ens5f0 and ens5f1 - also you can see the link going up / down as I unplugged it.

So... everything seems to be working? Except that it's not - when I ping the 10Gb card @192.168.1.137 I get results.
Then, I unplug the DAC cable - and I keep getting ping results - ummm.. what?
I reconnect the DAC cable, and unplug the RJ45 port (192.168.1.120) and the pinging to 192.168.1.137 starts timing out... ??

It seems that whether I ping 192.168.1.120, or 192.168.1.137 the pings are using the physical 1Gb nic, and never the 10Gb - and when that's not connected, nothing works.
I can also access the web gui via 192.168.1.137, but only as long as the 1Gb nic is connected.

Anyone have any ideas on this one? Would really appreciate any help.

Thanks!
 
Figured it out - in case anyone runs in to this...

You apparently can't have two bridges on the same subnet - when I put one on 192.168.2.xxx it started working correctly.
So I removed the RJ45 connection to the 1Gb nic - and am just using the 10Gb nic now on the 192.168.1.xxx network.
 
  • Like
Reactions: spitfine
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom