[SOLVED] NIC Not Working Reliably - New Install

[spiceygas]

Brand new proxmox 7.3-6 install on Supermicro MBD-M12SWA-TF.

The 10Gbt NIC stops working... sometimes.

• At the end of BIOS post the lights on the NIC are flashing.
• During the bootloader (where you can select which OS to boot) the lights on the NIC are flashing.
• When proxmox starts (EFI stub: Loaded initrd from command line option) the lights on the NIC stop flashing.
• When proxmox finishes booting and reaches the login screen, sometimes I get a message that the network bridge has started, sometimes not.
  • When the network bridge doesn't start, the lights don't flash.
  • When it starts, the lights flash and it displays the following message:

[48.312964] atlantic 0000:69:00.0 eth1: atlantic: link change old 0 new 1000
[48.313395] vmbr0: port 1(eth1) entered blocking state
[48.313417] vmbr0: port 1(eth1) entered forwarding state
[43.313883] IPv6:ADDRCONF(NETDEV_CHANGE): vmbr0: link becomes ready

(It's plugged into a gigabit switch while I'm building the system. Eventually I'll move it to a 10Gb switch)

If I get that message, then the network works correctly. I can ping out to the Internet, I can access the proxmox console in a web browser, etc. If I don't get that message then the network doesn't work.


Stuff that seems to effect it:

• If I do a reboot then the network bridge never loads
• If I have the other NIC (1 Gb) plugged in then the network bridge never loads
• If I do a hard shutdown, wait a few seconds, then power on, then the network bridge starts ~50% of the time.
  • Once it's loaded, I can plug in the other NIC and both work correctly. That's important because the 1Gb NIC is used for IPMI.

Any ideas what's wrong or how to fix it? My best guess is a driver issue, but I'd appreciate suggestions before trying anything else.

uname -r
5.15.85-1-pve

lspci | grep 'Ethernet'
66:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
69:00.0 Ethernet controller: Aquantia Corp. Device 14c0 (rev 03)

tcpdump --list-interfaces
1.eth1 [Up, Running, Connected]
2.vmbr0 [Up, Running, Connected]
3.any (Pseudo-device that captures on all interfaces) [Up, Running]
4.lo [Up, Running, Loopback]
5.bluetooth-monitor (Bluetooth Linux Monitor) [Wireless]
6.nflog (Linux netfilter log (NFLOG) interface) [none]
7.nfqueue (Linux netfilter queue (NFQUEUE) interface) [none]
8.dbus-system (D-Bus system bus) [none]
9.dbus-session (D-Bus session bus) [none]
10.eno1 [none, Disconnected]
11.enxb03af2b6059f [none, Disconnected]

ethtool eth1
Settings for eth1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Full
                                100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Full
                                100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: Symmetric
        Link partner advertised auto-negotiation: No
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        MDI-X: Unknown
        Supports Wake-on: pg
        Wake-on: g
        Current message level: 0x00000005 (5)
                               drv link
        Link detected: yes

ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3c:ec:ef:d6:da:8e brd ff:ff:ff:ff:ff:ff
    altname enp102s0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 3c:ec:ef:d6:dc:96 brd ff:ff:ff:ff:ff:ff
    altname enp105s0
4: enxb03af2b6059f: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b0:3a:f2:b6:05:9f brd ff:ff:ff:ff:ff:ff
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:ec:ef:d6:dc:96 brd ff:ff:ff:ff:ff:ff
    inet 10.10.1.15/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::3eec:efff:fed6:dc96/64 scope link
       valid_lft forever preferred_lft forever

 

[spiceygas]

My network has gone down and I can't get it back up. I've got the drivers from the manufacturer but to build them I need a bunch of dependencies.

I used apt-offline to get most dependencies, but I can't get linux-headers-5.15.85-1-pve


Even though I don't have a network connection, I tried running apt-get install linux-headers-5.15.85-1-pve and got the following message:

apt-get install linux-headers-5.15.85-1-pve
Reading package lists... Done
Building dependency tree... Done
Reading state information... done
Note, selecting 'linux-headers-5.15.85-1-pve-amd64' for regex 'linux-headers-5.15-85-1-pve'
Notes, selecting 'pve-headers-5.15.85.1.pve' instead of 'linux-headers-5.15.85-1-pve-amd64'
pve-headers-5.15.85.1.pve is already the newest version (5.15.85-1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.

Can anyone tell me how to get that dependency without a network connection? (I.e., I can't run apt-get)

 

[spiceygas]

I think I solved it. Leaving a paper trail here in-case anyone else runs into the same problems.

1. The driver for the aquantia NIC apparently isn't loaded by default. I ran modprobe -v atlantic to load it.
2. I then manually set the physical network's state as 'up'.
   1. ip a to get a list of network devices. Find the identifier of your device.
   2. ip link set [DEVICE] up
3. Next, I realized that the device name of my network device had changed. No idea why. It used to be eth0 but had changed to enp106s0. Therefore, I edited /etc/network/interfaces and changed the bridge adapter to point at the new network device.
   • I realize that it's not recommended to edit this file directly and instead work through the web UI, but if there's no network then you can't get to the web UI...
4. Finally, bring up the network bridge. ifup [DEVICE] where the bridge device's identifier should be noted from step 2.1 above.
5. Reboot. The network devices all come up automatically.
6. I've got a network drive mapped, but it needs to be manually mounted using mount -a.
   • I presume there's an order of operations issue during boot where it's trying to mount the drive before the network is up. Kinda silly, but given how rarely I plan to bring this server down, manually mounting the drives isn't the end of the world.

I wish I could reclaim the hours of my life spent trying to sort this out. But if anyone else ever gets stuck making an aquantia/atlantic NIC, I hope this helps.