NIC no longer works after kernel upgrade.

[Kigen]

Hi,

I just installed Proxmox VE 8.2 to a new node using the ISO. By default this comes with kernel 6.8.4-2.
This new node has a USB4 10Gbps NIC. A IO CREST USB 4.0 Type-C 10G Ethernet Adapter. It appears to run a Marvel AQtion AQC113 under the hood.

So on kernel 6.8.4-2 this NIC works fine.

Nov 07 05:22:23 pve-amd7940hs-04 kernel: atlantic: Detect ATL2FW 1050026
Nov 07 05:22:23 pve-amd7940hs-04 kernel: atlantic 0000:67:00.0 enp103s0: renamed from eth0
Nov 07 05:22:26 pve-amd7940hs-04 kernel: atlantic 0000:67:00.0 enp103s0: entered allmulticast mode
Nov 07 05:22:26 pve-amd7940hs-04 kernel: atlantic 0000:67:00.0 enp103s0: entered promiscuous mode
Nov 07 05:22:50 pve-amd7940hs-04 kernel: atlantic 0000:67:00.0 enp103s0: atlantic: link change old 0 new 5000

However, after running updates the new kernel 6.8.12-3-pve is no longer able to load the driver.

Nov 07 04:48:45 pve-amd7940hs-04 kernel: atlantic: FW Restart failed
Nov 07 04:48:45 pve-amd7940hs-04 kernel: atlantic: probe of 0000:67:00.0 failed with error -5

I assume the new kernel has a bug in it with the driver. Anyone have any ideas? For now just booting the old kernel from the grub menu works.

 

[Moayad]

Hi,

Can you check if the NIC have an available firmware update? Sometimes the firmware update can help. and/plus check if you have microcode is installed in your Proxmox VE.

 

[Kigen]

Hi,

There does not appear to be any firmware update. I have installed the amd64-microcode package from the Debian repo.

Here is lspci -v output on 6.8.4-2.

67:00.0 Ethernet controller: Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] (rev 03)
        Subsystem: Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G]
        Flags: bus master, fast devsel, latency 0, IRQ 39, IOMMU group 7
        Memory at ac400000 (64-bit, non-prefetchable) [size=512K]
        Memory at ac4a0000 (64-bit, non-prefetchable) [size=4K]
        Memory at ac000000 (64-bit, non-prefetchable) [size=4M]
        Expansion ROM at ac480000 [disabled] [size=128K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Virtual Channel
        Capabilities: [168] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [178] Secondary PCI Express
        Capabilities: [198] Physical Layer 16.0 GT/s <?>
        Capabilities: [1bc] Lane Margining at the Receiver <?>
        Capabilities: [1d4] Latency Tolerance Reporting
        Capabilities: [1dc] L1 PM Substates
        Capabilities: [1ec] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Capabilities: [2ec] Data Link Feature <?>
        Capabilities: [2f8] Precision Time Measurement
        Capabilities: [304] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>
        Kernel driver in use: atlantic
        Kernel modules: atlantic

I just rebooted it again and something weird happened. The device shows up but hasn't established a link.

Linux pve-amd7940hs-04 6.8.12-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-3 (2024-10-23T11:41Z) x86_64
root@pve-amd7940hs-04:~# dmesg | grep atlantic
[    1.040969] atlantic: FW Restart failed
[    1.041116] atlantic: probe of 0000:67:00.0 failed with error -5
[    6.651193] atlantic 0000:67:00.0: enabling device (0000 -> 0002)
[    6.733825] atlantic: Detect ATL2FW 1050026
[    6.749359] atlantic 0000:67:00.0 enp103s0: renamed from eth0
root@pve-amd7940hs-04:~# ethtool enp103s0
Settings for enp103s0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Full
                                100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Full
                                100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        MDI-X: Unknown
        Supports Wake-on: pg
        Wake-on: g
        Current message level: 0x00000005 (5)
                               drv link
        Link detected: no

 

[Moayad]

Could you please post the network configuration `cat /etc/netwrok/interfaces` and the output of `ip a`?

 

[Kigen]

root@pve-amd7940hs-04:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 58:47:ca:78:5b:7e brd ff:ff:ff:ff:ff:ff
3: wlp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether dc:97:ba:3f:32:e3 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 58:47:ca:78:5b:7e brd ff:ff:ff:ff:ff:ff
    inet 192.168.29.74/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::5a47:caff:fe78:5b7e/64 scope link
       valid_lft forever preferred_lft forever
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 22:75:2a:52:c6:52 brd ff:ff:ff:ff:ff:ff
    inet 192.168.29.75/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::2075:2aff:fe52:c652/64 scope link
       valid_lft forever preferred_lft forever
13: enp103s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 88:c9:b3:b3:17:49 brd ff:ff:ff:ff:ff:ff
15: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 6e:28:17:23:60:5c brd ff:ff:ff:ff:ff:ff
root@pve-amd7940hs-04:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface wlp3s0 inet manual

iface enp103s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.29.74/24
        gateway 192.168.29.1
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr1
iface vmbr1 inet static
        address 192.168.29.75/24
        bridge-ports enp103s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Test

source /etc/network/interfaces.d/*

 

[Moayad]

Thank you for the output. The NIC names look ok not changed after the proxmox-kernel updated. Could you please post the output of `pveversion -v`?

Do you have a atlantic.conf in the `/etc/modprobe.d/`?

 

[Kigen]

root@pve-amd7940hs-04:~# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.12-3-pve)
pve-manager: 8.2.7 (running version: 8.2.7/3e0176e6bb2ade3b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-3
proxmox-kernel-6.8.12-3-pve-signed: 6.8.12-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
amd64-microcode: 3.20240820.1~deb12u1
ceph-fuse: 18.2.4-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.8
libpve-cluster-perl: 8.0.8
libpve-common-perl: 8.2.5
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.10
libpve-storage-perl: 8.2.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-4
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.2.4
pve-cluster: 8.0.8
pve-container: 5.2.0
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.0.7
pve-firmware: 3.14-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.4
pve-qemu-kvm: 9.0.2-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1
root@pve-amd7940hs-04:/etc/modprobe.d# ls -alh
total 27K
drwxr-xr-x  2 root root   5 Nov  7 06:57 .
drwxr-xr-x 99 root root 192 Nov  7 07:13 ..
-rw-r--r--  1 root root 154 Aug 24 07:24 amd64-microcode-blacklist.conf
-rw-r--r--  1 root root 172 Apr 23  2024 pve-blacklist.conf
-rw-r--r--  1 root root  35 Nov  6 21:05 zfs.conf