NIC no longer works after kernel upgrade.

K

Kigen

New Member
Nov 7, 2024
4
0
1
Hi,

I just installed Proxmox VE 8.2 to a new node using the ISO. By default this comes with kernel 6.8.4-2.
This new node has a USB4 10Gbps NIC. A IO CREST USB 4.0 Type-C 10G Ethernet Adapter. It appears to run a Marvel AQtion AQC113 under the hood.

So on kernel 6.8.4-2 this NIC works fine.
Code: 
Nov 07 05:22:23 pve-amd7940hs-04 kernel: atlantic: Detect ATL2FW 1050026
Nov 07 05:22:23 pve-amd7940hs-04 kernel: atlantic 0000:67:00.0 enp103s0: renamed from eth0
Nov 07 05:22:26 pve-amd7940hs-04 kernel: atlantic 0000:67:00.0 enp103s0: entered allmulticast mode
Nov 07 05:22:26 pve-amd7940hs-04 kernel: atlantic 0000:67:00.0 enp103s0: entered promiscuous mode
Nov 07 05:22:50 pve-amd7940hs-04 kernel: atlantic 0000:67:00.0 enp103s0: atlantic: link change old 0 new 5000

However, after running updates the new kernel 6.8.12-3-pve is no longer able to load the driver.
Code: 
Nov 07 04:48:45 pve-amd7940hs-04 kernel: atlantic: FW Restart failed
Nov 07 04:48:45 pve-amd7940hs-04 kernel: atlantic: probe of 0000:67:00.0 failed with error -5

I assume the new kernel has a bug in it with the driver. Anyone have any ideas? For now just booting the old kernel from the grub menu works.
 
Hi,

Can you check if the NIC have an available firmware update? Sometimes the firmware update can help. and/plus check if you have microcode is installed in your Proxmox VE.
 
Hi,

There does not appear to be any firmware update. I have installed the amd64-microcode package from the Debian repo.

Here is lspci -v output on 6.8.4-2.
Code: 
67:00.0 Ethernet controller: Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] (rev 03)
        Subsystem: Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G]
        Flags: bus master, fast devsel, latency 0, IRQ 39, IOMMU group 7
        Memory at ac400000 (64-bit, non-prefetchable) [size=512K]
        Memory at ac4a0000 (64-bit, non-prefetchable) [size=4K]
        Memory at ac000000 (64-bit, non-prefetchable) [size=4M]
        Expansion ROM at ac480000 [disabled] [size=128K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Virtual Channel
        Capabilities: [168] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [178] Secondary PCI Express
        Capabilities: [198] Physical Layer 16.0 GT/s <?>
        Capabilities: [1bc] Lane Margining at the Receiver <?>
        Capabilities: [1d4] Latency Tolerance Reporting
        Capabilities: [1dc] L1 PM Substates
        Capabilities: [1ec] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Capabilities: [2ec] Data Link Feature <?>
        Capabilities: [2f8] Precision Time Measurement
        Capabilities: [304] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>
        Kernel driver in use: atlantic
        Kernel modules: atlantic

I just rebooted it again and something weird happened. The device shows up but hasn't established a link.
Code: 
Linux pve-amd7940hs-04 6.8.12-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-3 (2024-10-23T11:41Z) x86_64
root@pve-amd7940hs-04:~# dmesg | grep atlantic
[    1.040969] atlantic: FW Restart failed
[    1.041116] atlantic: probe of 0000:67:00.0 failed with error -5
[    6.651193] atlantic 0000:67:00.0: enabling device (0000 -> 0002)
[    6.733825] atlantic: Detect ATL2FW 1050026
[    6.749359] atlantic 0000:67:00.0 enp103s0: renamed from eth0
root@pve-amd7940hs-04:~# ethtool enp103s0
Settings for enp103s0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Full
                                100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Full
                                100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        MDI-X: Unknown
        Supports Wake-on: pg
        Wake-on: g
        Current message level: 0x00000005 (5)
                               drv link
        Link detected: no
 
Last edited:
Could you please post the network configuration `cat /etc/netwrok/interfaces` and the output of `ip a`?
 
Code: 
root@pve-amd7940hs-04:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 58:47:ca:78:5b:7e brd ff:ff:ff:ff:ff:ff
3: wlp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether dc:97:ba:3f:32:e3 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 58:47:ca:78:5b:7e brd ff:ff:ff:ff:ff:ff
    inet 192.168.29.74/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::5a47:caff:fe78:5b7e/64 scope link
       valid_lft forever preferred_lft forever
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 22:75:2a:52:c6:52 brd ff:ff:ff:ff:ff:ff
    inet 192.168.29.75/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::2075:2aff:fe52:c652/64 scope link
       valid_lft forever preferred_lft forever
13: enp103s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 88:c9:b3:b3:17:49 brd ff:ff:ff:ff:ff:ff
15: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 6e:28:17:23:60:5c brd ff:ff:ff:ff:ff:ff
root@pve-amd7940hs-04:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface wlp3s0 inet manual

iface enp103s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.29.74/24
        gateway 192.168.29.1
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr1
iface vmbr1 inet static
        address 192.168.29.75/24
        bridge-ports enp103s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Test

source /etc/network/interfaces.d/*
 
Thank you for the output. The NIC names look ok not changed after the proxmox-kernel updated. Could you please post the output of `pveversion -v`?

Do you have a atlantic.conf in the `/etc/modprobe.d/`?
 
Last edited:
Code: 
root@pve-amd7940hs-04:~# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.12-3-pve)
pve-manager: 8.2.7 (running version: 8.2.7/3e0176e6bb2ade3b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-3
proxmox-kernel-6.8.12-3-pve-signed: 6.8.12-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
amd64-microcode: 3.20240820.1~deb12u1
ceph-fuse: 18.2.4-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.8
libpve-cluster-perl: 8.0.8
libpve-common-perl: 8.2.5
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.10
libpve-storage-perl: 8.2.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-4
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.2.4
pve-cluster: 8.0.8
pve-container: 5.2.0
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.0.7
pve-firmware: 3.14-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.4
pve-qemu-kvm: 9.0.2-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1
root@pve-amd7940hs-04:/etc/modprobe.d# ls -alh
total 27K
drwxr-xr-x  2 root root   5 Nov  7 06:57 .
drwxr-xr-x 99 root root 192 Nov  7 07:13 ..
-rw-r--r--  1 root root 154 Aug 24 07:24 amd64-microcode-blacklist.conf
-rw-r--r--  1 root root 172 Apr 23  2024 pve-blacklist.conf
-rw-r--r--  1 root root  35 Nov  6 21:05 zfs.conf
 
Did you ever solve this problem?
I am seeing the same symptoms on Proxmox 9.0.10 (6.14.11-3-pve) with same IOCREST AQC113 10GB NIC. Link is down no matter what.
Same hardware works fine with windows OS installed and Marvell AQtion driver
 
vmblaster said:
Did you ever solve this problem?
I am seeing the same symptoms on Proxmox 9.0.10 (6.14.11-3-pve) with same IOCREST AQC113 10GB NIC. Link is down no matter what.
Same hardware works fine with windows OS installed and Marvell AQtion driver
Click to expand...

I had this problem with an IOCREST AQC113 adapter on Promox 8.4.1, sounds like the same adapter

Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G]

My fix in the end was to downgrade to kernel 6.8.4-4-pve, from memory there was a breaking change after this

edit: dug out my notes, these are the errors I saw:

Code: 
root@host:~# dmesg -T | grep -i atlantic

[Sun Jun 15 07:01:37 2025] WARNING: CPU: 8 PID: 147 at drivers/net/ethernet/aquantia/atlantic/hw_atl2/hw_atl2_utils_fw.c:112 aq_a2_fw_deinit+0xcf/0xe0 [atlantic]
[Sun Jun 15 07:01:37 2025]  snd_hda_core aesni_intel snd_acp_pci snd_acp_legacy_common drm_display_helper snd_hwdep snd_pci_acp6x crypto_simd cec cryptd snd_pcm ecdh_generic snd_pci_acp5x rc_core cfg80211 snd_timer snd_rn_pci_acp3x ecc i2c_algo_bit rapl snd pcspkr snd_acp_config k10temp snd_soc_acpi ccp soundcore snd_pci_acp3x amd_pmc input_leds mac_hid serio_raw vhost_net vhost vhost_iotlb tap thunderbolt_net efi_pstore dmi_sysfs ip_tables x_tables autofs4 zfs(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c cdc_ncm cdc_ether usbnet r8152 mii xhci_pci xhci_pci_renesas nvme atlantic crc32_pclmul thunderbolt r8169 xhci_hcd nvme_core macsec i2c_piix4 realtek nvme_auth video wmi i2c_hid_acpi i2c_hid hid
[Sun Jun 15 07:01:37 2025] RIP: 0010:aq_a2_fw_deinit+0xcf/0xe0 [atlantic]
[Sun Jun 15 07:01:37 2025]  ? aq_a2_fw_deinit+0xcf/0xe0 [atlantic]
[Sun Jun 15 07:01:37 2025]  ? aq_a2_fw_deinit+0xcf/0xe0 [atlantic]
[Sun Jun 15 07:01:37 2025]  aq_nic_deinit.part.0+0xb3/0xe0 [atlantic]
[Sun Jun 15 07:01:37 2025]  aq_nic_deinit+0x17/0x30 [atlantic]
[Sun Jun 15 07:01:37 2025]  aq_ndev_close+0x2b/0x40 [atlantic]
[Sun Jun 15 07:01:37 2025]  aq_pci_remove+0xb1/0xc0 [atlantic]
[Sun Jun 15 07:01:37 2025] atlantic 0000:07:00.0 ens1 (unregistering): left promiscuous mode
[Sun Jun 15 07:01:37 2025] atlantic 0000:07:00.0 ens1 (unregistering): left allmulticast mode

Wonder if it's the same thing? Would be interested to know if you can get it going on Proxmox 9

There's a thread about it here https://forum.proxmox.com/threads/network-not-working-in-aqc107-after-kernel-upgrade.149129/

I think I tried to install the driver at the time but wasn't able to get it working. The driver is here (public drivers -> linux -> acq113) https://www.marvell.com/support/downloads.html
 
Last edited:
robertyboberty said:
I had this problem with an IOCREST AQC113 adapter on Promox 8.4.1, sounds like the same adapter

Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G]

My fix in the end was to downgrade to kernel 6.8.4-4-pve, from memory there was a breaking change after this

edit: dug out my notes, these are the errors I saw:

...

Wonder if it's the same thing? Would be interested to know if you can get it going on Proxmox 9

There's a thread about it here https://forum.proxmox.com/threads/network-not-working-in-aqc107-after-kernel-upgrade.149129/

I think I tried to install the driver at the time but wasn't able to get it working. The driver is here (public drivers -> linux -> acq113) https://www.marvell.com/support/downloads.html
Click to expand...

Yep this is the same adapter i have.

Hmm so when you downgrade the kernel, does the card work properly with the atlantic driver?

Getting the marvell driver to work isn't easy, there's a github issue about it here:
https://github.com/Aquantia/AQtion/issues/69

It seems like marvell is not keeping it updated. Even if you can manage to compile it, it'll probably break on future kernel updates so it's not really a sustainable option.

Thanks for the links you provided, did you try setting thunderbolt.host_reset=false? any luck?
 
@robertyboberty i just tried adding that param on my system. NIC still doesn't work. Still showing as link down. Damnit.

Not seeing much in the logs about it
Code: 
# dmesg -T | grep -i atlantic
[Tue Oct 14 07:13:48 2025] atlantic: Detect ATL2FW 1050026
[Tue Oct 14 07:13:48 2025] atlantic 0000:3c:00.0 enp60s0: renamed from eth0

hmmm running a kernel so much older.. wont that cause other issues? (not an expert, dunno...)

given how attractive these nics are i'm hoping that more people using them might attract a fix at some point..
 
robertyboberty said:
I had this problem with an IOCREST AQC113 adapter on Promox 8.4.1, sounds like the same adapter
....
Click to expand...
Hm are you also using usb4/thunderbolt version? or a pci-e card?
I'm not sure if the issues are related to thunderbolt regression, atlantic nic driver or both..
 
You must log in or register to reply here.
Top Bottom