[SOLVED] Nuc8 Thunderbolt 10Gb NIC and Proxmox 7

[ypk]

Hello,

I've been unable to get QNAP T310G1S and a Sabrent TH-S3EA Thunderbolt 3, 10Gb adapters running in Proxmox (v7.0-11, kernel 5.11.22-4-pve). Both seem to use the Marvel Aquantic chipset, leveraging the "atlantic" kernel module/driver.
I'm running the latest NUC bios, latest thunderbolt firmware, latest firmware for the Sabrent adapter and no dice.

Before updating to Proxmox 7, using Proxmox 6.4, the Sabrent adapters were being recognized but completely hung up after a few hours/days. A complete of the NUC was necessary to get it back working. Which is why I opted to update, but since then, the adapters are not detected at all!

A series of things I tried:

• Updating all bios, firmwares etc. I could possibly find
• Tried atlantic driver v2.4.10 (via Sabrent website), v2.4.15 (via QNAP website) and latest from the AQtion github repo
  • I used the dkms install for each, by modifying the kernel header check to use pve kernel headers (see diff below)
• The adapters work fine on a MacBook and on Windows installed on those NUCs
• Tested on 2 different NUC8s
• Tried different thunderbolt cables (desperate, I know)
• Going through numerous posts on this forum or reddit, most seem outdated and haven't helped getting any closer to a solution

Diff for dkms.sh:

diff --git a/dkms.sh b/dkms.sh
old mode 100644
new mode 100755
index a4ebba4..195d25f
--- a/dkms.sh
+++ b/dkms.sh
@@ -107,7 +107,7 @@ main ()
        DISTRO=$(get_distro_os)
        if [ "${DISTRO}" = "debian" ] ; then
                PACKET_MNG="${APT_GET}"
-               LINUX_HEADERS="linux-headers-`uname -r`"
+               LINUX_HEADERS="pve-headers-`uname -r`"
                TOOLS="build-essential"
                CMD="dpkg-query -l"
        elif [ "${DISTRO}" = "redhat" ] ; then

Kernel logs when plugging in the QNAP adapter:

[59345.810878] thunderbolt 0-1: device disconnected
[59353.576600] thunderbolt 0-1: new device found, vendor=0x56 device=0x10d2
[59353.576603] thunderbolt 0-1: QNAP Systems, Inc. QNA-T310G1S
[59356.486875] usb usb4-port1: Cannot enable. Maybe the USB cable is bad?
[59360.578618] usb usb4-port1: Cannot enable. Maybe the USB cable is bad?
[59360.578657] usb usb4-port1: attempt power cycle

Would love to know if I'm the only one out there or if anyone would have an idea on how to resolve this.

Thanks!

 

[ramblurr]

FWIW, I am using a Sonnet TB3 10 GbE adapter on a NUC10i7 with a fresh install of PVE 7 (iso release 2).. and the adapter is working fine out of the box as expected. AFAIK they also use the atlantic module. However I do see the "cannot enable" message, but it seems to be harmless.

[   71.034950] thunderbolt 0-1: new device found, vendor=0x8 device=0x36
[   71.034952] thunderbolt 0-1: Sonnet Technologies, Inc Solo 10G SFP+ Thunderbolt 3 Edition
[   71.756339] usb usb4-port1: Cannot enable. Maybe the USB cable is bad?
[   75.844601] usb usb4-port1: Cannot enable. Maybe the USB cable is bad?
[   75.844665] usb usb4-port1: attempt power cycle

 

[ypk]

FWIW, I am using a Sonnet TB3 10 GbE adapter on a NUC10i7 with a fresh install of PVE 7 (iso release 2).. and the adapter is working fine out of the box as expected. AFAIK they also use the atlantic module. However I do see the "cannot enable" message, but it seems to be harmless.

[   71.034950] thunderbolt 0-1: new device found, vendor=0x8 device=0x36
[   71.034952] thunderbolt 0-1: Sonnet Technologies, Inc Solo 10G SFP+ Thunderbolt 3 Edition
[   71.756339] usb usb4-port1: Cannot enable. Maybe the USB cable is bad?
[   75.844601] usb usb4-port1: Cannot enable. Maybe the USB cable is bad?
[   75.844665] usb usb4-port1: attempt power cycle

That helps narrow down the issue! Thanks!

I assume you haven't done any changes in your bios? I may try a new install on one of them

 

[ramblurr]

Correct. I just verified that the Thunderbolt ports were enabled and that it was set to Legacy mode in the security settings (so no device authorization is necessary). My NUCs came out of the box with those settings. So I think that `Cannot enable.` message is a red herring.

Here's my lsmod output btw:

Module                  Size  Used by
ebtable_filter         16384  0
ebtables               36864  1 ebtable_filter
ip_set                 53248  0
ip6table_raw           16384  0
iptable_raw            16384  0
ip6table_filter        16384  0
ip6_tables             32768  2 ip6table_filter,ip6table_raw
iptable_filter         16384  0
bpfilter               16384  0
bonding               172032  0
tls                    94208  1 bonding
softdog                16384  2
nfnetlink_log          20480  1
nfnetlink              20480  3 ip_set,nfnetlink_log
snd_hda_codec_hdmi     65536  1
snd_sof_pci            24576  0
snd_sof_intel_hda_common    94208  1 snd_sof_pci
snd_sof_intel_hda      20480  1 snd_sof_intel_hda_common
snd_sof_intel_byt      28672  1 snd_sof_pci
snd_sof_intel_ipc      20480  1 snd_sof_intel_byt
snd_sof               126976  4 snd_sof_pci,snd_sof_intel_hda_common,snd_sof_intel_byt,snd_sof_intel_ipc
snd_sof_xtensa_dsp     16384  2 snd_sof_intel_hda_common,snd_sof_intel_byt
snd_soc_hdac_hda       24576  1 snd_sof_intel_hda_common
snd_hda_ext_core       32768  3 snd_sof_intel_hda_common,snd_soc_hdac_hda,snd_sof_intel_hda
intel_rapl_msr         20480  0
snd_soc_acpi_intel_match    49152  2 snd_sof_pci,snd_sof_intel_hda_common
snd_soc_acpi           16384  3 snd_soc_acpi_intel_match,snd_sof_intel_hda_common,snd_sof_intel_byt
intel_rapl_common      24576  1 intel_rapl_msr
ledtrig_audio          16384  1 snd_sof
x86_pkg_temp_thermal    20480  0
intel_powerclamp       20480  0
coretemp               20480  0
snd_hda_intel          53248  0
snd_intel_dspcfg       28672  3 snd_hda_intel,snd_sof_pci,snd_sof_intel_hda_common
soundwire_intel        40960  2 snd_sof_intel_hda_common,snd_intel_dspcfg
kvm_intel             282624  0
soundwire_generic_allocation    16384  1 soundwire_intel
soundwire_cadence      32768  1 soundwire_intel
snd_hda_codec         147456  3 snd_hda_codec_hdmi,snd_hda_intel,snd_soc_hdac_hda
kvm                   823296  1 kvm_intel
snd_hda_core           94208  7 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_ext_core,snd_hda_codec,snd_sof_intel_hda_common,snd_soc_hdac_hda,snd_sof_intel_hda
irqbypass              16384  1 kvm
snd_hwdep              16384  1 snd_hda_codec
iwlmvm                413696  0
i915                 2306048  2
crct10dif_pclmul       16384  1
soundwire_bus          77824  3 soundwire_intel,soundwire_generic_allocation,soundwire_cadence
ghash_clmulni_intel    16384  0
aesni_intel           372736  0
btusb                  61440  0
crypto_simd            16384  1 aesni_intel
snd_soc_core          286720  4 soundwire_intel,snd_sof,snd_sof_intel_hda_common,snd_soc_hdac_hda
mac80211             1019904  1 iwlmvm
cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
btrtl                  24576  1 btusb
drm_kms_helper        245760  1 i915
snd_compress           24576  1 snd_soc_core
mei_hdcp               24576  0
glue_helper            16384  1 aesni_intel
btbcm                  16384  1 btusb
ac97_bus               16384  1 snd_soc_core
cec                    53248  2 drm_kms_helper,i915
snd_pcm_dmaengine      16384  1 snd_soc_core
libarc4                16384  1 mac80211
btintel                28672  1 btusb
rapl                   20480  0
rc_core                57344  1 cec
snd_pcm               118784  11 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,soundwire_intel,snd_sof,snd_sof_intel_hda_common,snd_sof_intel_ipc,snd_compress,snd_soc_core,snd_hda_core,snd_pcm_dmaengine
i2c_algo_bit           16384  1 i915
fb_sys_fops            16384  1 drm_kms_helper
intel_cstate           20480  0
snd_timer              40960  1 snd_pcm
syscopyarea            16384  1 drm_kms_helper
bluetooth             643072  5 btrtl,btintel,btbcm,btusb
iwlwifi               368640  1 iwlmvm
snd                    94208  8 snd_hda_codec_hdmi,snd_hwdep,snd_hda_intel,snd_hda_codec,snd_timer,snd_compress,snd_soc_core,snd_pcm
wmi_bmof               16384  0
intel_wmi_thunderbolt    20480  0
pcspkr                 16384  0
sysfillrect            16384  1 drm_kms_helper
soundcore              16384  1 snd
efi_pstore             16384  0
ecdh_generic           16384  1 bluetooth
ee1004                 20480  0
sysimgblt              16384  1 drm_kms_helper
ecc                    32768  1 ecdh_generic
mei_me                 40960  1
cfg80211              888832  3 iwlmvm,iwlwifi,mac80211
mei                   122880  3 mei_hdcp,mei_me
intel_pch_thermal      20480  0
zfs                  4190208  6
mac_hid                16384  0
acpi_pad              184320  0
acpi_tad               20480  0
zunicode              331776  1 zfs
zzstd                 532480  1 zfs
zlua                  151552  1 zfs
zavl                   16384  1 zfs
icp                   294912  1 zfs
zcommon                98304  2 zfs,icp
znvpair                98304  2 zfs,zcommon
spl                   102400  6 zfs,icp,zzstd,znvpair,zcommon,zavl
vhost_net              32768  0
vhost                  53248  1 vhost_net
vhost_iotlb            16384  1 vhost
tap                    24576  1 vhost_net
ib_iser                40960  0
rdma_cm               118784  1 ib_iser
iw_cm                  49152  1 rdma_cm
ib_cm                 122880  1 rdma_cm
ib_core               360448  4 rdma_cm,iw_cm,ib_iser,ib_cm
iscsi_tcp              24576  0
libiscsi_tcp           32768  1 iscsi_tcp
libiscsi               65536  3 libiscsi_tcp,iscsi_tcp,ib_iser
scsi_transport_iscsi   126976  5 libiscsi_tcp,iscsi_tcp,ib_iser,libiscsi
drm                   548864  3 drm_kms_helper,i915
sunrpc                544768  1
ip_tables              32768  2 iptable_filter,iptable_raw
x_tables               49152  7 ebtables,ip6table_filter,ip6table_raw,iptable_filter,ip6_tables,iptable_raw,ip_tables
autofs4                45056  2
btrfs                1331200  0
blake2b_generic        20480  0
xor                    24576  1 btrfs
raid6_pq              114688  1 btrfs
dm_thin_pool           69632  3
dm_persistent_data     73728  1 dm_thin_pool
dm_bio_prison          20480  1 dm_thin_pool
dm_bufio               32768  1 dm_persistent_data
libcrc32c              16384  2 dm_persistent_data,btrfs
crc32_pclmul           16384  0
nvme                   45056  2
sdhci_pci              57344  0
e1000e                266240  0
cqhci                  28672  1 sdhci_pci
nvme_core             122880  6 nvme
i2c_i801               32768  0
intel_lpss_pci         20480  0
ahci                   40960  0
atlantic              188416  0
intel_lpss             16384  1 intel_lpss_pci
xhci_pci               20480  0
xhci_pci_renesas       20480  1 xhci_pci
idma64                 20480  0
i2c_smbus              20480  1 i2c_i801
sdhci                  69632  1 sdhci_pci
macsec                 49152  1 atlantic
thunderbolt           249856  0
libahci                36864  1 ahci
xhci_hcd              290816  1 xhci_pci
virt_dma               20480  1 idma64
wmi                    32768  2 intel_wmi_thunderbolt,wmi_bmof
video                  53248  1 i915
pinctrl_cannonlake     36864  0

 

[ypk]

So I tried a few things, without any success:

1. Checked bios thunderbolt security -> Set to legacy
2. Re-install proxmox 7 from scratch -> the network adapter worked during the install, not once proxmox was installed (no ENI showing at after than)
3. Latest Debian bullseye install also doesn't work.

I tried both fresh installs doing the dkms install of the latest AQtion and Marvell driver. The module loads fine, but when plugging in the adapter, it just doesn't load up the adapter.

Other than waiting for a kernel update, I don't see what to do here :/

 

[ypk]

The latest pve kernel (5.11.22-4-pve) seems to fix this.

 

[fletch]

The latest pve kernel (5.11.22-4-pve) seems to fix this.

I think I am having the same issue with the solo10g adapter. Didn't do ANYTHING until I made the Legacy change in the bios (thank you for that tip) and now another nic shows up under the proxmox node network area. However it won't connect to my switch on ether the 10g side or even the 1g side and not getting any link lights. Can't make it go active. Not sure if I got a bad one or if this is another software issue. Is this the same symptoms you had? I have the latest update from 7 so don't think that is the issue. Thanks.

 

[fletch]

For anyone else with this issue- these are the additional steps I had to do to get this working. My way was not the correct way but it got me going until I can get help setting up a vlan. Like the tips from the other posters , you have make sure thunderbolt is set to legacy in bios or Proxmox won't see it. After doing this, it should then see the new adapter but the interface won't automatically go up and link with your switch. You then need to force the link up by using this command : ip link set enp7s0 up. enp7s0 is the name of my 10g adapter, you need to change it to what ever proxmox assigns to your 10g adapter. After this , it still will not work until you assign it to a bridge. I had never done this before but figured it out by looking at my existing configuration. I was fairly certain this would cause a network conflic and it did until I pulled the 1g cable out of nuc. I also had to reboot my 10g switch but afterwards it started working. After this, you should not have to manually start the interface again. I want to set this up as a vlan but until I can find some decent instructions this will work and it should work for you as well. Good luck!

 

[Docop2]

Just curious here, if you unplug the device, then all will fail and the only way to get back up is by doing a full reboot ?

 

[fletch]

If you are talking about the 10g switch, yes I had to pull power and reboot to get back web control. I created a network conflict and was not surprised when things went wrong. I can't really do any more until the nic for the nas comes in...right now they are saying Friday. Then hopefully I can figure out how to make this all work as a vlan.

 

[okiedokie]

I am trying to get a Sonnet Thunderbolt 10Gb NIC to work in Proxmox with a NUC7.
I have it working, but with very poor performance. I get only ~350Mb/s on an iperf to a machine with a 10Gb NIC. When I switch over to the internal 1Gb adapter in the NUC I get ~990Mb/s.
Is there a driver issue or am I missing something with the setup? Thanks very much, I would appreciate some tips. I'm fairly new to Proxmox.

I have tried alternative LAN cables, and switches. I get the same 350Mbps from a 1Gb switch as I get from a 10Gb switch.

 

[Docop2]

I did try a quad 25g card on the nuc6 and was running quite fine. Just boot xubuntu iso or garuda lin , put iperf and try. get a working hrwd, then check prox later.

 

[okiedokie]

I did try a quad 25g card on the nuc6 and was running quite fine. Just boot xubuntu iso or garuda lin , put iperf and try. get a working hrwd, then check prox later.

Quad 25Gb on a NUC! Wow!

 

[okiedokie]

I am trying to get a Sonnet Thunderbolt 10Gb NIC to work in Proxmox with a NUC7.
I have it working, but with very poor performance. I get only ~350Mb/s on an iperf to a machine with a 10Gb NIC. When I switch over to the internal 1Gb adapter in the NUC I get ~990Mb/s.
Is there a driver issue or am I missing something with the setup? Thanks very much, I would appreciate some tips. I'm fairly new to Proxmox.

I have tried alternative LAN cables, and switches. I get the same 350Mbps from a 1Gb switch as I get from a 10Gb switch.

Update:
I tried again under Proxmox 8 and it is working fine. I set legacy mode in the bios, it works without issue. in iperf3 in a VM I am getting ~9Gb/s. The only thing that is different to before, is the fact that I upgraded from Proxmox 7 to 8.

 

[skittlebrau]

For anyone else with this issue- these are the additional steps I had to do to get this working. My way was not the correct way but it got me going until I can get help setting up a vlan. Like the tips from the other posters , you have make sure thunderbolt is set to legacy in bios or Proxmox won't see it. After doing this, it should then see the new adapter but the interface won't automatically go up and link with your switch. You then need to force the link up by using this command : ip link set enp7s0 up. enp7s0 is the name of my 10g adapter, you need to change it to what ever proxmox assigns to your 10g adapter. After this , it still will not work until you assign it to a bridge. I had never done this before but figured it out by looking at my existing configuration. I was fairly certain this would cause a network conflic and it did until I pulled the 1g cable out of nuc. I also had to reboot my 10g switch but afterwards it started working. After this, you should not have to manually start the interface again. I want to set this up as a vlan but until I can find some decent instructions this will work and it should work for you as well. Good luck!

Thanks so much for this. I can confirm my Sonnet Solo 10G Thunderbolt 3 NIC works on my Intel NUC12 Pro after forcing it up with ip link set enp60s0 up as before it was just showing up as an interface, but it refused to go online and my switch was saying the link was down. All good now at 10G speed.