M.2 Stick killed my NIC PVE 8.1.3.

[Smash_Override]

I am still new to Proxmox and Linux learning the hard way through doing. I greatly appreciate everyone in this community!

Config:
Proxmox PVE 8.1.3
CPU: AMD Ryzen Threadripper 2990WX 32-Core Processor
MB: ASROCK X399
RAM: 64GB
Storage: 1x 4TB Spinning Rust (hosting Proxmox 7.2.3)
3x 12TB Spinning Rust TrueNAS Storage

I installed the M.2 4TB NVME stick, deployed PVE 8.1.3 to said stick as a new install, bridged network to enp3s0 at install in the gui, and now no link lights on the switch. I know it’s not a physical layer problem because the link light does come on when the machine first power on. But once Proxmox starts to load the light goes out.

I have googled the S*** out of this and come up with a few potential causes.

1. PCIe lD has changed after installing NVME stick? Not sure why this would matter since I installed a fresh new copy on the stick and setting it up for the first time? Screenshots attached.
2. Wrong driver? I looked up the driver for the Aquantia AQC-107 (on MB) and the readme says “This driver supports Linux kernels >= 3.10, and includes support for x86_64 and ARM Linux system.”. The “ethtool --driver” output just says atlantic version 6.5.11-4-pve.
3. NIC is down vs up? I ran the “ifup -v” command for that specific interface enp3s0 and I am not really sure what it was telling me so hopefully someone can tell me if this is important. I also ran “ip link set dev enp3s0 up” because I saw it in another post somewhere and still no joy.

If anyone has any direction to point me and what commands to type or at least where I could find said commands (cause FULL NEWB), I would greatly appreciate your vectors!

 

[leesteken]

PCI ID's change (by one) when adding (or removing) PCI(e) devices or enabling/disabling on-board devices. The name of the network devices is based on the PCI ID. Login to the Proxmxo host console and determine the new network device name with ip a and adjust the network configuration accordingly with nano /etc/network/interfaces.

 

[Smash_Override]

So a couple of questions.
1. If this is a fresh install of PVE, why would PCI ID matter? In my mind PVE would examine the hardware for the first time and configure the interfaces file accordingly because there is no previous record of devices and assigned PCI IDs? Does it not work like that?

2. Please forgive my ignorance because I really am this new to PVE/Linux. Looking at the attached images from "ip a" and the interfaces file, what exact lines do I need to change and what does it need to look like? After I make said changes, I assume a reboot should bring everything online?

3. Unrelated to my problems, how does one make the font of a command on this forum appear the way you did in the previous post?

 

[leesteken]

So a couple of questions.
1. If this is a fresh install of PVE, why would PCI ID matter? In my mind PVE would examine the hardware for the first time and configure the interfaces file accordingly because there is no previous record of devices and assigned PCI IDs? Does it not work like that?

Right, sorry, I did not read carefully enough and I am mistaken. You probably have some driver issue or the network controller is not compatible with Linux out of the box.

2. Please forgive my ignorance because I really am this new to PVE/Linux. Looking at the attached images from "ip a" and the interfaces file, what exact lines do I need to change and what does it need to look like? After I make said changes, I assume a reboot should bring everything online?

It looks fine, see my apology above.

3. Unrelated to my problems, how does one make the font of a command on this forum appear the way you did in the previous post?

Use the (inline) code button, which are just above the message you are writing (unless something is broken).

 

[emunt6]

Hi!

add "auto" directive to the "/etc/network/interfaces"

auto enp3s0
iface enp3s0 inet manual

$> systemctl restart networking

 

[leesteken]

add "auto" directive to the "/etc/network/interfaces"

auto enp3s0
iface enp3s0 inet manual

That's usually not necessary for devices used as bridge ports (see the example in the manual). Why is it needed here?

 

[emunt6]

That's usually not necessary for devices used as bridge ports (see the example in the manual). Why is it needed here?

https://manpages.debian.org/testing/ifupdown2/interfaces.5.en.html

Lines beginning with the word "auto" are used to identify the physical interfaces to be brought up when ifup is run with the -a option. (This option is used by the system boot scripts.) Physical interface names should follow the word "auto" on the same line. There can be multiple "auto" stanzas.

 

[Smash_Override]

@leesteken Thanks for trying. I have considered reinstalling the driver...just need to learn how to put it on a flash drive in a way that would be readable to Linux? Or is NTFS/Windows 10 default file system readable by Linux? Then how to mount the flash drive in a usb port, copy and un tar the file into the right location. The be honest, I don't even know if I said all that correctly? How'd I do?

@emunt6 I went ahead and added auto enp3s0 about iface enp3s0 inet manual as recommended. No Joy. Unless I did it wrong?

 

[leesteken]

just need to learn how to put it on a flash drive in a way that would be readable to Linux? Or is NTFS/Windows 10 default file system readable by Linux? Then how to mount the flash drive in a usb port, copy and un tar the file into the right location. The be honest, I don't even know if I said all that correctly? How'd I do?

I'm sure there are lots of guides for Linux out there. Maybe FAT (in one of its many variants) would be a better start than NTFS (and most USB drives come with FAT by default)? Installing a driver on Linux is not the same as Windows. Make sure the manufacturer supports Linux and the kernel version that you are currently using (uname -a).

I went ahead and added auto enp3s0 about iface enp3s0 inet manual as recommended. No Joy. Unless I did it wrong?

Personally, I think it has no effect as the bridge-ports are automatically brought up when the virtual bridge is. It might even hinder the proper order of operations, not sure.

 

[emunt6]

As @leesteken mentioned, it seems driver problem.
You have I211 chipset, but from the pics (ethtool) reporting insane speeds "5G/10G".

Can you post the output of this command?

$> lshw -c network

 

[Smash_Override]

@leesteken I reverted the interfaces file back to the original without the auto enp3s0. The driver in PVE 7.3 appeared to work fine. I am looking for tutorials now on how to install the driver I downloaded from Atlantic. The Readme says it will support Linux kernels >= 3.10 so maybe this will work?

@emunt6 lshw command not found. It is a 10 gig ethernet port even though the rest of network is 1gig.

 

[emunt6]

@leesteken I reverted the interfaces file back to the original without the auto enp3s0. The driver in PVE 7.3 appeared to work fine. I am looking for tutorials now on how to install the driver I downloaded from Atlantic. The Readme says it will support Linux kernels >= 3.10 so maybe this will work?

@emunt6 lshw command not found. It is a 10 gig ethernet port even though the rest of network is 1gig.

$> apt-get install lshw

The drivers is built with the kernel (included), try another version of kernel ( yours is 6.5 )

$> apt-get install pve-kernel-6.2

Boot the system with 6.2 kernel.

 

[emunt6]

@emunt6 lshw command not found. It is a 10 gig ethernet port even though the rest of network is 1gig.

Okay, so your 10G card is not working "Aquantia AQC107" - (red line on the pic was misleading).
Maybe the "autoneg" is not working properly, try to set manually the speed and other side too:

example:

$> ethtool -s enp3s0 autoneg off speed 1000

 

[Smash_Override]

Okay, so your 10G card is not working "Aquantia AQC107" - (red line on the pic was misleading).
Maybe the "autoneg" is not working properly, try to set manually the speed and other side too:

example:

$> ethtool -s enp3s0 autoneg off speed 1000

@emunt6 Sorry that my screenshot was misleading. Intent was to show what I believed was all my NIC devices.

I ran the command as instructed and got an operation not permitted response. In the screen shot you will see I changed the speed to 100 vs 1000 because at the moment it is hitting a cheap 10/100 poe switch. I have eliminated the switch as the problem because I have other devices plugged in that are working fine and before I threw the M.2 into this machine, it was working. I normally keep this machine in my closet wired to my unifi gig switch.

I did figure out how to mount the usb flash drive with the driver I downloaded from the NIC manufacturer. I unzipped the tar.gz file into directory I created in /mnt/usbstick/unzipped directory just to play with the commands and to keep it in a place I would remember so I can delete later. Now that I know how to use the tar -xvzf filename.tar.gz -C /mnt/usbstick/unzipped command...how do I install this driver? Do I unzip it to a specific directory? Orrrrr.....apt-get something?

I also tried the get kernel-6.2 and lshw with no luck. Unable to locate.

Thank you all for putting up with my ignorance. This is painful but in a weird way I am enjoying learning how to navigate these things.

 

[emunt6]

@emunt6 Sorry that my screenshot was misleading. Intent was to show what I believed was all my NIC devices.

I ran the command as instructed and got an operation not permitted response. In the screen shot you will see I changed the speed to 100 vs 1000 because at the moment it is hitting a cheap 10/100 poe switch. I have eliminated the switch as the problem because I have other devices plugged in that are working fine and before I threw the M.2 into this machine, it was working. I normally keep this machine in my closet wired to my unifi gig switch.

I did figure out how to mount the usb flash drive with the driver I downloaded from the NIC manufacturer. I unzipped the tar.gz file into directory I created in /mnt/usbstick/unzipped directory just to play with the commands and to keep it in a place I would remember so I can delete later. Now that I know how to use the tar -xvzf filename.tar.gz -C /mnt/usbstick/unzipped command...how do I install this driver? Do I unzip it to a specific directory? Orrrrr.....apt-get something?

I also tried the get kernel-6.2 and lshw with no luck. Unable to locate.

Thank you all for putting up with my ignorance. This is painful but in a weird way I am enjoying learning how to navigate these things.

Before you change the kernel, try to place the NIC-card to another PCI-E slot ( it possible some PCI-bus problem or that slot is shared with M2's PCI-E slot) - I have seen this, changing PCI-E card position solved the problem.

# Update repo database

$> apt-get clean all
$> apt-get autoclean
$> apt-get update

# Install package

$> apt-get install lshw
$> apt-get install pve-kernel-6.2

 

[Smash_Override]

@emunt6 I will try moving the nvme stick to another slot since I have three. The NICs are built into the board so moving the stick is all I can do there. I am looking for instructions on how to manually install the driver. What I have found so far is that I need to install Linux headers before running ./dkms.sh install from the .tar.gz file I downloaded from Atlantic;s site. I am not even sure if this is how to properly install a driver so my apologies if this is the wrong way of doing things.

 

[Smash_Override]

@emunt6 So I ran through the commands as instructed...but how can this machine update from any of these sources when my NIC isn't working? Unfortunately nothing was downloaded due to the NIC being in op.

I did move the M.2 NVME to a different slot and the NIC is still dead in the water...but...On boot as all the lines of text were scrolling I saw a message for a very short second that said something to the effect of "ethernet out of bounds". Is there a way that I can pull up all that information that loaded prior to the login promt? A boot log somewhere?

 

[Smash_Override]

@leesteken @emunt6

So I did some googling on logs and ran "journalctl -b". At line 1644 I found the out of bounds error and it reads as follows...

UBSAN: array-index-out-of-bounds in driver/net/ethernet/aquantia/atlantic/aq_nic.c:484:48 index 8 is out of range for type 'aq_vec_s *[8]'

So what does this mean to everyone? I need to load the driver? If so, now I have to learn how to load the driver from the terminal. Good news is that I already loaded it to /mnt from my usb flash drive. Pretty sure I need to load Linux Headers and then I can run the tar.gz file? Is this logical or do I just sound like the guy licking the walls saying "The snozzberries taste like snozzberries"?

Thanks again for all the life rafts everyone has been throwing me.

 

[leesteken]

UBSAN: array-index-out-of-bounds in driver/net/ethernet/aquantia/atlantic/aq_nic.c:484:48 index 8 is out of range for type 'aq_vec_s *[8]'

That does look like a bug in the driver, detected by UBSAN.

Pretty sure I need to load Linux Headers and then I can run the tar.gz file?

Install the kernel headers with apt install pve-headers (but sometimes a more specific version is needed). Make sure you enabled a Proxmox repository ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories ) and are already fully up to date ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#system_software_updates ) first.

Follow the instructions from the manufacturer about unzipping the tar.gz and compiling and installing the driver. This is more or less the same for all kinds of third-party drivers that are not in the Linux kernel (and not specific to Proxmox). Make sure the kernel version (uname -a) is supported by the driver/manufacturer.

EDIT: These kind of issues are usually fixed in newer Linux kernel versions, so update your Proxmox if possible. Then you don't need to install a driver yourself. And you'll need an internet connection to installed all the requirements for the updates and your driver, so maybe install a working network card first.

 

[pengu1n]

sorry to but in but from the first post it seems you have a Proxmox v 7.2.3, so was the NIC was working before? Trying to see if it was and then the whole installation of a driver is unnecessary.
Additionally, your pictures have with ethtool what seems to me like there is a working driver (maybe not the latest but shouldn't be a problem in itself). The red line I don't yet think is misleading. What I _think_ is that you have one add-on card with three ports, each can sync at 100, 1000 i.e. Gigabit, 2.5, 5 and 10 Gbit. However the link is down and can't establish at any speed. Also that you have some wireless nic, and what throws me a bit is the messages of finding Intel I211, which are normally used in wired (not Wifi) NICs.
Can you confirm what are the addon cards and how many ports, to get a lay of the land sort of, and please check that on the switch you might have to set the port from auto to a specific link speed.
Edit: you've been asked this auto speed already.
My gut feeling is that the problem is not driver but getting the link up, especially given that

"I ran the command as instructed and got an operation not permitted response. In the screen shot you will see I changed the speed to 100 vs 1000 because at the moment it is hitting a cheap 10/100 poe switch. I have eliminated the switch as the problem because I have other devices plugged in that are working fine and before I threw the M.2 into this machine, it was working. I normally keep this machine in my closet wired to my unifi gig switch."