network card drop, igc 0000:09:00.0 eno1: PCIE link lost

hashacki · Jan 19, 2023

Hello everyone,

I am coming to you today because I am really troubled. Completely randomly, I have this error message: igc 0000:09:00.0 eno1: PCIE link ost igc failed to read reg 0xc030!

I have tried a lot of things:

Change ethernet cable
Change switch
Downgrade my BIOS (0611 0705 and 0805)
Test 3 versions of proxmox: 6.4-1 7.2 and 7.3-4
I added this parameter to the grub: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_port_pm=off pcie_aspm.policy=performance"

Unless I missed something, in the BIOS, there is no network card standby or different types of performance.
I have the feeling that it is not the right "network driver", but it exceeds my skills.

A hardware point:

Motherboard: ASUS ROG STRIX X670E-E GAMING WIFI
CPU: AMD Ryzen 9 7950X
SSD: Kingston KC3000 512 Go (*2 with ZFS RAID 1)
RAM: G.Skill RipJaws S5 Low Profile 32 Go (2 x 16 Go) DDR5 5600 MHz CL30

Do you have a solution for me or a clue to give me?

Thank you for reading me and thank you in advance.

Neobin · Jan 19, 2023

With (again?!) on PVE 7, you could try a newer kernel: [1].

Otherwise I have no idea, sorry.

[1] https://forum.proxmox.com/threads/opt-in-linux-6-1-kernel-for-proxmox-ve-7-x-available.119483

gobuchul74 · Jan 23, 2023

I just had a nearly identical problem. It's a new server I'm setting up. Clean install of the latest Proxmox VE with the latest no-subscription updates. The server was powered on, but not really doing anything. I lost network connectivity to it and found the following on the server screen.

[ 8166.094987] igb 0000:03:00.0 eno1: PCIe link lost
[ 8297.261277] igb 0000:03:00.1 eno2: PCIe link lost

Im using a Supermicro X10DRI motherboard (using i350 gigabit NICs)

The only thing I did with it today was install an NVME drive on an PCI x16 card, which required the slot to be bifurcated. Rebooting fixed the issue for now, but I don't feel I can trust this thing.

[UPDATE} Possible solution
This may be a power management issue on the NIC. It seems to be similar to other issues reported on various internet sites.

From Reddit:
We can disable power management on the PCIe entirely with pcie_port_pm=off

In the file /etc/default/grub, line GRUB_CMDLINE_LINUX_DEFAULT we can add pcie_port_pm=off and then run update-grub to rebuild the boot config.

hashacki · Jan 23, 2023

Neobin said:
With (again?!) on PVE 7, you could try a newer kernel: [1].

Otherwise I have no idea, sorry.

[1] https://forum.proxmox.com/threads/opt-in-linux-6-1-kernel-for-proxmox-ve-7-x-available.119483

i will try, thanks

gobuchul74 said:
I just had a nearly identical problem. It's a new server I'm setting up. Clean install of the latest Proxmox VE with the latest no-subscription updates. The server was powered on, but not really doing anything. I lost network connectivity to it and found the following on the server screen.

[ 8166.094987] igb 0000:03:00.0 eno1: PCIe link lost
[ 8297.261277] igb 0000:03:00.1 eno2: PCIe link lost

Im using a Supermicro X10DRI motherboard (using i350 gigabit NICs)

The only thing I did with it today was install an NVME drive on an PCI x16 card, which required the slot to be bifurcated. Rebooting fixed the issue for now, but I don't feel I can trust this thing.

[UPDATE} Possible solution
This may be a power management issue on the NIC. It seems to be similar to other issues reported on various internet sites.

From Reddit:
We can disable power management on the PCIe entirely with pcie_port_pm=off

In the file /etc/default/grub, line GRUB_CMDLINE_LINUX_DEFAULT we can add pcie_port_pm=off and then run update-grub to rebuild the boot config.

I do'nt understand why you change the emplacement.

And from REDDIT if you look the topic and mine, he try to add the power management and it doesn't work. Topic was solved but a update say that it's doesn't work

hashacki · Jan 25, 2023

Neobin said:
With (again?!) on PVE 7, you could try a newer kernel: [1].

Otherwise I have no idea, sorry.

[1] https://forum.proxmox.com/threads/opt-in-linux-6-1-kernel-for-proxmox-ve-7-x-available.119483

hi !
i've install the 6.1 kernel and 40 hours later, same problem

gobuchul74 · Jan 28, 2023

I've tried the 5.15 kernel driver:
root@pve:~# ethtool -i eno1
driver: igb
version: 5.15.83-1-pve
firmware-version: 1.63, 0x80000a05
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

I've tried the 6.1 kernel driver:
root@pve:~# ethtool -i eno1
driver: igb
version: 6.1.2-1-pve
firmware-version: 1.63, 0x80000a05
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Now I'm trying the most recent driver on Intel's website:
root@pve:~# ethtool -i eno1
driver: igb
version: 5.13.7
firmware-version: 1.63, 0x80000a05
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

I will update if the problem continues......

gobuchul74 · Jan 31, 2023

gobuchul74 said:
I've tried the 5.15 kernel driver:
root@pve:~# ethtool -i eno1
driver: igb
version: 5.15.83-1-pve
firmware-version: 1.63, 0x80000a05
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

I've tried the 6.1 kernel driver:
root@pve:~# ethtool -i eno1
driver: igb
version: 6.1.2-1-pve
firmware-version: 1.63, 0x80000a05
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Now I'm trying the most recent driver on Intel's website:
root@pve:~# ethtool -i eno1
driver: igb
version: 5.13.7
firmware-version: 1.63, 0x80000a05
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

I will update if the problem continues......

I'm comfortable saying the new driver resolved this issue. Loading the driver is a bit of a hassle, but here's how I did it.

Download latest driver (5.13.7) from Intel's website.
https://www.intel.com/content/www/us/en/download/14098/intel-network-adapter-driver-for-82575-6-82580-i350-and-i210-211-based-gigabit-network-connections-for-linux.html

Create a directory called IGB in the Proxmox /root directory.
Place the downloaded tar in /root/IGB

Prep system to build driver.

apt-get install linux-headers-$(uname -r)
apt-get install gcc make ethtool

Ethtool will show current driver.

root@pve:~# ethtool -i eno1
driver: igb
version: 6.1.2-1-pve
firmware-version: 1.63, 0x80000a05
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Extract tar file.
tar zxf igb-5.13.7.tar.gz

CD into 'src' directory.

cd igb-5.13.7/src/

To avoid build errors, edit 'igb_main.c', use nano or whatever you prefer.
Locate line 1145
Near the end of the line will be "igb_poll, 64);"
Edit the line as such "igb_poll);"

Save igb_main.c
Now it will be ready to build.

make install

Reboot, and ethtool should show the new driver.

root@pve:~# ethtool -i eno1
driver: igb
version: 5.13.7
firmware-version: 1.63, 0x80000a05
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
root@pve:~#

hashacki · Feb 1, 2023

Hi, thanks for this answer. I try this and test.
I will respond in a week if there hasn't been a drop to confirm that this solution works.

Neobin · Feb 1, 2023

gobuchul74 said:
apt-get install linux-headers-$(uname -r)

Maybe better use the: pve-headers instead.

hashacki · Feb 1, 2023

Network card was drop ... after 20mn working

gobuchul74 · Feb 2, 2023

hashacki said:
Network card was drop ... after 20mn working

It looks like your motherboard uses the I225 networking device. Googleing that doesn't lead to much linux info.

Neobin said:
Maybe better use the: pve-headers instead.

Sounds good, I don't really compile things very often. I'm going to reinstall with the 5.13 kernel.

hashacki · Feb 7, 2023

gobuchul74 said:
It looks like your motherboard uses the I225 networking device. Googleing that doesn't lead to much linux info.

Sounds good, I don't really compile things very often. I'm going to reinstall with the 5.13 kernel.

Hi, i've try to reinstall with the 5.13 kernel and newtork card was drop after 2days...

sentoz · Feb 7, 2023

Hello, friends.

Got stuck with the same problem.
The only thing is not on proxmox. But I think it will be the same for everyone.
At the moment I have tried different versions of linux kernels, from 5.18 to 6.2.0rc*, the problem persists everywhere.

I noticed that the problem is less pronounced when you limit the connection speed to 100Mbps

Tell me, did rebuilding the Intel driver from source code help?

gobuchul74 · Feb 7, 2023

sentoz said:
Hello, friends.

Got stuck with the same problem.
The only thing is not on proxmox. But I think it will be the same for everyone.
At the moment I have tried different versions of linux kernels, from 5.18 to 6.2.0rc*, the problem persists everywhere.

I noticed that the problem is less pronounced when you limit the connection speed to 100Mbps

Tell me, did rebuilding the Intel driver from source code help?

I've not had a repeat of the issue since I rebuilt the intel driver. However, I am using a different network adapter than hashacki.

sentoz · Feb 7, 2023

gobuchul74 said:
I've not had a repeat of the issue since I rebuilt the intel driver. However, I am using a different network adapter than hashacki.

Yes, I'm more interested in whether hashacki was able to solve the problem, because I have the same motherboard and processor.

sentoz · Feb 7, 2023

hashacki said:
Hi, i've try to reinstall with the 5.13 kernel and newtork card was drop after 2days...

Hello, I found a solution to the problem. The problem is with how Asus manages power. See reddit https://www.reddit.com/r/buildapc/comments/xypn1m/network_card_intel_ethernet_controller_i225v_igc/

hashacki · Feb 13, 2023

Hi, look my first post i've try to do this.... but thank you

I've see too after launch a VM, one hour after, network card drop. I d'ont understand where i can find a problem/solution

hashacki · Feb 17, 2023

hi,

I tried to check in the grub if the network card sleep settings were being properly applied. It turned out they were not. Actually, the boot file is not in /etc/default/grub, and Proxmox does not support it. However, it does support the cmdline file located in /etc/kernel/cmdline. I added the parameters quiet splash pcie_port_pm=off pcie_aspm.policy and amd_iommu=on iommu=pt. After two days, I experienced a new drop and a new error: igc 0000:08:00.0 eno1: PCIE link lost, not igc 0000:09:00.0 eno1: PCIE link lost.

Neobin · Feb 17, 2023

hashacki said:
Actually, the boot file is not in /etc/default/grub, and Proxmox does not support it. However, it does support the cmdline file located in /etc/kernel/cmdline.

It depends:

AZYxV · Oct 9, 2023

Hello, have you found a solution to this problem? I have the same problem and I can't solve it. I'm using proxmox 8.03 on a bee-link SER 6 MAX mini pc. Anyone can help me please ?

network card drop, igc 0000:09:00.0 eno1: PCIE link lost

New Member

Distinguished Member

New Member

New Member

New Member

New Member

New Member

New Member

Distinguished Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

Distinguished Member

New Member