network card drop, igc 0000:09:00.0 eno1: PCIE link lost

AZYxV said:
Hello, have you found a solution to this problem? I have the same problem and I can't solve it. I'm using proxmox 8.03 on a bee-link SER 6 MAX mini pc. Anyone can help me please ?
Click to expand...
I have the exact same computer, the Beelink SER 6 Max and am having the same issue. While I'm glad it's not a hardware issue with the computer itself, it's obviously an issue with the driver or something like that. I haven't tried any of the above solutions yet. The odd part is that this was working for about 3 weeks with no issues. The problem started yesterday after a reboot. I'm going to try a reinstall for starters.
 
Supermicro H13SAE-MF with 2 x i210 onboard. I have 4 servers with this board, all with the same problem. Disabled all Power Management.
Since this is a verry old problem, maybe no fix in the near future will add some i350 network cards to fix the random loss of network.

root@PVE-01:~# cat /proc/cmdline
initrd=\EFI\proxmox\6.5.11-7-pve\initrd.img-6.5.11-7-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs pcie_aspm=off pcie_port_pm=off
 
lakeki said:
I have the exact same computer, the Beelink SER 6 Max and am having the same issue. While I'm glad it's not a hardware issue with the computer itself, it's obviously an issue with the driver or something like that. I haven't tried any of the above solutions yet. The odd part is that this was working for about 3 weeks with no issues. The problem started yesterday after a reboot. I'm going to try a reinstall for starters.
Click to expand...
The solution was a BIOS update for the computer provided by the manufacturer, no issues after that. I found out when I tried to reinstall Proxmox and had the network drop during the install. This also happened with Ubuntu.
 
Trying smth now, I had in BIOS ASPM Off and kernel parameters "pcie_aspm=off pcie_port_pm=off"

root@PVE-01:~# lspci -vvv | grep LnkCtl:
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

Removed "pcie_aspm=off pcie_port_pm=off" and added only "pcie_aspm.policy=performance"

root@PVE-01:~# lspci -vvv | grep LnkCtl:
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+


And yes the Intel i210 was "ASPM L1 Enabled"
 
  • Like
Reactions: proxale
try to disable network card offloading
ethtool must be installed


iface eno1 inet manual
pre-up ethtool --offload $IFACE rx off tx off sg off tso off gso off gro off lro off rxvlan off txvlan off rxhash off

iface eno2 inet manual
pre-up ethtool --offload $IFACE rx off tx off sg off tso off gso off gro off lro off rxvlan off txvlan off rxhash off
 
proteus said:
Supermicro H13SAE-MF with 2 x i210 onboard. I have 4 servers with this board, all with the same problem. Disabled all Power Management.
Since this is a verry old problem, maybe no fix in the near future will add some i350 network cards to fix the random loss of network.

root@PVE-01:~# cat /proc/cmdline
initrd=\EFI\proxmox\6.5.11-7-pve\initrd.img-6.5.11-7-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs pcie_aspm=off pcie_port_pm=off
Click to expand...
Same here, 1 Server with Supermicro H13SAE-MF with 2 x i210. One of the nics keeps crashing. Now trying with
pcie_port_pm=off pcie_aspm.policy=performance
I hope that will get solved...
 
You can check with

lspci -vvv | grep LnkCtl:

The output should be smth like this:

LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
 
  • Like
Reactions: proxale and henkesn
Warning for ZFS users: if you are booting to ZFS disks, updating /etc/default/grub and update-grub will not help.

If you want to add pcie_port_pm=off pcie_aspm.policy=performance to the kernel even when booting on ZFS, you must add these parameters to /etc/kernel/cmdline and then run proxmox-boot-tool refresh. After the reboot, you should already find these parameters by cat /proc/cmdline.

I also use a SuperMicro H13SAE-MF motherboard with Ryzen 7950X3D and the latest Proxmox VE 8.2, and after a few days the server always disconnected from the network. I also tried different kernel versions from 6.2 to 6.8 and only this helped.
 
jan.reges said:
Warning for ZFS users: if you are booting to ZFS disks, updating /etc/default/grub and update-grub will not help.

If you want to add pcie_port_pm=off pcie_aspm.policy=performance to the kernel even when booting on ZFS, you must add these parameters to /etc/kernel/cmdline and then run proxmox-boot-tool refresh. After the reboot, you should already find these parameters by cat /proc/cmdline.

I also use a SuperMicro H13SAE-MF motherboard with Ryzen 7950X3D and the latest Proxmox VE 8.2, and after a few days the server always disconnected from the network. I also tried different kernel versions from 6.2 to 6.8 and only this helped.
Click to expand...
I think this is not related to ZFS, it depends if Proxmox is installed with GRUB or Systemd-boot.

For more details:
Link
 
Last edited:
  • Like
Reactions: jan.reges
proteus said:
Trying smth now, I had in BIOS ASPM Off and kernel parameters "pcie_aspm=off pcie_port_pm=off"

root@PVE-01:~# lspci -vvv | grep LnkCtl:
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

Removed "pcie_aspm=off pcie_port_pm=off" and added only "pcie_aspm.policy=performance"

root@PVE-01:~# lspci -vvv | grep LnkCtl:
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+


And yes the Intel i210 was "ASPM L1 Enabled"
Click to expand...

Good evening,

I also use H13SAE-MF mainboards and now have the same problem with Ubuntu 22.04. What exactly was your solution with the H13SAE-MF mainboards?
 
dbr said:
Good evening,

I also use H13SAE-MF mainboards and now have the same problem with Ubuntu 22.04. What exactly was your solution with the H13SAE-MF mainboards?
Click to expand...
I have the same board Supermicro H13SAE-MF board with the Intel i210 NIC for a new server and resolved the problem based on the post by @proteus

No changes to BIOS or other settings, only added pcie_aspm.policy=performance to kernel parameters, so far the NIC has stayed alive for >24 hours with low/no network activity on the server.

I tested also pcie_aspm=disabled and/or pcie_port_pm=off which did not change the outcome, the NIC would be unresponsive within hours if there are no network activity.

Update: ProxMox host and VM had been up for a week without encountering the link lost issue since adding the performance parameter.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back