network card drop, igc 0000:09:00.0 eno1: PCIE link lost

AZYxV said:
Hello, have you found a solution to this problem? I have the same problem and I can't solve it. I'm using proxmox 8.03 on a bee-link SER 6 MAX mini pc. Anyone can help me please ?
Click to expand...
I have the exact same computer, the Beelink SER 6 Max and am having the same issue. While I'm glad it's not a hardware issue with the computer itself, it's obviously an issue with the driver or something like that. I haven't tried any of the above solutions yet. The odd part is that this was working for about 3 weeks with no issues. The problem started yesterday after a reboot. I'm going to try a reinstall for starters.
 
Supermicro H13SAE-MF with 2 x i210 onboard. I have 4 servers with this board, all with the same problem. Disabled all Power Management.
Since this is a verry old problem, maybe no fix in the near future will add some i350 network cards to fix the random loss of network.

root@PVE-01:~# cat /proc/cmdline
initrd=\EFI\proxmox\6.5.11-7-pve\initrd.img-6.5.11-7-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs pcie_aspm=off pcie_port_pm=off
 
lakeki said:
I have the exact same computer, the Beelink SER 6 Max and am having the same issue. While I'm glad it's not a hardware issue with the computer itself, it's obviously an issue with the driver or something like that. I haven't tried any of the above solutions yet. The odd part is that this was working for about 3 weeks with no issues. The problem started yesterday after a reboot. I'm going to try a reinstall for starters.
Click to expand...
The solution was a BIOS update for the computer provided by the manufacturer, no issues after that. I found out when I tried to reinstall Proxmox and had the network drop during the install. This also happened with Ubuntu.
 
Trying smth now, I had in BIOS ASPM Off and kernel parameters "pcie_aspm=off pcie_port_pm=off"

root@PVE-01:~# lspci -vvv | grep LnkCtl:
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

Removed "pcie_aspm=off pcie_port_pm=off" and added only "pcie_aspm.policy=performance"

root@PVE-01:~# lspci -vvv | grep LnkCtl:
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+


And yes the Intel i210 was "ASPM L1 Enabled"
 
  • Like
Reactions: proxale
try to disable network card offloading
ethtool must be installed


iface eno1 inet manual
pre-up ethtool --offload $IFACE rx off tx off sg off tso off gso off gro off lro off rxvlan off txvlan off rxhash off

iface eno2 inet manual
pre-up ethtool --offload $IFACE rx off tx off sg off tso off gso off gro off lro off rxvlan off txvlan off rxhash off
 
proteus said:
Supermicro H13SAE-MF with 2 x i210 onboard. I have 4 servers with this board, all with the same problem. Disabled all Power Management.
Since this is a verry old problem, maybe no fix in the near future will add some i350 network cards to fix the random loss of network.

root@PVE-01:~# cat /proc/cmdline
initrd=\EFI\proxmox\6.5.11-7-pve\initrd.img-6.5.11-7-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs pcie_aspm=off pcie_port_pm=off
Click to expand...
Same here, 1 Server with Supermicro H13SAE-MF with 2 x i210. One of the nics keeps crashing. Now trying with
pcie_port_pm=off pcie_aspm.policy=performance
I hope that will get solved...
 
You can check with

lspci -vvv | grep LnkCtl:

The output should be smth like this:

LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
 
  • Like
Reactions: proxale and henkesn
Warning for ZFS users: if you are booting to ZFS disks, updating /etc/default/grub and update-grub will not help.

If you want to add pcie_port_pm=off pcie_aspm.policy=performance to the kernel even when booting on ZFS, you must add these parameters to /etc/kernel/cmdline and then run proxmox-boot-tool refresh. After the reboot, you should already find these parameters by cat /proc/cmdline.

I also use a SuperMicro H13SAE-MF motherboard with Ryzen 7950X3D and the latest Proxmox VE 8.2, and after a few days the server always disconnected from the network. I also tried different kernel versions from 6.2 to 6.8 and only this helped.
 
jan.reges said:
Warning for ZFS users: if you are booting to ZFS disks, updating /etc/default/grub and update-grub will not help.

If you want to add pcie_port_pm=off pcie_aspm.policy=performance to the kernel even when booting on ZFS, you must add these parameters to /etc/kernel/cmdline and then run proxmox-boot-tool refresh. After the reboot, you should already find these parameters by cat /proc/cmdline.

I also use a SuperMicro H13SAE-MF motherboard with Ryzen 7950X3D and the latest Proxmox VE 8.2, and after a few days the server always disconnected from the network. I also tried different kernel versions from 6.2 to 6.8 and only this helped.
Click to expand...
I think this is not related to ZFS, it depends if Proxmox is installed with GRUB or Systemd-boot.

For more details:
Link
 
Last edited:
  • Like
Reactions: jan.reges
proteus said:
Trying smth now, I had in BIOS ASPM Off and kernel parameters "pcie_aspm=off pcie_port_pm=off"

root@PVE-01:~# lspci -vvv | grep LnkCtl:
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

Removed "pcie_aspm=off pcie_port_pm=off" and added only "pcie_aspm.policy=performance"

root@PVE-01:~# lspci -vvv | grep LnkCtl:
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk-
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+


And yes the Intel i210 was "ASPM L1 Enabled"
Click to expand...

Good evening,

I also use H13SAE-MF mainboards and now have the same problem with Ubuntu 22.04. What exactly was your solution with the H13SAE-MF mainboards?
 
dbr said:
Good evening,

I also use H13SAE-MF mainboards and now have the same problem with Ubuntu 22.04. What exactly was your solution with the H13SAE-MF mainboards?
Click to expand...
I have the same board Supermicro H13SAE-MF board with the Intel i210 NIC for a new server and resolved the problem based on the post by @proteus

No changes to BIOS or other settings, only added pcie_aspm.policy=performance to kernel parameters, so far the NIC has stayed alive for >24 hours with low/no network activity on the server.

I tested also pcie_aspm=disabled and/or pcie_port_pm=off which did not change the outcome, the NIC would be unresponsive within hours if there are no network activity.

Update: ProxMox host and VM had been up for a week without encountering the link lost issue since adding the performance parameter.
 
Last edited:
You must log in or register to reply here.
Top Bottom