networking.service not starting since kernel 6.8.12-1-pve

[tangiv]

Hello,

Since I upgraded my Proxmox 8.2 to kernel 6.8.12-1-pve I have an issue with networking, the service won't start automaticaly at boot.
If I start the networking.service manually (systemctl start networking.service) everything is fine.
For me the network interface names didn't change after the upgrade to 8.2, indeed if I do an ip a I still have the old names but with altnames.

What I tried :

• systemctl enable networking => fail
• rename old interface with new name => fail
• boot in 6.5.13-6 => ok (altnames where already available, identical to 6.8.12, still unused)
• boot in 6.8.8-4 => ok (altnames where already available, identical to 6.8.12, still unused)
• boot in 6.8.12-1 => fail (altnames available in cli not in the webui)
• ifdown && ifup with new names => fail (cf screenshot)

For now I have a cron :
@reboot root sleep 60 && /usr/bin/systemctl start networking

Spoiler: NICs

02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
12:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
12:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)

Spoiler: Packages

proxmox-ve: 8.2.0 (running kernel: 6.8.8-4-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-1
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1
proxmox-kernel-6.8.8-4-pve-signed: 6.8.8-4
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.174-2-pve: 5.4.174-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph: 17.2.7-pve3
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
intel-microcode: 3.20240514.1~deb12u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.2
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
openvswitch-switch: 3.1.0-2+deb12u1
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.13-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.2-2
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

Spoiler: NIC map

eno1 => enp2s0f0
eno2 => enp2s0f1
eno3 => enp2s0f2
eno4 => enp2s0f3

ens1f0 => enp18s0f0
ens1f1 => enp18s0f1

Spoiler: Network config

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
address 192.168.100.106/24
gateway 192.168.100.1
#mgmt

iface eno2 inet manual

iface eno3 inet manual

auto ens1f0
iface ens1f0 inet static
address 192.168.50.106/24
#STOR

iface ens1f1 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge-ports ens1f1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VM

Somewhat I think I'm in a similar situation of this post : https://forum.proxmox.com/threads/no-networking-after-upgrade-to-8-2.145727/post-665799

If I have to switch to the new names it's absolutely not a problem but they doesn't seem to work and I still see the old names everywhere.

Thanks for your help

 

[Moayad]

Hi,

Could you please try to restart the server, then provide us with the output of `ifreload -a -d` and `systemctl status networking.service`?

 

[tangiv]

In fact sometimes it works and sometimes it doesn't, hard to reproduce.
Here is the output, nothing special in the output of `systemctl status networking.service`.

 

[Moayad]

Is there something written in the ifreload file `cat ifreload`?

 

[tangiv]

empty

 

[Moayad]

Can you please attach the syslog from boot to this thread?

You can generate the syslog since last boot using journalctl as the following command:

journalctl -b > /tmp/$(hostname)-syslog.txt

 

[tangiv]

Here is the output

 

[reetp]

Possibly related but I had a similar experience on 6.8.8 after upgrading to v8

Previously I just had the bridges set to autostart on boot but it seems this may have changed and I needed to set the actual network devices to autostart as well.

After that I rebooted and it came up OK.

 

[reetp]

snmpd[1737]: systemstats_linux: unexpected header length in /proc/net/snmp. 237 != 224

Seen that too:

https://github.com/net-snmp/net-snmp/issues/829

 

[coro.nas]

also breaks networking on all my NICs, a downgrade to kernel 6.8.8-4 fixed this

System was showing these error messages previously

mlx4_en: enp1s0: TX timeout on queue: 2, QP: 0x20a, CQ: 0x88, Cons: 0x4/48, Prod: 0x4ffe

mlx4_core 0000:01:00.0 enp1s0: NETDEV WATCHDOG: CPU: 2: transmit queue 2 timed out 1300224 ms

 

[ElvisCZ]

same problem here. 6.8.12-1-pve kills the network. it was necessary to pin the old kernel.
fix ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#network_override_device_names ) works good.

 

[UnitZero]

All my NICs are also DOWN on 6.8.12-1. Ethernet LEDs don't even light up after the OS boots. Pinning older kernel works.

 

[tasmandevil]

Yeah same here, it broke our whole cluster. We've been on 6.8.12-2-pve
This helped on all of our nodes sudo pve-efiboot-tool kernel pin 6.5.13-6-pve
assuming you have this kernel installed

 

[mattlach]

Do you by any chance have a NIC that depends on the i40e kernel driver?

I just had this problem, posted in the forums and solved it with the help of another member. I figured I'd drop this here in case anyone else is searching for a solution and comes across this thread, and not one that has the solution in it.

It turns out that in their wisdom, the devs who maintain the i40e kernel NIC driver decided to rename the network devices by adding np0, np1 etc, for network port numbers at the end of them.

So, when you boot with kernel 6.8.x on a system that was originally installed using a kernel from before 6.8.x and you have one of the affected NIC's, the device names change, and as a result the system can't bring up the network.

Luckily the fix is as easy as finding the new device names using "ip a", and editing /etc/network/interfaces to match.

For me, this meant changing my enp133s0f0 and enp133s0f1 to enp133s0f0np0 and enp133s0f1np1 respectively.
I feel sorry for everyone with a remote server and no out of band management capability who have to go on a road trip because of this one. What were they thinking?

Shout out to @carles89 for pointing me in the right direction in my thread here.