networking.service not starting since kernel 6.8.12-1-pve

[tangiv]

Hello,

Since I upgraded my Proxmox 8.2 to kernel 6.8.12-1-pve I have an issue with networking, the service won't start automaticaly at boot.
If I start the networking.service manually (systemctl start networking.service) everything is fine.
For me the network interface names didn't change after the upgrade to 8.2, indeed if I do an ip a I still have the old names but with altnames.

What I tried :

• systemctl enable networking => fail
• rename old interface with new name => fail
• boot in 6.5.13-6 => ok (altnames where already available, identical to 6.8.12, still unused)
• boot in 6.8.8-4 => ok (altnames where already available, identical to 6.8.12, still unused)
• boot in 6.8.12-1 => fail (altnames available in cli not in the webui)
• ifdown && ifup with new names => fail (cf screenshot)

For now I have a cron :
@reboot root sleep 60 && /usr/bin/systemctl start networking

Spoiler: NICs

02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
12:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
12:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)

Spoiler: Packages

proxmox-ve: 8.2.0 (running kernel: 6.8.8-4-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-1
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1
proxmox-kernel-6.8.8-4-pve-signed: 6.8.8-4
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.174-2-pve: 5.4.174-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph: 17.2.7-pve3
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
intel-microcode: 3.20240514.1~deb12u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.2
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
openvswitch-switch: 3.1.0-2+deb12u1
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.13-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.2-2
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

Spoiler: NIC map

eno1 => enp2s0f0
eno2 => enp2s0f1
eno3 => enp2s0f2
eno4 => enp2s0f3

ens1f0 => enp18s0f0
ens1f1 => enp18s0f1

Spoiler: Network config

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
address 192.168.100.106/24
gateway 192.168.100.1
#mgmt

iface eno2 inet manual

iface eno3 inet manual

auto ens1f0
iface ens1f0 inet static
address 192.168.50.106/24
#STOR

iface ens1f1 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge-ports ens1f1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#VM

Somewhat I think I'm in a similar situation of this post : https://forum.proxmox.com/threads/no-networking-after-upgrade-to-8-2.145727/post-665799

If I have to switch to the new names it's absolutely not a problem but they doesn't seem to work and I still see the old names everywhere.

Thanks for your help

 

[Moayad]

Hi,

Could you please try to restart the server, then provide us with the output of `ifreload -a -d` and `systemctl status networking.service`?

 

[tangiv]

In fact sometimes it works and sometimes it doesn't, hard to reproduce.
Here is the output, nothing special in the output of `systemctl status networking.service`.

 

[Moayad]

Is there something written in the ifreload file `cat ifreload`?

 

[tangiv]

empty

 

[Moayad]

Can you please attach the syslog from boot to this thread?

You can generate the syslog since last boot using journalctl as the following command:

journalctl -b > /tmp/$(hostname)-syslog.txt

 

[tangiv]

Here is the output

 

[reetp]

Possibly related but I had a similar experience on 6.8.8 after upgrading to v8

Previously I just had the bridges set to autostart on boot but it seems this may have changed and I needed to set the actual network devices to autostart as well.

After that I rebooted and it came up OK.

 

[reetp]

snmpd[1737]: systemstats_linux: unexpected header length in /proc/net/snmp. 237 != 224

Seen that too:

https://github.com/net-snmp/net-snmp/issues/829

 

[coro.nas]

also breaks networking on all my NICs, a downgrade to kernel 6.8.8-4 fixed this

System was showing these error messages previously

mlx4_en: enp1s0: TX timeout on queue: 2, QP: 0x20a, CQ: 0x88, Cons: 0x4/48, Prod: 0x4ffe

mlx4_core 0000:01:00.0 enp1s0: NETDEV WATCHDOG: CPU: 2: transmit queue 2 timed out 1300224 ms

 

[ElvisCZ]

same problem here. 6.8.12-1-pve kills the network. it was necessary to pin the old kernel.