3 PVE 8.1.4 fresh install and updated, after reboot networking does not start at boot.

[federico.fiordoliva]

Hey everyone,

I'm running into a bit of a snag here. I've got three identical Supermicro servers, and after a fresh install of PVE 8.1.4 on all of them, the networking service just won't kick in when they boot up.

I've attached the log for the networking service so you can see what's going on.

Jul 14 10:26:14 virt12 systemd[1]: Dependency failed for networking.service - Network initialization.
Jul 14 10:26:14 virt12 systemd[1]: networking.service: Job networking.service/start failed with result 'dependency'.

Basically, I have to hop onto the IPMI console and manually restart the networking service every single time. Once I do that, though, the network card and all the settings come alive. It's a bit of a pain!

The networking dependency is: ifupdown2-pre, this are the logs:

Jul 16 10:20:08 virt12 udevadm[2359]: Timed out for waiting the udev queue being empty.
Jul 16 10:20:08 virt12 systemd[1]: ifupdown2-pre.service: Main process exited, code=exited, status=1/FAILURE

The ifupdown2-pre dependency is: systemd-udevd:

Jul 16 10:19:08 virt12 systemd-udevd[1597]: bnxt_en.rdma.0: Worker [1600] processing SEQNUM=22324 is taking a long time
Jul 16 10:19:08 virt12 systemd-udevd[1597]: bnxt_en.rdma.1: Worker [1621] processing SEQNUM=22327 is taking a long time
Jul 16 10:21:08 virt12 systemd-udevd[1597]: bnxt_en.rdma.0: Worker [1600] processing SEQNUM=22324 killed
Jul 16 10:21:08 virt12 systemd-udevd[1597]: bnxt_en.rdma.1: Worker [1621] processing SEQNUM=22327 killed
Jul 16 10:21:08 virt12 systemd-udevd[1597]: bnxt_en.rdma.0: Worker [1600] terminated by signal 9 (KILL).
Jul 16 10:21:32 virt12 systemd-udevd[1597]: bnxt_en.rdma.1: Worker [1621] terminated by signal 9 (KILL).

Sounds like the problem is either with the Broadcom NetXtreme-E driver or the network configuration. I'm leaning away from the driver being the issue since it's just a basic active-backup bond that the default vmbr0 sits on.
Alright, so I updated the firmware on the Broadcom card using the .deb package from the manufacturer's website. The package installed without a hitch, but after a reboot, the new firmware just wouldn't load. I even updated the BIOS from Supermicro, but no luck, the problem's still there.

root@virt12:~# lspci -k | grep -EA3 'Ethernet|Network'
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
    DeviceName: Broadcom Ethernet BCM57416/5720L #1
    Subsystem: Super Micro Computer Inc BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
    Kernel driver in use: bnxt_en
    Kernel modules: bnxt_en
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
    DeviceName: Broadcom Ethernet BCM57416/5720L #2
    Subsystem: Super Micro Computer Inc BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
    Kernel driver in use: bnxt_en
    Kernel modules: bnxt_en
root@virt12:~# ethtool -i eno1np0
driver: bnxt_en
version: 6.8.12-11-pve
firmware-version: 226.0.145.0/pkg 226.1.107.1
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

The new firmware should be: 234.0.151.0-1

Can anyone lend a hand here?

 

[federico.fiordoliva]

I found this work around that resolve my problem.

root@virt12:~# echo "blacklist bnxt_re" | sudo tee /etc/modprobe.d/blacklist-bnxt_re.conf
blacklist bnxt_re
root@virt12:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-6.8.12-11-pve

Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/7441-CF01
    Copying kernel and creating boot-entry for 6.8.12-11-pve
    Copying kernel and creating boot-entry for 6.8.12-9-pve
Copying and configuring kernels on /dev/disk/by-uuid/7442-6A80
    Copying kernel and creating boot-entry for 6.8.12-11-pve
    Copying kernel and creating boot-entry for 6.8.12-9-pve

With this step i remove the RDMA functionality, i don't need this on this nic, and i prefer to resolve with firmware update.

Thank all
bye
Federico

 

[MarkusKo]

I think i saw another thread in this forum with the same issue

The solution was to either update the firmware for the broadcom nics, disable RDMA in the bios or disable RDMA with the broadcom nic utility or as you did, blacklist the kernel driver.