Update to PVE 8 Udev issue

[LeeDraper]

Hello all,

I just updated one of our proxmox servers to PVE 8 and now network fails to work on boot.

The server has a Broadcom BCM57414 using the bnxt_en driver, and systemd-udevd is struggling with it. Which causes ifupdown2-pre.service to fail because it depends on systemd-udev-settle.service. See syslog below.

Once the server has booted I can login via the console and restart the following service to get networking run:

systemd-udev-settle.service
ifupdown2-pre.service
networking.service

I've booted a Debian live iso and the same problem does not happen, so I don't think it's a hardware issue.

I have also update a another identical (ish) server without issue.

I'm not sure what to try next and any suggestions would be greatly appreciated.

Thank you
Lee Draper

2024-06-04T11:05:26.266353+01:00 PROX-C systemd[1]: systemd-fsckd.service: Deactivated successfully.
2024-06-04T11:05:26.266357+01:00 PROX-C systemd-udevd[1032]: bnxt_en.rdma.0: Worker [1175] processing SEQNUM=8122 is taking a long time
2024-06-04T11:05:26.266361+01:00 PROX-C systemd-udevd[1032]: bnxt_en.rdma.1: Worker [1067] processing SEQNUM=8124 is taking a long time
2024-06-04T11:05:26.266368+01:00 PROX-C kernel: [    0.007930] ACPI: Reserving SRAT table memory at [mem 0x6ffe9000-0x6ffebd2f]
2024-06-04T11:05:26.266369+01:00 PROX-C kernel: [    0.007931] ACPI: Reserving SVOS table memory at [mem 0x6ffe8000-0x6ffe8031]
2024-06-04T11:05:26.266365+01:00 PROX-C udevadm[1057]: Timed out for waiting the udev queue being empty.
2024-06-04T11:05:26.266370+01:00 PROX-C kernel: [    0.007932] ACPI: Reserving WSMT table memory at [mem 0x6ffe7000-0x6ffe7027]
2024-06-04T11:05:26.266374+01:00 PROX-C kernel: [    0.007933] ACPI: Reserving OEM4 table memory at [mem 0x6fc4a000-0x6fcf71c0]
2024-06-04T11:05:26.266375+01:00 PROX-C kernel: [    0.007934] ACPI: Reserving SSDT table memory at [mem 0x6fc12000-0x6fc49464]
2024-06-04T11:05:26.266375+01:00 PROX-C udevadm[1056]: Timed out for waiting the udev queue being empty.
2024-06-04T11:05:26.266378+01:00 PROX-C kernel: [    0.007935] ACPI: Reserving SSDT table memory at [mem 0x6fffd000-0x6fffda1e]
2024-06-04T11:05:26.266379+01:00 PROX-C kernel: [    0.007936] ACPI: Reserving SSDT table memory at [mem 0x6fc0d000-0x6fc1057e]
2024-06-04T11:05:26.266380+01:00 PROX-C kernel: [    0.007937] ACPI: Reserving DMAR table memory at [mem 0x6fc0c000-0x6fc0c187]
2024-06-04T11:05:26.266381+01:00 PROX-C kernel: [    0.007937] ACPI: Reserving HEST table memory at [mem 0x6fc0b000-0x6fc0b17b]
2024-06-04T11:05:26.266382+01:00 PROX-C kernel: [    0.007938] ACPI: Reserving BERT table memory at [mem 0x6fc0a000-0x6fc0a02f]
2024-06-04T11:05:26.266384+01:00 PROX-C kernel: [    0.007940] ACPI: Reserving ERST table memory at [mem 0x6fc09000-0x6fc0922f]
2024-06-04T11:05:26.266381+01:00 PROX-C systemd[1]: ifupdown2-pre.service: Main process exited, code=exited, status=1/FAILURE
2024-06-04T11:05:26.266385+01:00 PROX-C kernel: [    0.007940] ACPI: Reserving EINJ table memory at [mem 0x6fc08000-0x6fc0814f]
2024-06-04T11:05:26.266390+01:00 PROX-C kernel: [    0.007998] APIC: Switched APIC routing to: cluster x2apic
2024-06-04T11:05:26.266390+01:00 PROX-C systemd[1]: ifupdown2-pre.service: Failed with result 'exit-code'.
2024-06-04T11:05:26.266391+01:00 PROX-C kernel: [    0.008029] SRAT: PXM 0 -> APIC 0x0000 -> Node 0
2024-06-04T11:05:26.266395+01:00 PROX-C kernel: [    0.008031] SRAT: PXM 1 -> APIC 0x0040 -> Node 1
2024-06-04T11:05:26.266396+01:00 PROX-C kernel: [    0.008032] SRAT: PXM 0 -> APIC 0x0008 -> Node 0
2024-06-04T11:05:26.266397+01:00 PROX-C kernel: [    0.008032] SRAT: PXM 1 -> APIC 0x0048 -> Node 1
2024-06-04T11:05:26.266398+01:00 PROX-C kernel: [    0.008033] SRAT: PXM 0 -> APIC 0x0002 -> Node 0
2024-06-04T11:05:26.266396+01:00 PROX-C systemd[1]: Failed to start ifupdown2-pre.service - Helper to synchronize boot up for ifupdown.
2024-06-04T11:05:26.266401+01:00 PROX-C kernel: [    0.008034] SRAT: PXM 1 -> APIC 0x0042 -> Node 1
2024-06-04T11:05:26.266402+01:00 PROX-C kernel: [    0.008035] SRAT: PXM 0 -> APIC 0x0006 -> Node 0
2024-06-04T11:05:26.266403+01:00 PROX-C kernel: [    0.008035] SRAT: PXM 1 -> APIC 0x0046 -> Node 1
2024-06-04T11:05:26.266402+01:00 PROX-C systemd[1]: Dependency failed for networking.service - Network initialization.
2024-06-04T11:05:26.266405+01:00 PROX-C kernel: [    0.008036] SRAT: PXM 0 -> APIC 0x0004 -> Node 0
2024-06-04T11:05:26.266406+01:00 PROX-C kernel: [    0.008037] SRAT: PXM 1 -> APIC 0x0044 -> Node 1
2024-06-04T11:05:26.266407+01:00 PROX-C kernel: [    0.008038] SRAT: PXM 0 -> APIC 0x0016 -> Node 0
2024-06-04T11:05:26.266410+01:00 PROX-C kernel: [    0.008038] SRAT: PXM 1 -> APIC 0x0056 -> Node 1
2024-06-04T11:05:26.266411+01:00 PROX-C kernel: [    0.008039] SRAT: PXM 0 -> APIC 0x0010 -> Node 0
2024-06-04T11:05:26.266412+01:00 PROX-C kernel: [    0.008040] SRAT: PXM 1 -> APIC 0x0050 -> Node 1
2024-06-04T11:05:26.266407+01:00 PROX-C systemd[1]: networking.service: Job networking.service/start failed with result 'dependency'.BCM57414BCM57414
2024-06-04T11:05:26.266412+01:00 PROX-C kernel: [    0.008040] SRAT: PXM 0 -> APIC 0x0014 -> Node 0
2024-06-04T11:05:26.266416+01:00 PROX-C kernel: [    0.008041] SRAT: PXM 1 -> APIC 0x0054 -> Node 1
2024-06-04T11:05:26.266417+01:00 PROX-C kernel: [    0.008042] SRAT: PXM 0 -> APIC 0x0012 -> Node 0

 

[LeeDraper]

I fixed this issue by updating the firmware for the BCM57414.

 

[hzk916]

Hi,
I have same issue in a Lenovo SR850 with both a Broadcom 57414 10/25GbE SFP28 2-port and Broadcom NX-E PCIe 10Gb 2-Port Base-T. I have just flashed the firmware to the latest 227ver, but still no luck in connecting. Proxmox is latest 8.2.2 all updates applied.

 

[LeeDraper]

Hello,

If it helps, I updated via the Dell Lifecycle, which shows the firmware version as 22.71.11.13

 

[gfngfn256]

Hi,
I have same issue in a Lenovo SR850 with both a Broadcom 57414 10/25GbE SFP28 2-port and Broadcom NX-E PCIe 10Gb 2-Port Base-T. I have just flashed the firmware to the latest 227ver, but still no luck in connecting. Proxmox is latest 8.2.2 all updates applied.

Maybe take a look at this thread. From what I glanced there you would need to also blacklist the Broadcom infiniband-driver.

 

[hzk916]

Maybe take a look at this thread. From what I glanced there you would need to also blacklist the Broadcom infiniband-driver.

OK addded the blacklist. It has connected but extreamly slow, only get 1 ping in 4. I did a reboot after blacklist.

 

[gfngfn256]

It has connected but extreamly slow, only get 1 ping in 4

I can't directly help or relate as I don't have a similar HW/NW to test on.
However look at that thread - and maybe attempt to update the driver/firmware as the first post indicates. Read the whole thread. Maybe post there with your problem - the posters there will be in a more similar situation to yourself.