Broadcom P210P - 2 x 10GbE PCIe NIC causing insanity for newb

[Alex White 101]

Hi

I'm trying to install the drivers for the above NIC.

I've been at this for a while (weeks) and have succumb to seeking help.
I can't use the Linux_Installer for it because the installer doesn't supports pve.

https://techdocs.broadcom.com/us/en...-software-installation-and-configuration.html

I've managed to install the headers for both 5.15.107-2-pve & 6.2.11-2-pve having tried both kernels and deploy the driver manually.

dpkg -i pve-headers-5.15.102-1-pve_5.15.102-1_amd64.deb
tar xvf bcm_225.1.95.0.tar.gz
export REL_DIR=/root/bcm_225.1.95.0
apt install -y make automake autoconf libtool libibverbs-dev ibverbs-utils infiniband-diags perftest ethtool
cd $REL_DIR/Linux/Linux_Driver
tar xvf netxtreme-bnxt_en-1.10.2-225.0.143.0.tar.gz
cd netxtreme-bnxt_en-1.10.2-225.0.143.0/
make
make install
depmod -a
update-initramfs -u

pveversion
pve-manager/7.4-3/9002ab8a (running kernel: 6.2.11-2-pve)

bnxtnvm version
Broadcom NetXtreme-C/E/S firmware update and configuration utility
Version v225.0.130.0
The card seems to function for about five minutes before produces the above error.
I'm sure I'm missing something if anyone could help.

bnxt_en 0000:02:00.0 enp2s0f0np0: Abandoning msg {0x23 0xb49} len: 0 due to firmware status: 0xffffffff

lspci
00:00.0 Host bridge: Intel Corporation Device a700 (rev 01)
00:01.0 PCI bridge: Intel Corporation Device a70d (rev 01)
00:01.1 PCI bridge: Intel Corporation Device a72d (rev 01)
00:06.0 PCI bridge: Intel Corporation Device a74d (rev 01)
00:0a.0 Signal processing controller: Intel Corporation Device a77d (rev 01)
00:0e.0 RAID bus controller: Intel Corporation Device a77f
00:14.0 USB controller: Intel Corporation Device 7a60 (rev 11)
00:14.2 RAM memory: Intel Corporation Device 7a27 (rev 11)
00:15.0 Serial bus controller [0c80]: Intel Corporation Device 7a4c (rev 11)
00:15.1 Serial bus controller [0c80]: Intel Corporation Device 7a4d (rev 11)
00:15.2 Serial bus controller [0c80]: Intel Corporation Device 7a4e (rev 11)
00:16.0 Communication controller: Intel Corporation Device 7a68 (rev 11)
00:17.0 SATA controller: Intel Corporation Device 7a62 (rev 11)
00:1a.0 PCI bridge: Intel Corporation Device 7a48 (rev 11)
00:1b.0 PCI bridge: Intel Corporation Device 7a40 (rev 11)
00:1c.0 PCI bridge: Intel Corporation Device 7a38 (rev 11)
00:1c.2 PCI bridge: Intel Corporation Device 7a3a (rev 11)
00:1c.3 PCI bridge: Intel Corporation Device 7a3b (rev 11)
00:1c.4 PCI bridge: Intel Corporation Device 7a3c (rev 11)
00:1d.0 PCI bridge: Intel Corporation Device 7a30 (rev 11)
00:1f.0 ISA bridge: Intel Corporation Device 7a04 (rev 11)
00:1f.4 SMBus: Intel Corporation Device 7a23 (rev 11)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Device 7a24 (rev 11)
01:00.0 VGA compatible controller: NVIDIA Corporation Device 2216 (rev a1)
01:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)
03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO
06:00.0 Ethernet controller: Aquantia Corp. Device 94c0 (rev 03)
07:00.0 Network controller: Intel Corporation Device 2725 (rev 1a)
08:00.0 Ethernet controller: Intel Corporation Device 125c (rev 06)
09:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0a:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0a:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0a:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0a:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02)
0b:00.0 USB controller: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020]
3f:00.0 USB controller: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020]
73:00.0 Multimedia video controller: Blackmagic Design Intensity Pro 4K

bnxtnvm list
Device #1
---------
Device: enp2s0f0np0
item type ord.ext data/length attr version
1 VPD 0.0 404/4096 0000
2 update 0.0 1241528/1572864 0000
3 chimpBoot 0.0 347708/348160 0000 787.0.0.0
4 PHYucode 0.0 33084/36864 0010 D10B_23
5 QoSProfiles 0.0 3484/4096 0000
6 CrashDmpData 0.0 524288/524288 0001
7 PCIEucode 0.0 17508/20480 0010 D10F_15
8 MBA 0.0 355616/356352 0010 225.0.144.0
9 CrashDmpFW 0.0 11948/12288 0000 225.0.132.0
10 chimpPatch 1.0 7716/8192 0000 225.0.132.0
11 iSCSIboot 0.0 63952/69632 0010 214.0.152.0
12 CrashDmpData 1.0 524288/524288 0001
13 apeFW 0.0 229796/233472 0000 225.0.132.0
14 CCM 0.0 62352/65536 0010 216.0.52.1
15 bonoFW 0.0 73140/73728 0000 225.0.132.0
16 pkgLog 0.0 944/4096 0000 225.1.95.0
17 sharedCfg 0.0 800/4096 0000
18 portCfg 0.0 800/4096 0000
19 funcCfg 0.0 800/4096 0000
20 portCfg 1.0 800/4096 0000
21 funcCfg 1.0 800/4096 0000
22 funcCfg 2.0 800/4096 0000
23 funcCfg 3.0 800/4096 0000
24 funcCfg 4.0 800/4096 0000
25 funcCfg 5.0 800/4096 0000
26 funcCfg 6.0 800/4096 0000
27 funcCfg 7.0 800/4096 0000
28 funcCfg 8.0 800/4096 0000
29 funcCfg 9.0 800/4096 0000
30 funcCfg 10.0 800/4096 0000
31 funcCfg 11.0 800/4096 0000
32 funcCfg 12.0 800/4096 0000
33 funcCfg 13.0 800/4096 0000
34 funcCfg 14.0 800/4096 0000
35 funcCfg 15.0 800/4096 0000
36 systemCfg 0.0 0/20480 0000
37 cfgJournal 0.0 4096/4096 0001
38 kongFW 0.0 136004/139264 0000 225.0.132.0
39 cfgTable 0.0 6112/8192 0000

device_info
Device Interface Name : enp2s0f1np1
MACAddress : 00:62:0b:fa:92:91
Base MACAddress : 00:62:0B:FA:92:90
Device Serial Number : A412022460022EFG
Chip Number : BCM57412
Part Number : BCM957412A4120AC
Description : Broadcom P210p NetXtreme-E Dual-port 10Gb Ethernet PCIe Adapter
PCI Vendor Id : 14e4
PCI Device Id : 16d6
PCI Subsys Vendor Id : 14e4
PCI Subsys Device Id : 4124
PCI Device Name : 0000:02:00.1
Adapter Rev : 01
Active Package version : 225.1.95.0
Package version on NVM : 225.1.95.0
Firmware version : 225.0.132.0
Active NVM config version : 216.0.1
NVM config version : 216.0.1
Firmware Reset Counter : 0
Error Recovery Counter : 0
Crash Dump Timestamp : Sat 2023-05-20 19:41:53 BST
Reboot Required : No
Secure Boot : Enabled
Secure Firmware Update : Enabled

Any help would be so gratefully appreciated
Any commands to help please let me know.
If I'm my time trying to use these cards can someone advice me the right card to get.

Many Thanks

Alex

 

[Alex White 101]

Just to add, the errors only appear once I start a VM.

qm config 100
agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0;ide0
cores: 8
cpu: host
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:73:00,pcie=1
hostpci1: 0000:01:00,pcie=1,x-vga=1
ide0: Synology-Nas:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
ide2: Synology-Nas:iso/Win11_22H2_EnglishInternational_x64v1.iso,media=cdrom,size=5418024K
machine: pc-q35-7.2
memory: 51200
meta: creation-qemu=7.2.0,ctime=1684519130
name: vMix1
net0: e1000=56:47:2BA:09:8B,bridge=vmbr0,firewall=1
net1: virtio=BA:97:8E:4F:AF:0B,bridge=vmbr1,firewall=1
net2: virtio=06:88:1D:25:CB:4D,bridge=vmbr2,firewall=1
numa: 0
ostype: win11
scsi0: local-lvm:vm-100-disk-1,cache=writeback,discard=on,iothread=1,size=100G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=1f460ace-07b0-4c22-bd9a-2713bac48558
sockets: 1
tpmstate0: local-lvm:vm-100-disk-2,size=4M,version=v2.0
vmgenid: 15a68bac-afc4-43e6-8971-a267ef825f09

 

[Alex White 101]

Having no luck with this as a bridged network. I'm retrying passing the card through PCI passthrough.
Are there any passthrough configurations to set for NIC's.

I've so far passed the card and deployed the drivers for it on the guest OS.
However in the event viewer I get:

Source: bnxtnd
Broadcom P210p NetXtreme-E Dual-port 10Gb Ethernet PCIe Adapter #2: Firmware failed to respond within allocated time.

 

[Alex White 101]

Progress, maybe.

Firstly the driver being used in pve is not the right driver.
modinfo bnxt_en
filename: /lib/modules/5.15.102-1-pve/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko
description: Broadcom BCM573xx network driver

My card is the BCM57412

I've blacklisted the driver in vfio.conf
Blacklist bnxt_en

lspci -v
Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev ff) (prog-if ff)
Kernel driver in use: vfio-pci
Kernel modules: bnxt_en

 

[Alex White 101]

Having gone all round the houses, it look like this problem only happens with the windows 11 vm.
My test was simple
Ping the nic and after a short period of time it would fail on the host and require a power cycle.
bnxt_en 0000:02:00.0 enp2s0f0np0: Error (timeout: 15)

I created a new VM with debian and create a static IP on the same bridged network from the host and it has been happily working for over 30 minutes.

It can only be the virto drivers on Win 11 or something with Win 11 itself

 

[Alex White 101]

Installed Win10 and no problems with card or win10 virtio drivers.

 

[SheridansNL]

I have a X570D4U-2L2T Bcm motherboard and having issues with the 10G Nics (BCM57416)

I get the following error :
[1623.750006] bnxt_en 0000:24:00.0 enp36s0f1np1 abandoning message {0x23 0x12b2} len:0 die to firmware status: 0xffffffff

I also have a win11 vm with gpu passthrough, but its not running and stil I get this issue.

Have you figured out what caused this issue and how did you fix it?

 

[Alex White 101]

Hi, I never figured it out and eventually bought 2 new NIC's. I drove me insane and went round in circles trying to figure it out.
Just as I thought I was getting somewhere I still ended up with the same issues.
I'm sorry I found very little help to help me solve the problem.
In the end I bought 2 Intel E10G42B X520-DA2 and they just plugged and played nicely.

Good luck with what you try, you may be more knowledgeable than me and get it figured.

 

[melroy89]

So some people say they have stability issues with Intel like the x520 serie. And now you are saying you have issues with broadcom and go to Intel. WHAT should I buy now when I have dual 10G SFP+?