Proxmox 8.3.2 and Broadcom iSER connections

[starnetwork]

Hi,
we using Proxmox updated version: 8.3.2 (we did now upgrade from 7.4.x) in env with Broadcom BCM57504 NetXtreme-E 10Gb/25Gb on Dell Server
we using remote iscsi over iser storage and pbs for backups, the problem is that each time we have backup for disks > 100Gb it's stop, Hangs the server and shows the following error: bnxt_en Failed to create HW QP
(attached as screenshot)

we have seen the Threads

Thread 'Opt-in Linux 6.8 Kernel for Proxmox VE 8 available on test & no-subscription'

Apr 5, 2024

We recently uploaded a 6.8 kernel into our repositories, it will be used as new default kernel in the next Proxmox VE 8.2 point release (Q2'2024).
This follows our tradition of upgrading the Proxmox VE kernel to match the current Ubuntu version until we reach an (Ubuntu) LTS release. This kernel is based on the upcoming Ubuntu 24.04 Noble release.

We have run this kernel on some parts of our test setups over the last few days without any notable issues.

How to install:

1. Ensure that either the pve-no-subscription or pvetest repository is set up correctly.
   You can...

• t.lamprecht
• kernel 6.8
• Replies: 221
• Forum: Proxmox VE: Installation and configuration

• 

and

[TUTORIAL] Thread 'Broadcom NICs down after PVE 8.2 (Kernel 6.8)'

Apr 30, 2024

We had some issues with some broadcom nics going down after update to 6.8

Workaround: NICs go up if you do a service networking restart
FIX: Update Broadcom Firmware to latest firmware and blacklist their "beautiful" infiniband-driver

This will update ALL YOUR Broadcom-Network Cards to their latest firmware (live) (but reboot needed after it):

apt install unzip
cat << 'EOF' > bcm-nic-update.sh
wget https://www.thomas-krenn.com/redx/tools/mb_download.php/ct.YuuHGw/mid.y9b3b4ba2bf7ab3b8/bnxtnvm.zip
unzip bnxtnvm.zip
chmod +x bnxtnvm
for i in $(./bnxtnvm listdev...

• jsterr
• broadcom kernel 6.8 network p225g p425g pve 8.2
• Replies: 60
• Forum: Proxmox VE: Installation and configuration

• 

but in our case we don't want to disable the iSER/RDMA support since it's in use, we did firmware update to the NIC, now it's v231.1.162.1, didn't help any advice?

# pveversion
pve-manager/8.3.2/3e76eec21c4a14a7 (running kernel: 6.8.12-5-pve)

root@testnode1:~# ethtool -i ens1f0np0
driver: bnxt_en
version: 6.8.12-5-pve
firmware-version: 231.0.154.0/pkg 231.1.162.1
expansion-rom-version:
bus-info: 0000:5e:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
root@testnode1:~#

Kind Regards,

 

[dakralex]

Hi!

Is there anything printed to the log before these errors occur? As the other error message states, the driver seems to be unable to allocate resources for queueing the data transfer, but I'm not entirely sure what the error code 110 is for. It is normally used for ETIMEDOUT, but I couldn't verify whether this is the actual error returned here.

 

[starnetwork]

Dear @dakralex
thanks for your response!
attached additional error messages:

2025-01-14T08:47:05.359511+01:00 node404 kernel: [ 7657.161991] DMAR: DRHD: handling fault status reg 2
2025-01-14T08:47:05.359523+01:00 node404 kernel: [ 7657.162495] DMAR: [DMA Read NO_PASID] Request device [5e:00.3] fault addr 0xe3d7d000 [fault reason 0x06] PTE Read access is not set
2025-01-14T08:47:07.602146+01:00 node404 corosync[4529]:   [KNET  ] link: host: 3 link: 1 is down
2025-01-14T08:47:07.602509+01:00 node404 corosync[4529]:   [KNET  ] link: host: 6 link: 1 is down
2025-01-14T08:47:07.602556+01:00 node404 corosync[4529]:   [KNET  ] link: host: 4 link: 1 is down
2025-01-14T08:47:07.602602+01:00 node404 corosync[4529]:   [KNET  ] link: host: 2 link: 1 is down
2025-01-14T08:47:07.602645+01:00 node404 corosync[4529]:   [KNET  ] link: host: 1 link: 1 is down
2025-01-14T08:47:07.602710+01:00 node404 corosync[4529]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
2025-01-14T08:47:07.602754+01:00 node404 corosync[4529]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
2025-01-14T08:47:07.602797+01:00 node404 corosync[4529]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
2025-01-14T08:47:07.602841+01:00 node404 corosync[4529]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
2025-01-14T08:47:07.602886+01:00 node404 corosync[4529]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
2025-01-14T08:47:15.347469+01:00 node404 kernel: [ 7667.149858]  connection1:0: detected conn error (1011)
2025-01-14T08:47:15.518430+01:00 node404 kernel: [ 7667.320891]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4302322256, last ping 4302327296, now 4302332416
2025-01-14T08:47:15.518438+01:00 node404 kernel: [ 7667.321871]  connection2:0: detected conn error (1022)
2025-01-14T08:47:15.518840+01:00 node404 kernel: [ 7667.322000] iser: iser_qp_event_callback: qp event QP fatal error (1)
2025-01-14T08:47:16.032699+01:00 node404 iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (3)
2025-01-14T08:47:16.032836+01:00 node404 iscsid: Kernel reported iSCSI connection 2:0 error (1022 - ISCSI_ERR_NOP_TIMEDOUT: A NOP has timed out) state (3)
2025-01-14T08:47:17.566471+01:00 node404 kernel: [ 7669.368820] bnxt_en 0000:5e:00.3 ens2f3np3: NETDEV WATCHDOG: CPU: 2: transmit queue 17 timed out 10752 ms

Please let me know if you need anything else to check this issue

 

[dakralex]

Okay, so it probably was ETIMEDOUT after all. Could you try to increase the timeout interval for No-Op requests? This should be node.conn[0].timeo.noop_out_timeout in the /etc/iscsi/iscsid.conf, depending on your configuration, e.g. to 15 seconds. Does the problem persist?

 

[starnetwork]

thanks, I did for 20 seconds and check new backup, same issue
suggestions?

Kind Regards,