i am curious to see if this also applies to the iommu issues in conjunction with amd
Proxmox VE 8.2 released!
- Thread starter martin
- Start date
Hi there,
Server Experiencing Random Freezes After Proxmox Update to 8.2.2
System Specifications:
Following the recent update of Proxmox to version 8.2.2, the dedicated server has experienced two instances of complete system freezes. Prior to this update, no such freezes were observed. The freezes occurred while the server was idle and under minimal load, with most VMs powered off. The incidents happened during overnight hours when the administrator was asleep. Seems like a broader issue based on the postings here?
lspci output:
Server Experiencing Random Freezes After Proxmox Update to 8.2.2
System Specifications:
- CPU: Intel Core i9-9900K
- Motherboard: Gigabyte B360 HD3PLM
- Kernel: Linux 6.8.4-2-pve (compiled on 2024-04-10T17:36Z)
- Hosting: Hetzner Dedicated Server
- Virtualization: Licensed Proxmox with 12 VMs (approximately 3 running)
Following the recent update of Proxmox to version 8.2.2, the dedicated server has experienced two instances of complete system freezes. Prior to this update, no such freezes were observed. The freezes occurred while the server was idle and under minimal load, with most VMs powered off. The incidents happened during overnight hours when the administrator was asleep. Seems like a broader issue based on the postings here?
lspci output:
Code:
lspci
00:00.0 Host bridge: Intel Corporation 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0d)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0d)
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Device a308 (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
01:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe SSD [Hendrix]
02:00.0 Non-Volatile memory controller: Micron Technology Inc 3400 NVMe SSD [Hendrix]
Last edited:
NUC 13 i7.
2 days ago I updated to pve-manager 8.2.2 and kernel 6.8.4-2-pve.
everything except the Intel iGPU SR-IOV worked flawlessly. Yesterday I decided to pin the 6.5.13-5-pve kernel to have Intel iGPU SR-IOV support.
This configuration has been working without any problems since yesterday.
2 days ago I updated to pve-manager 8.2.2 and kernel 6.8.4-2-pve.
everything except the Intel iGPU SR-IOV worked flawlessly. Yesterday I decided to pin the 6.5.13-5-pve kernel to have Intel iGPU SR-IOV support.
This configuration has been working without any problems since yesterday.
I'm so glad to hear this. I'm slowly making my way towards testing this tomorrow. I've gotten so far as confirming the VFs show up in lspci.
I'm still confused about how the VRAM gets sliced up; it seems to be automatic, but I'm not sure what it's based on. I made separate thread about it here: https://forum.proxmox.com/threads/c...does-the-vram-get-divided-between-vfs.146003/
Just upgraded and everything worked fine (Xeon E3-1230 v3 from 2013). I did add network name overrides, as suggested, before doing the upgrade.
1. Get Mac address `watch ip link` for eno1 and eno2
2. Create links for the two network cards in my sys, with priority 10 and 20:
- avoid eno1 and eno2 naming scheme
```
nano /etc/systemd/network/10-enman0.link
nano /etc/systemd/network/20-envlan0.link
```
```
[Match]
MACAddress=xx:xx:xx:xx:xx:xx
Type=ether
[Link]
Name=envlan0
```
3. Update network config in `nano /etc/network/interfaces`
- replace eno1 with enman0
- replace eno2with envlan0
4. Reboot; check that network is working
5. Do the 8.2 Update
I did notice the following error in dmesg afterwards:
```
[Sun Apr 28 07:11:10 2024] i2c i2c-0: Failed to register i2c client jc42 at 0x18 (-16)
[Sun Apr 28 07:11:10 2024] at24 0-0050: 256 byte spd EEPROM, read-only
[Sun Apr 28 07:11:10 2024] at24 0-0051: supply vcc not found, using dummy regulator
[Sun Apr 28 07:11:10 2024] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
[Sun Apr 28 07:11:10 2024] i2c i2c-0: Failed to register i2c client jc42 at 0x19 (-16)
[Sun Apr 28 07:11:10 2024] at24 0-0051: 256 byte spd EEPROM, read-only
[Sun Apr 28 07:11:10 2024] at24 0-0052: supply vcc not found, using dummy regulator
[Sun Apr 28 07:11:10 2024] input: PC Speaker as /devices/platform/pcspkr/input/input5
[Sun Apr 28 07:11:10 2024] i2c i2c-0: Failed to register i2c client jc42 at 0x1a (-16)
[Sun Apr 28 07:11:10 2024] at24 0-0052: 256 byte spd EEPROM, read-only
[Sun Apr 28 07:11:10 2024] at24 0-0053: supply vcc not found, using dummy regulator
[Sun Apr 28 07:11:10 2024] fbcon: astdrmfb (fb0) is primary device
[Sun Apr 28 07:11:10 2024] i2c i2c-0: Failed to register i2c client jc42 at 0x1b (-16)
```
which looks like it relates to this. It looks like my four DRAM modules now have temperature sensors that want to become available.
> People running such systems would typically run sensors-detect to setup
their hardware monitoring, so the jc42 driver would be loaded at boot
by the lm-sensors service. This is "manual" from the kernel's
perspective, but still this is integrated and has been working for
years. If you break that, this is a functional regression.
(I am using lm-sensors)
But haven't noticed any consequences yet. `i2c-0` is the "SMBus I801 adapter at 0580"
```
sudo apt-get install i2c-tools
i2cdetect -l
```
```
> i2c-0 smbus SMBus I801 adapter at 0580 SMBus adapter
> i2c-1 i2c AST i2c bit bus I2C adapter
```
1. Get Mac address `watch ip link` for eno1 and eno2
2. Create links for the two network cards in my sys, with priority 10 and 20:
- avoid eno1 and eno2 naming scheme
```
nano /etc/systemd/network/10-enman0.link
nano /etc/systemd/network/20-envlan0.link
```
```
[Match]
MACAddress=xx:xx:xx:xx:xx:xx
Type=ether
[Link]
Name=envlan0
```
3. Update network config in `nano /etc/network/interfaces`
- replace eno1 with enman0
- replace eno2with envlan0
4. Reboot; check that network is working
5. Do the 8.2 Update
I did notice the following error in dmesg afterwards:
```
[Sun Apr 28 07:11:10 2024] i2c i2c-0: Failed to register i2c client jc42 at 0x18 (-16)
[Sun Apr 28 07:11:10 2024] at24 0-0050: 256 byte spd EEPROM, read-only
[Sun Apr 28 07:11:10 2024] at24 0-0051: supply vcc not found, using dummy regulator
[Sun Apr 28 07:11:10 2024] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
[Sun Apr 28 07:11:10 2024] i2c i2c-0: Failed to register i2c client jc42 at 0x19 (-16)
[Sun Apr 28 07:11:10 2024] at24 0-0051: 256 byte spd EEPROM, read-only
[Sun Apr 28 07:11:10 2024] at24 0-0052: supply vcc not found, using dummy regulator
[Sun Apr 28 07:11:10 2024] input: PC Speaker as /devices/platform/pcspkr/input/input5
[Sun Apr 28 07:11:10 2024] i2c i2c-0: Failed to register i2c client jc42 at 0x1a (-16)
[Sun Apr 28 07:11:10 2024] at24 0-0052: 256 byte spd EEPROM, read-only
[Sun Apr 28 07:11:10 2024] at24 0-0053: supply vcc not found, using dummy regulator
[Sun Apr 28 07:11:10 2024] fbcon: astdrmfb (fb0) is primary device
[Sun Apr 28 07:11:10 2024] i2c i2c-0: Failed to register i2c client jc42 at 0x1b (-16)
```
which looks like it relates to this. It looks like my four DRAM modules now have temperature sensors that want to become available.
> People running such systems would typically run sensors-detect to setup
their hardware monitoring, so the jc42 driver would be loaded at boot
by the lm-sensors service. This is "manual" from the kernel's
perspective, but still this is integrated and has been working for
years. If you break that, this is a functional regression.
(I am using lm-sensors)
But haven't noticed any consequences yet. `i2c-0` is the "SMBus I801 adapter at 0580"
```
sudo apt-get install i2c-tools
i2cdetect -l
```
```
> i2c-0 smbus SMBus I801 adapter at 0580 SMBus adapter
> i2c-1 i2c AST i2c bit bus I2C adapter
```
Last edited:
Indeed it does affect bridges and vlans also by default - hence the need for "Type=" matching. See the linked posts here
Hey, is it possible to add to the Backup Filter, tags?
With pools its a bit useless, because a vm can only be in one pool. But with tags, i can add a simple "b" for backup and the vm gets automatically backupped.
Thanks a lot
Edit:
And it would be amazing if there would be an option to Make sequential Backups and not Parallel.
When the Backup Tasks starts, it starts on all nodes at same time.
The issue is, the backup server has only one hdd-pool with deduplication/compression and so on, so parallel is very bad here, it slows down the backup speed to 50%.
I mean combined already 50%, one node backups at 200MB/s and the other around 30-80MB/s.
If just one is running, then i get 400-600MB/s sometimes even more.
With pools its a bit useless, because a vm can only be in one pool. But with tags, i can add a simple "b" for backup and the vm gets automatically backupped.
Thanks a lot
Edit:
And it would be amazing if there would be an option to Make sequential Backups and not Parallel.
When the Backup Tasks starts, it starts on all nodes at same time.
The issue is, the backup server has only one hdd-pool with deduplication/compression and so on, so parallel is very bad here, it slows down the backup speed to 50%.
I mean combined already 50%, one node backups at 200MB/s and the other around 30-80MB/s.
If just one is running, then i get 400-600MB/s sometimes even more.
Last edited:
Hey, is it possible to add to the Backup Filter, tags?
With pools its a bit useless, because a vm can only be in one pool. But with tags, i can add a simple "b" for backup and the vm gets automatically backupped.
Thanks a lot
For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=4186
Apr 25 14:53:05 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog
Apr 25 14:53:05 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0: Unable to read VPD
Apr 25 14:53:05 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.1 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog
Apr 25 14:53:05 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.1: Unable to read VPD
Apr 25 14:53:06 HVS-PMX-001P kernel: ? __pfx_bnxt_re_probe+0x10/0x10 [bnxt_re]
Apr 25 14:54:07 HVS-PMX-001P systemd-udevd[1506]: bnxt_en.rdma.2: Worker [1613] processing SEQNUM=23152 is taking a long time
Apr 25 14:54:07 HVS-PMX-001P systemd-udevd[1506]: bnxt_en.rdma.1: Worker [1538] processing SEQNUM=23256 is taking a long time
Apr 25 14:54:07 HVS-PMX-001P systemd-udevd[1506]: bnxt_en.rdma.0: Worker [1579] processing SEQNUM=23253 is taking a long time
Apr 25 14:54:07 HVS-PMX-001P systemd-udevd[1506]: bnxt_en.rdma.3: Worker [1588] processing SEQNUM=23155 is taking a long time
Apr 25 14:54:47 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0: QPLIB: bnxt_re_is_fw_stalled: FW STALL Detected. cmdq[0xe]=0x3 waited (101242 > 100000) msec active 1
Apr 25 14:54:47 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0 bnxt_re0: Failed to modify HW QP
Apr 25 14:54:47 HVS-PMX-001P kernel: infiniband bnxt_re0: Couldn't change QP1 state to INIT: -110
Apr 25 14:54:47 HVS-PMX-001P kernel: infiniband bnxt_re0: Couldn't start port
Apr 25 14:54:47 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0 bnxt_re0: Failed to destroy HW QP
Apr 25 14:54:47 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0 bnxt_re0: Free MW failed: 0xffffff92
Apr 25 14:54:47 HVS-PMX-001P kernel: infiniband bnxt_re0: Couldn't open port 1
Apr 25 14:54:47 HVS-PMX-001P kernel: infiniband bnxt_re0: Device registered with IB successfully
Apr 25 14:55:06 HVS-PMX-001P udevadm[1720]: Timed out for waiting the udev queue being empty.
Apr 25 14:55:06 HVS-PMX-001P udevadm[1717]: Timed out for waiting the udev queue being empty.
Apr 25 14:55:06 HVS-PMX-001P systemd[1]: ifupdown2-pre.service: Main process exited, code=exited, status=1/FAILURE
Apr 25 14:55:06 HVS-PMX-001P systemd[1]: ifupdown2-pre.service: Failed with result 'exit-code'.
Apr 25 14:55:06 HVS-PMX-001P systemd[1]: Failed to start ifupdown2-pre.service - Helper to synchronize boot up for ifupdown.
Apr 25 14:55:06 HVS-PMX-001P systemd[1]: Dependency failed for networking.service - Network initialization.
Apr 25 14:55:06 HVS-PMX-001P systemd[1]: networking.service: Job networking.service/start failed with result 'dependency'.
Apr 25 14:55:06 HVS-PMX-001P systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE
Apr 25 14:55:06 HVS-PMX-001P systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'.
Apr 25 14:55:06 HVS-PMX-001P systemd[1]: Failed to start systemd-udev-settle.service - Wait for udev To Complete Device Initialization.
Apr 25 14:56:07 HVS-PMX-001P systemd-udevd[1506]: bnxt_en.rdma.2: Worker [1613] processing SEQNUM=23152 killed
Apr 25 14:56:07 HVS-PMX-001P systemd-udevd[1506]: bnxt_en.rdma.1: Worker [1538] processing SEQNUM=23256 killed
Apr 25 14:56:07 HVS-PMX-001P systemd-udevd[1506]: bnxt_en.rdma.0: Worker [1579] processing SEQNUM=23253 killed
Apr 25 14:56:07 HVS-PMX-001P systemd-udevd[1506]: bnxt_en.rdma.3: Worker [1588] processing SEQNUM=23155 killed
Apr 25 14:56:29 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.1: QPLIB: bnxt_re_is_fw_stalled: FW STALL Detected. cmdq[0xe]=0x3 waited (102347 > 100000) msec active 1
Apr 25 14:56:29 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.1 bnxt_re1: Failed to modify HW QP
Apr 25 14:56:29 HVS-PMX-001P kernel: infiniband bnxt_re1: Couldn't change QP1 state to INIT: -110
Apr 25 14:56:29 HVS-PMX-001P kernel: infiniband bnxt_re1: Couldn't start port
Apr 25 14:56:29 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.1 bnxt_re1: Failed to destroy HW QP
Apr 25 14:56:29 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.1 bnxt_re1: Free MW failed: 0xffffff92
Apr 25 14:56:29 HVS-PMX-001P kernel: infiniband bnxt_re1: Couldn't open port 1
Apr 25 14:56:29 HVS-PMX-001P kernel: infiniband bnxt_re1: Device registered with IB successfully
Apr 25 14:56:29 HVS-PMX-001P kernel: infiniband (null): Low latency framework is enabled
Apr 25 14:56:29 HVS-PMX-001P kernel: infiniband bnxt_re2: Device registered with IB successfully
Apr 25 14:56:29 HVS-PMX-001P kernel: bnxt_en 0000:41:00.0: QPLIB: cmdq[0xb]=0x8c status 0x5
For those having problems with broadcom nics not going up automatically after upgrade to 8.2 see: https://forum.proxmox.com/threads/broadcom-nics-down-after-pve-8-2-kernel-6-8.146185/
(this also fixed having problems with a zfs pool on pbs 3.2 - maybe because udev did not work propely because of broadcom nic)
1. Update to latest Firmware from Broadcom
2. Blacklist Infiniband Driver
3. Reboot Network and ZFS-Pool backup again
3.1 ZFS failed because udev failed because of Broadcom infiniband driver not working
(this also fixed having problems with a zfs pool on pbs 3.2 - maybe because udev did not work propely because of broadcom nic)
1. Update to latest Firmware from Broadcom
2. Blacklist Infiniband Driver
3. Reboot Network and ZFS-Pool backup again
3.1 ZFS failed because udev failed because of Broadcom infiniband driver not working
Hi, after having successfully upgraded to 8.2.2, I noticed that memory usage displayed in the datacenter summary page has doubled compared to latest 8.1 release. Today I'm consuming around 70% of memory while I was around 35% with same LXC and QEMU running.
A friend of mine is having exactly the same behavior.
What happened?
A friend of mine is having exactly the same behavior.
What happened?
we updated 1 of 3 hosts ceph cluster and rebootet the host
The LAN network interface (eno1np0) now comes up automatically at boot
but the two 100Gbit interfaces for ceph cluster (enp65s0f0np0 & enp65s0f1np1) stay down
ifup enp65s0f0np0
or ifup bond0
or systemctl restart networking
does not bring them up
These got the firmware update
root@HVS-PMX-001P:~# ./bcm-nic-update.sh
--2024-04-30 18:44:22-- https://www.thomas-krenn.com/redx/tools/mb_download.php/ct.YuuHGw/mid.y9b3b4ba2bf7ab3b8/bnxtnvm.zip
Resolving www.thomas-krenn.com (www.thomas-krenn.com)... 185.65.88.120
Connecting to www.thomas-krenn.com (www.thomas-krenn.com)|185.65.88.120|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘bnxtnvm.zip’
bnxtnvm.zip [ <=> ] 1015K --.-KB/s in 0.1s
2024-04-30 18:44:23 (8.43 MB/s) - ‘bnxtnvm.zip’ saved [1039248]
Archive: bnxtnvm.zip
inflating: bnxtnvm
Broadcom NetXtreme-C/E/S firmware update and configuration utility version v222.0.144.0
NetXtreme-E Controller #1 at PCI Domain:0000 Bus:41 Dev:00
Firmware on NVM - v216.0.293.8
NetXtreme-E Controller #1 will be updated to firmware version v229.1.123.0
NetXtreme-C/E/S Controller #1 is being updated.......................................................
Firmware update is completed.
A system reboot is needed for firmware update to take effect (reboot not needed for the live patch).
Broadcom NetXtreme-C/E/S firmware update and configuration utility version v222.0.144.0
NetXtreme-E Controller #1 at PCI Domain:0000 Bus:c3 Dev:00
This adapter is not supported for online firmware update.
root@HVS-PMX-001P:~# echo "blacklist bnxt_re" >> /etc/modprobe.d/blacklist-bnxt_re.conf
root@HVS-PMX-001P:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-6.8.4-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/7B52-822F
Copying kernel and creating boot-entry for 6.5.13-5-pve
Copying kernel and creating boot-entry for 6.8.4-2-pve
Removing old version 6.5.13-1-pve
Copying and configuring kernels on /dev/disk/by-uuid/7B52-E306
Copying kernel and creating boot-entry for 6.5.13-5-pve
Copying kernel and creating boot-entry for 6.8.4-2-pve
Removing old version 6.5.13-1-pve
root@HVS-PMX-001P:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether 3c:ec:ef:e0:98:90 brd ff:ff:ff:ff:ff:ff
altname enp195s0f0np0
3: eno2np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 3c:ec:ef:e0:98:91 brd ff:ff:ff:ff:ff:ff
altname enp195s0f1np1
4: enp65s0f0np0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9000 qdisc mq master bond0 state DOWN mode DEFAULT group default qlen 1000
link/ether 84:16:0c:6f:41:40 brd ff:ff:ff:ff:ff:ff
5: enp65s0f1np1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9000 qdisc mq master bond0 state DOWN mode DEFAULT group default qlen 1000
link/ether 84:16:0c:6f:41:40 brd ff:ff:ff:ff:ff:ff permaddr 84:16:0c:6f:41:41
6: enxbe3af2b6059f: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether be:3a:f2:b6:05:9f brd ff:ff:ff:ff:ff:ff
11: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 9000 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 84:16:0c:6f:41:40 brd ff:ff:ff:ff:ff:ff
12: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 3c:ec:ef:e0:98:90 brd ff:ff:ff:ff:ff:ff
root@HVS-PMX-001P:~# lspci -nnk
41:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1750] (rev 11)
Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E Dual-port 100G QSFP56 Ethernet PCIe4.0 x16 Adapter (BCM957508-P2100G) [14e4:2100]
Kernel driver in use: bnxt_en
Kernel modules: bnxt_en
41:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1750] (rev 11)
Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E Dual-port 100G QSFP56 Ethernet PCIe4.0 x16 Adapter (BCM957508-P2100G) [14e4:2100]
Kernel driver in use: bnxt_en
Kernel modules: bnxt_en
c3:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller [14e4:16d8] (rev 01)
DeviceName: Broadcom Ethernet BCM57416/5720L #1
Subsystem: Super Micro Computer Inc BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller [15d9:16d8]
Kernel driver in use: bnxt_en
Kernel modules: bnxt_en
c3:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller [14e4:16d8] (rev 01)
DeviceName: Broadcom Ethernet BCM57416/5720L #2
Subsystem: Super Micro Computer Inc BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller [15d9:16d8]
Kernel driver in use: bnxt_en
Kernel modules: bnxt_en
Apr 30 19:08:52 HVS-PMX-001P kernel: bond0: (slave enp65s0f0np0): Enslaving as an active interface with a down link
Apr 30 19:08:52 HVS-PMX-001P kernel: bond0: (slave enp65s0f1np1): Enslaving as an active interface with a down link
Apr 30 19:08:53 HVS-PMX-001P kernel: vmbr0: port 1(eno1np0) entered blocking state
Apr 30 19:08:53 HVS-PMX-001P kernel: vmbr0: port 1(eno1np0) entered disabled state
Apr 30 19:08:53 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0 eno1np0: entered allmulticast mode
Apr 30 19:08:53 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0 eno1np0: NIC Link is Up, 1000 Mbps full duplex, Flow control: none
Apr 30 19:08:53 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0 eno1np0: EEE is not active
Apr 30 19:08:53 HVS-PMX-001P kernel: bnxt_en 0000:c3:00.0 eno1np0: FEC autoneg off encoding: None
Apr 30 19:08:53 HVS-PMX-001P kernel: vmbr0: port 1(eno1np0) entered blocking state
Apr 30 19:08:53 HVS-PMX-001P kernel: vmbr0: port 1(eno1np0) entered forwarding state
Last edited:
Thats the Problem:
This adapter is not supported for online firmware update.
Do you also have Mezzanine Broadcom Card in it?
Edit: did you reboot after Blacklisting?
This adapter is not supported for online firmware update.
Do you also have Mezzanine Broadcom Card in it?
Edit: did you reboot after Blacklisting?
Last edited:
yes rebootet after Blacklisting, also tried shutdown and start via bmc
we only have the BCM57508 as PCIe card and BCM57416 onboard
NetXtreme-E Controller #1 at PCI Domain:0000 Bus:41 Dev:00
Firmware on NVM - v216.0.293.8
NetXtreme-E Controller #1 will be updated to firmware version v229.1.123.0
These are the BCM57508 which are now in down state
ok, also BIOS says Disconnected
I will try to disconnect and reconnect cables in half an hour (not at work at the moment)
#
reconnected cables on the updated first node -> no link
reinserted qsfp modules on the first node -> no link
reinserted qsfp modules on the other two nodes -> link ok, ceph rebuild ok
I also tried ifdown/ifup on the other nodes before and played with port speeds in NIC BIOS
but only physical reinstall of the modules made it work again
then did tutorial on second node
connection to third node was down after reboot
but connection to updated first node was up
reinserted qsfp module on third node, link ok, ceph rebuild ok
did tutorial on third node
reboot
all interfaces up (LAN interface and both 100G ceph interfaces)
no need to reinsert module
ceph rebuild ok
initial problem solved
thanks for providing the workaround
Hi,
After upgrade to 8.2 I noticed, that all VM didn't start because of error:
I did some experiments on disk settings and it looks that the reason is combination "cache =writeback" and "aio=io_uring" ("native" too). With "cache=writeback" I'm able to start VMs only when "aio=threads". Without cache I can run VMs regardless of aio type (except Windows type but I couldnt do it in the past too), but for example Linux opens only with it's GUI in Proxmox's console when "aio=threads" (other shows only Linux terminal prompt). When I migrate such VM to node having old Proxmox 8.1 it still starts fine.
What may be the reason of such behavior on latest Proxmox? IMO it's connected with changes in Qemu but I cannot find it in Proxmox 8.2 release notes or Qemu 8.1.5 removed features. I'm affraid to continue upgrading other nodes in cluster. Please, help.
After upgrade to 8.2 I noticed, that all VM didn't start because of error:
I did some experiments on disk settings and it looks that the reason is combination "cache =writeback" and "aio=io_uring" ("native" too). With "cache=writeback" I'm able to start VMs only when "aio=threads". Without cache I can run VMs regardless of aio type (except Windows type but I couldnt do it in the past too), but for example Linux opens only with it's GUI in Proxmox's console when "aio=threads" (other shows only Linux terminal prompt). When I migrate such VM to node having old Proxmox 8.1 it still starts fine.
What may be the reason of such behavior on latest Proxmox? IMO it's connected with changes in Qemu but I cannot find it in Proxmox 8.2 release notes or Qemu 8.1.5 removed features. I'm affraid to continue upgrading other nodes in cluster. Please, help.
Hi,
Can you check if booting into a 6.5 kernel helps or if downgrading
what kind of filesystem is mounted at
/mnt/lun2? It might be the same issue as reported here for OCFS2: https://bugzilla.proxmox.com/show_bug.cgi?id=5430Can you check if booting into a 6.5 kernel helps or if downgrading
pve-qemu-kvm helps?Hi fiona,
Bingo. It's OCSF2. It's exactly the same problem. Thank you very much for help!
I pinned old 6.5 kernel as you suggested and it helped. Now I'm thinking what to do with other nodes:
- leave it as it is and wait for new fixed kernel
- upgrade them too to Proxmox 8.2 with old kernel pinned