Opt-in Linux 6.8 Kernel for Proxmox VE 8 available on test & no-subscription

jaminmc said:
ntfs-3g
Click to expand...
Have you tried NTFS3 which is a kernel implementation (developed by Paragon) & much newer as opposed to the older NTFS-3G which is a FUSE layer.

I know that NTFS3 had its teething problems when introduced - but I imagine it must have matured by now, although it may still have issues, IDK.

Disclosure: I don't personally use NTFS at all for interoperability between Windows & Linux, I use exFAT for that exclusively & have had zero issues.
 
  • Like
Reactions: jaminmc and spirit
Stoiko Ivanov said:
from a quick search (don't have this card lying around) - there might be a new firmware available (based on the version printed in the lshw output you shared above...
https://www.broadcom.com/products/ethernet-connectivity/network-adapters/bcm57416-1gbase-t-ic
https://www.broadcom.com/products/ethernet-connectivity/network-adapters/n210tp

maybe updating the firmware to the latest version provided by the manufacturer could help here
Click to expand...

after the update to 6.8 the bnxt_en driver does also not work for me.

In this case it is the onboard ports of a SuperMicro H13SSL-NT mainboard and there seems to be no firmware update for them

Code: 
Apr 09 19:55:10 gcd-virthost3 kernel: bnxt_en 0000:c1:00.1: QPLIB: bnxt_re_is_fw_stalled: FW STALL Detected. cmdq[0xe]=0x3 waited (102364 > 100000) msec active 1
Apr 09 19:55:10 gcd-virthost3 kernel: bnxt_en 0000:c1:00.1 bnxt_re1: Failed to modify HW QP
Apr 09 19:55:10 gcd-virthost3 kernel: infiniband bnxt_re1: Couldn't change QP1 state to INIT: -110
Apr 09 19:55:10 gcd-virthost3 kernel: infiniband bnxt_re1: Couldn't start port
Apr 09 19:55:10 gcd-virthost3 kernel: bnxt_en 0000:c1:00.1 bnxt_re1: Failed to destroy HW QP
Apr 09 19:55:10 gcd-virthost3 kernel: bnxt_en 0000:c1:00.1 bnxt_re1: Free MW failed: 0xffffff92
Apr 09 19:55:10 gcd-virthost3 kernel: infiniband bnxt_re1: Couldn't open port 1
Apr 09 19:55:10 gcd-virthost3 kernel: infiniband bnxt_re1: Device registered with IB successfully
 
Ramalama said:
I think tbh, that this change isn't that bad. Sure it nerves but it takes 2 seconds to fix /etc/network/interfaces over bmc.
However, the reason why i think that it isn't that bad, if im not wrong, you probably see now directly that your NIC supports some sort of SR-IOV.
Click to expand...
Is there any way to check - while still on the old kernel - whether this renaming will affect oneself when rebooting with the 6.8 kernel, so that one could take the appropriate measures in the `/etc/network/interfaces` blindly before rebooting?

This renaming may not be a big issue for a machine where you have direct access to, but for everything administered remotely - or even your virtual firewall host - this can be quite annoying.
 
  • Like
Reactions: thiagotgc
Daniel Keller said:
after the update to 6.8 the bnxt_en driver does also not work for me.

In this case it is the onboard ports of a SuperMicro H13SSL-NT mainboard and there seems to be no firmware update for them
Click to expand...
the latest bios for those boards is from March (maybe it contains the firmware for the NICs as well)?
https://www.supermicro.com/en/support/resources/downloadcenter/firmware/MBD-H13SSL-NT/BIOS

apart from that - could you share your network-config and setup in general - maybe we can reproduce the issue here as well
why do you use infiniband? (maybe that's the culprit)
 
Stoiko Ivanov said:
the latest bios for those boards is from March (maybe it contains the firmware for the NICs as well)?
https://www.supermicro.com/en/support/resources/downloadcenter/firmware/MBD-H13SSL-NT/BIOS

apart from that - could you share your network-config and setup in general - maybe we can reproduce the issue here as well
why do you use infiniband? (maybe that's the culprit)
Click to expand...
yes I might try the bios update.

the network configuration is just a simple bond over both interfaces

Code: 
auto enp193s0f0np0
iface enp193s0f0np0 inet manual

auto enp193s0f1np1
iface enp193s0f1np1 inet manual

auto bond0
iface bond0 inet static
        address --------
        gateway --------
        bond-slaves enp193s0f0np0 enp193s0f1np1
        bond-miimon 100
        bond-mode active-backup
        mtu 9000

we do not use infiniband.

but also kernel 6.5 shows some messages with infiniband

Code: 
[   11.828468] infiniband bnxt_re0: Device registered with IB successfully
[   11.870741] infiniband bnxt_re1: Device registered with IB successfully
[   14.623305] bnxt_en 0000:c1:00.1 enp193s0f1np1: NIC Link is Up, 10000 Mbps full duplex, Flow control: none
[   14.623311] bnxt_en 0000:c1:00.1 enp193s0f1np1: EEE is not active
[   14.623312] bnxt_en 0000:c1:00.1 enp193s0f1np1: FEC autoneg off encoding: None
[   14.987787] bnxt_en 0000:c1:00.0 enp193s0f0np0: NIC Link is Up, 1000 Mbps full duplex, Flow control: none
[   14.987793] bnxt_en 0000:c1:00.0 enp193s0f0np0: EEE is not active
[   14.987794] bnxt_en 0000:c1:00.0 enp193s0f0np0: FEC autoneg off encoding: None
[   15.014741] bnxt_en 0000:c1:00.0 enp193s0f0np0: entered allmulticast mode
[   15.014828] bnxt_en 0000:c1:00.0 enp193s0f0np0: entered promiscuous mode
 
sbellon said:
Is there any way to check - while still on the old kernel - whether this renaming will affect oneself when rebooting with the 6.8 kernel, so that one could take the appropriate measures in the `/etc/network/interfaces` blindly before rebooting?

This renaming may not be a big issue for a machine where you have direct access to, but for everything administered remotely - or even your virtual firewall host - this can be quite annoying.
Click to expand...
Youre right, sure if you have an FirewallVM on the PVE Host it gets shitty xD

Im not sure, i would say probably yes, but i don't know, i think google or chatgpt could find an answer how the renaming works exactly and how to check before reboot, maybe network/interfaces supports some sort of wildcard? i mean in theory you could define multiple interfaces with the same ip etc.
However there is surely a way, may be a scripting way to execute simply a script on boot that greps the correct name based on the mac address, then changes network interfaces and reload the interfaces.

But that leads me to another idea, im sure there is a way to define an interface name based on the mac address, but you have to google.

Cheers
 
mac.linux.free said:
Apr 07 00:07:44 fbo-vmh-024 kernel: UBSAN: shift-out-of-bounds in ./include/linux/log2.h:57:13
Apr 07 00:07:44 fbo-vmh-024 kernel: shift exponent 64 is too large for 64-bit type 'long unsigned int'
Click to expand...
I managed to reproduce the UBSAN warning with a machine with broadcom NICs here - but did not get the firmware hangs:
Daniel Keller said:
bnxt_en 0000:c1:00.1: QPLIB: bnxt_re_is_fw_stalled: FW STALL Detected. cmdq[0xe]=0x3 waited (102364 > 100000) msec active 1
Click to expand...
(but that might be due to different firmwares, or because I did not send loads of traffic over the interfaces)

To get the UBSAN warnings (undefined behavior sanitzier - to my knowledge these are warnings printed by the kernel to notify driver-maintainers about potentially problematic uses in their code, but they do not by themselves cause problems) - I needed to enable rdma/ib for the NIC using broadcom's niccli [0] utility (you can download it from the broadcom website, and need to install the utility and the dkms package for the `sliff` driver).

Sadly I don't have too much experience with Broadcom infiniband NICs, and whether changing their settings (or loading the third-party sliff module) while running can cause problems - so please be careful, and don't do this in production!

On a hunch - maybe the BCM57416 in general, or only as on-board NICs on the supermicro board have the RDMA setting enabled by default, while most other broadcom NICs do not (we would have a larger number of reports with problems if many broadcom nics are affected).

You could try disabling the rdma-support on the NICs:
Code: 
niccli -i 1 nvm -setoption support_rdma -scope 0 -value 0
niccli -i 1 reset

for the interface index and scope setting - and a general info - please consult the broadcom documentation - I also found the following article from the thomas-krenn wiki helpful (in German):
https://www.thomas-krenn.com/de/wiki/Broadcom_NICCLI_Configuration_Utility


Alternatively you could also try to unload/blocklist the bnxt_re module from getting loaded (afaict it is the module that provides RDMA/IB functionality for broadcom NICs)

If this does not help - I'd suggest to open a new thread here (feel free to mention me @Stoiko Ivanov, so I do not overlook it), to keep the general thread for the 6.8 kernel less noisy.

[0] https://techdocs.broadcom.com/us/en...-software-installation-and-configuration.html
 
  • Like
Reactions: fireon, mariol, jsterr and 1 other person
Hi. Is it possible to include bcacheFS in the kernel? I want to do some testing on this COW filesystem.
 
Hey @t.lamprecht
With todays update:

Code: 
ksm-control-daemon/stable 1.5-1 all [upgradable from: 1.4-1]
libnvpair3linux/stable 2.2.3-pve2 amd64 [upgradable from: 2.2.3-pve1]
libuutil3linux/stable 2.2.3-pve2 amd64 [upgradable from: 2.2.3-pve1]
libzfs4linux/stable 2.2.3-pve2 amd64 [upgradable from: 2.2.3-pve1]
libzpool5linux/stable 2.2.3-pve2 amd64 [upgradable from: 2.2.3-pve1]
proxmox-kernel-6.8/stable 6.8.4-2 all [upgradable from: 6.8.1-1]
spl/stable 2.2.3-pve2 all [upgradable from: 2.2.3-pve1]
zfs-initramfs/stable 2.2.3-pve2 all [upgradable from: 2.2.3-pve1]
zfs-zed/stable 2.2.3-pve2 amd64 [upgradable from: 2.2.3-pve1]
zfsutils-linux/stable 2.2.3-pve2 amd64 [upgradable from: 2.2.3-pve1]

zfs-import@HDD\\x2dSATA.service got deleted/dissapeared after reboot.
zfs-import@HDD\\x2dSAS.service was keeped, just for reference, because i have 2 Pools.

Just for refernce the Servers Pools:
Code: 
root@bs:~# zpool status
  pool: HDD-SAS
 state: ONLINE
  scan: scrub repaired 0B in 00:19:59 with 0 errors on Sun Mar 10 00:44:00 2024
config:

        NAME                        STATE     READ WRITE CKSUM
        HDD-SAS                     ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            scsi-350000399d8914779  ONLINE       0     0     0
            scsi-35000cca25806ef60  ONLINE       0     0     0
          mirror-1                  ONLINE       0     0     0
            scsi-350000399d892a6a1  ONLINE       0     0     0
            scsi-35000cca25806aee0  ONLINE       0     0     0
          mirror-2                  ONLINE       0     0     0
            scsi-35000cca259029b90  ONLINE       0     0     0
            scsi-35000cca2db04db2c  ONLINE       0     0     0

errors: No known data errors

  pool: HDD-SATA
 state: ONLINE
config:

        NAME                          STATE     READ WRITE CKSUM
        HDD-SATA                      ONLINE       0     0     0
          raidz1-0                    ONLINE       0     0     0
            ata-MB4000GCWDC_S1Zxxxxx  ONLINE       0     0     0
            ata-MB4000GCWDC_Z1Zxxxxx  ONLINE       0     0     0
            ata-MB4000GCWLV_PEGxxxxx  ONLINE       0     0     0
          raidz1-1                    ONLINE       0     0     0
            ata-MB4000GCWLV_PEGxxxxx  ONLINE       0     0     0
            ata-MB4000GCWLV_PEGxxxxx  ONLINE       0     0     0
            ata-MB4000GCWLV_PEGxxxxx  ONLINE       0     0     0
          raidz1-2                    ONLINE       0     0     0
            ata-MB4000GCWLV_PEGxxxxx  ONLINE       0     0     0
            ata-MB4000GCWLV_PEHxxxxx  ONLINE       0     0     0
            ata-MB4000GCWLV_PEHxxxxx  ONLINE       0     0     0
          raidz1-3                    ONLINE       0     0     0
            ata-MB4000GCWLV_PEHxxxxx  ONLINE       0     0     0
            ata-MB4000GCWLV_PEHxxxxx  ONLINE       0     0     0
            ata-MB4000GCWLV_PEHxxxxx  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 00:02:42 with 0 errors on Sun Mar 10 00:26:48 2024
config:

        NAME                              STATE     READ WRITE CKSUM
        rpool                             ONLINE       0     0     0
          mirror-0                        ONLINE       0     0     0
            scsi-35000cca05b23c294-part3  ONLINE       0     0     0
            scsi-35000cca05b23aa78-part3  ONLINE       0     0     0

errors: No known data errors

Otherwise no issues, i just recreated the Service and rebooted, the SATA Pool got imported and all is fine.
Just wondering why the service was deleted.

Cheers
 
ness1602 said:
Isn't it included already in 6.8
Click to expand...
To me it seems they have not included it when compiling the kernel.

Yes, I've installed bcachefs-tool, but can't format or mount any filesystem. That's why I asked for them to include it in the kernel.
 
Ramalama said:
zfs-import@HDD\\x2dSATA.service got deleted/dissapeared after reboot.
zfs-import@HDD\\x2dSAS.service was keeped, just for reference, because i have 2 Pools.
Click to expand...
Not sure I understand what you mean by disappeared/deleted and kept ?

The new kernel should not touch your enabled/present services!
One thing that changed is that the zfs-import@.service units are ordered after the zfs-import-cache.service or zfs-import-scan.service (to prevent a cosmetic warning about the pool not being able to be imported) - but the services are still there.

If this is some actual issue - please open a new thread (and notify me with @Stoiko Ivanov ) - and we can take a look
 
Stoiko Ivanov said:
Not sure I understand what you mean by disappeared/deleted and kept ?

The new kernel should not touch your enabled/present services!
One thing that changed is that the zfs-import@.service units are ordered after the zfs-import-cache.service or zfs-import-scan.service (to prevent a cosmetic warning about the pool not being able to be imported) - but the services are still there.

If this is some actual issue - please open a new thread (and notify me with @Stoiko Ivanov ) - and we can take a look
Click to expand...
Thanks @Stoiko Ivanov
That systemd service gets automatically created if you create an zfs pool inside the proxmox GUI i believe.
At least i don't know what creates that service usually, just after the mentioned update my HDD-SATA pool wasn't imported on boot.
Then I checked why and found that that service is missing.

I mean it's not really an issue, since you can simply add that service and the pool will get imported again at boot time, but i just wanted to mention that.
That maybe something was forgotten in the updated packages...
Otherwise everything is perfect.
And it's not an issue for me, since it's extremely simple to add again.

I would not open a new thread for this, because im not sure if it's specific to my system and no one else will have the same issue. So let's simply see if someone else stumble about that.
Cheers
 
McKajVah said:
To me it seems they have not included it when compiling the kernel.

Yes, I've installed bcachefs-tool, but can't format or mount any filesystem. That's why I asked for them to include it in the kernel.
Click to expand...
SORRY! My bad.

It's already included in the kernel. I was getting some strange errors and it seems it was a typo error from my side all along.
 
Installed 6.8 without issues, seems to be snappier. Windows VM performance seems improved as well.

X570
Ryzen 5800X
 
Ramalama said:
That systemd service gets automatically created if you create an zfs pool inside the proxmox GUI i believe.
Click to expand...
yes - exactly! - this is why it's really odd that the service vanished - I'd suggest to check your journal (a few boots before the 6.8 kernel) - to see if the service used to exist - and maybe also check /var/log/apt/term.log if there's any indication that the service got removed during upgrade (it should not)

but glad your system is running fine!
 
  • Like
Reactions: Ramalama
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom