Opt-in Linux 6.5 Kernel with ZFS 2.2 for Proxmox VE 8 available on test & no-subscription

I have installes 6.5 kernel and ZFS 2.2.0 on all my three servers and it works without any problems:
Server 1 - AMD Epyc 7272
Server 2 - 2x Intel Xeon E5-2637 with PBS additional
Server 3 - Intel i7-10700K with PBS additional
 
Hi,
Nemesiz said:
ZFS 2.2.0 have a lot of bugs. I prefer to wait for 'more' stable release.

2.2.1 is out https://github.com/openzfs/zfs/releases/tag/zfs-2.2.1
Click to expand...
you're free to wait of course. But please note that kernel packages >= 6.5.11-3 already include known important fixes staged for ZFS 2.2.1:
https://git.proxmox.com/?p=pve-kernel.git;a=commit;h=5111be4ff3c456ae4c580babce7370ef196a415b
https://git.proxmox.com/?p=zfsonlinux.git;a=commit;h=96c807af63f70dc930328e5801659a5bd40e6d47
 
t.lamprecht said:
Yes, it was, see the commit list in the git repo: https://git.proxmox.com/?p=pve-firmware.git;a=summary


Thanks for your report. Firmware matching is especially a bit hard with intel drivers, as the only register the most recent version to the firmware list shipped my the module (e.g., see modinfo -F firmware iwlwifi) that we use to filter out all those firmware files that the kernels wouldn't use anyway to avoid wasting hundreds of MB (or more) of space.

Anyhow, I checked on a workstation that has an Intel AX210 card and see some kernel splat oops dump too, but firmware also loads successfully there too, a bit before the error get reported. The linux-firmware repo has some newer firmware files for intel wifi cards, so maybe indeed a bug in the FW there, we'll probably update pve-firmware again relatively soon anyway.

edit: potentially related:
https://bugzilla.kernel.org/show_bug.cgi?id=217894
https://lore.kernel.org/linux-wirel...35ca0b1079f2c2ea92fd8d14d8101a89d03@changeid/
Click to expand...
I pulled the machine having this issue apart... it's an intel AX210 PCIe device. It's not actually embedded like I thought, so I might be able to replace it with something that has less drivers issues. While I've never used the wifi, I use it for bluetooth ( passed to home assistant.) I can't find anything in BIOS that would let me disable the device entirely. Perhaps I can switch to just a generic bluetooth USB dongle.

Thank you VERY much for looking into the report. While I'm just a home user with a home "lab" (1 machine), the fact that you're responding to even me really makes me feel good about the product (and perhaps I get my company to use your product, with a subscription, in the near future.)

Take care
Gary

Edit:

For giggles, I downloaded the latest tagged linux firmware package (https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git - linux-firmware-20231111.tar.gz), and copied any files missing from pve-firmware into /lib/firmware.

(I didn't overwrite any existing files.) I reinstalled kernel 6.5 and rebooted - and everything worked perfectly.

What surprised me was that after the working reboot, I noticed that the same firmware file was being loaded: 83.e8f84e98.0 so-a0-hr-b0-83.ucode. My only guess (and it's ONLY a guess) is that the pve-firmware package has compressed versions of the firmware files (.ucode.xz extension) while the downloaded tarball had uncompressed versions. (.ucode)

I'm not sure. I really need to figure out a way to prevent the wifi device from loading while still allowing the BT functionality to work.
 
Last edited:
t.lamprecht said:
This is due to the new notification framework, you can either drop the mail from the jobs to use the new infrastructure, or edit/disable the default matcher under Datacenter -> Notifications.
Click to expand...
I did this on mine to mitigate the issue, however turning off the notification matcher does not stop the emails being sent unfortunatelly.
 
Ever since upgrading to the new kernel, I see a one-time kernel WARN() message upon booting:

[ 161.759197] WARNING: CPU: 37 PID: 0 at net/core/flow_dissector.c:1052 __skb_flow_dissect+0xb2f/0x1ee0 [ 161.769748] Modules linked in: cfg80211 xt_multiport xt_nat xt_addrtype xt_mark xt_tcpudp xt_conntrack xt_MASQUERADE xt_comment nft_compat nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_masq nft_ct nft_nat nft_chain_nat nf_nat nf_c onntrack nf_defrag_ipv6 nf_defrag_ipv4 veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nvme_fabrics nf_tables softdog sunrpc 8021q garp mrp binfmt_misc bonding tls nfnetlink_log nfnetl ink ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel acpi_ipmi aesni_intel crypto_simd ast cryptd ipmi_si drm_shmem_helper rapl pcspkr ch341 wmi_bmof ipmi_devintf drm_kms_helper ccp usbserial k10temp ptdma ipmi_msghandler 8250_dw joydev input_leds mac_hid vhost_net vhost vhost_iotlb tap msr vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd sit tunnel4 ip_tunnel efi_pstore drm dmi_s ysfs ip_tables x_tables autofs4 zfs(PO) [ 161.770922] spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c hid_generic usbkbd usbmouse usbhid hid cdc_ether usbnet mii xhci_pci qede xhci_pci_renesas crc32_pclmul nvme igb i2c_algo_bit ahci qed xhci_hcd nvme_core libahci dca crc8 i2c_piix4 nvme_common wmi [ 161.900978] CPU: 37 PID: 0 Comm: swapper/37 Tainted: P O 6.5.11-3-pve #1 [ 161.910702] Hardware name: ASUSTeK COMPUTER INC. RS500A-E11-RS12U/KMPA-U16 Series, BIOS 1301 01/05/2023 [ 161.921830] RIP: 0010:__skb_flow_dissect+0xb2f/0x1ee0 [ 161.928578] Code: 41 89 d9 49 89 fc 83 f8 04 0f 84 be 01 00 00 85 c0 0f 84 b6 01 00 00 83 f8 02 0f 84 69 fd ff ff 0f b7 5c 24 44 e9 de fb ff ff <0f> 0b e9 e9 f7 ff ff f6 c2 20 0f 84 34 f6 ff ff 41 0f b7 45 0e 4c [ 161.950748] RSP: 0018:ffffb18ec0c28930 EFLAGS: 00010246 [ 161.957837] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 161.966729] RDX: 0000000000000008 RSI: ffffa047024ade00 RDI: 0000000000000000 [ 161.975606] RBP: ffffb18ec0c28a20 R08: 0000000000000000 R09: 0000000000000000 [ 161.984306] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa047024ade00 [ 161.993151] R13: ffffffff9bf7b460 R14: ffffa04701ff2310 R15: ffffb18ec0c28a50 [ 162.001983] FS: 0000000000000000(0000) GS:ffffa0c44ed40000(0000) knlGS:0000000000000000 [ 162.011754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 162.019122] CR2: 000055e5e1c45190 CR3: 000000026010c005 CR4: 0000000000770ee0 [ 162.027940] PKRU: 55555554 [ 162.032577] Call Trace: [ 162.036650] <IRQ> [ 162.040205] ? show_regs+0x6d/0x80 [ 162.045128] ? __warn+0x89/0x160 [ 162.049864] ? __skb_flow_dissect+0xb2f/0x1ee0 [ 162.055642] ? report_bug+0x17e/0x1b0 [ 162.060976] ? handle_bug+0x46/0x90 [ 162.066023] ? exc_invalid_op+0x18/0x80 [ 162.071414] ? asm_exc_invalid_op+0x1b/0x20 [ 162.077429] ? __skb_flow_dissect+0xb2f/0x1ee0 [ 162.083387] ? srso_alias_return_thunk+0x5/0x7f [ 162.089408] ? srso_alias_return_thunk+0x5/0x7f [ 162.095415] ? default_send_IPI_single_phys+0x3c/0x70 [ 162.101923] ? srso_alias_return_thunk+0x5/0x7f [ 162.107933] ? __smp_call_single_queue+0xe0/0x140 [ 162.114136] ? srso_alias_return_thunk+0x5/0x7f [ 162.120133] ? ttwu_queue_wakelist+0x101/0x110 [ 162.126052] ? srso_alias_return_thunk+0x5/0x7f [ 162.132041] ? try_to_wake_up+0x28d/0x5d0 [ 162.137518] __skb_get_hash+0x63/0x200 [ 162.142699] ? enqueue_to_backlog+0x124/0x330 [ 162.148479] nft_trace_init+0x104/0x110 [nf_tables] [ 162.154861] ? srso_alias_return_thunk+0x5/0x7f [ 162.160746] nft_do_chain+0x61f/0x6b0 [nf_tables] [ 162.166812] ? srso_alias_return_thunk+0x5/0x7f [ 162.172678] nft_nat_do_chain+0x9f/0x110 [nft_chain_nat] [ 162.179288] ? __nf_ct_refresh_acct+0x67/0x80 [nf_conntrack] [ 162.186342] ? srso_alias_return_thunk+0x5/0x7f [ 162.192125] nf_nat_inet_fn+0x161/0x340 [nf_nat] [ 162.197975] nf_nat_ipv4_local_fn+0x50/0x140 [nf_nat] [ 162.204225] nf_hook_slow+0x46/0xd0 [ 162.208877] __ip_local_out+0xee/0x170 [ 162.213756] ? __pfx_dst_output+0x10/0x10 [ 162.218878] ip_local_out+0x1c/0x70 [ 162.223455] igmp_send_report.isra.0+0x258/0x2e0 [ 162.229341] igmp_timer_expire+0x10e/0x130 [ 162.234486] ? srso_alias_return_thunk+0x5/0x7f [ 162.240046] ? __pfx_igmp_timer_expire+0x10/0x10 [ 162.245622] call_timer_fn+0x2c/0x160 [ 162.250258] ? __pfx_igmp_timer_expire+0x10/0x10 [ 162.255827] __run_timers+0x259/0x310 [ 162.260457] run_timer_softirq+0x1d/0x40 [ 162.265315] __do_softirq+0xd4/0x303 [ 162.269826] __irq_exit_rcu+0x75/0xa0 [ 162.274401] irq_exit_rcu+0xe/0x20 [ 162.278695] sysvec_apic_timer_interrupt+0x92/0xd0 [ 162.284287] </IRQ> [ 162.287176] <TASK> [ 162.290059] asm_sysvec_apic_timer_interrupt+0x1b/0x20 [ 162.296034] RIP: 0010:cpuidle_enter_state+0xce/0x470 [ 162.301719] Code: 28 10 ff e8 64 f6 ff ff 8b 53 04 49 89 c6 0f 1f 44 00 00 31 ff e8 22 25 0f ff 80 7d d7 00 0f 85 e7 01 00 00 fb 0f 1f 44 00 00 <45> 85 ff 0f 88 83 01 00 00 49 63 d7 4c 89 f1 48 8d 04 52 48 8d 04 [ 162.321945] RSP: 0018:ffffb18ec03d7e50 EFLAGS: 00000246 [ 162.327977] RAX: 0000000000000000 RBX: ffffa0471ebb5c00 RCX: 0000000000000000 [ 162.335919] RDX: 0000000000000025 RSI: 0000000000000000 RDI: 0000000000000000 [ 162.343860] RBP: ffffb18ec03d7e88 R08: 0000000000000000 R09: 0000000000000000 [ 162.351987] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 [ 162.359917] R13: ffffffff9be77c60 R14: 00000025a93b27e9 R15: 0000000000000002 [ 162.367871] cpuidle_enter+0x2e/0x50 [ 162.372518] call_cpuidle+0x23/0x60 [ 162.377080] do_idle+0x202/0x260 [ 162.381255] cpu_startup_entry+0x2a/0x30 [ 162.386107] start_secondary+0x119/0x140 [ 162.391057] secondary_startup_64_no_verify+0x17e/0x18b [ 162.397323] </TASK>

As far as I can tell, everything continues running as expected, though.
 
garyd9 said:
(I didn't overwrite any existing files.) I reinstalled kernel 6.5 and rebooted - and everything worked perfectly.

What surprised me was that after the working reboot, I noticed that the same firmware file was being loaded: 83.e8f84e98.0 so-a0-hr-b0-83.ucode. My only guess (and it's ONLY a guess) is that the pve-firmware package has compressed versions of the firmware files (.ucode.xz extension) while the downloaded tarball had uncompressed versions. (.ucode)
Click to expand...
The kernel supports indeed loading compressed firmware, depending on the config that can be zstd or xz.
And what's also true is that the loader checks for uncompressed files first, so if both, an uncompressed (downloaded by you) and a compressed one (installed by our current pve-firmware) exist, then the former uncompressed will be loaded.

So in your case it would mean that the from linux-firmware.git downloaded version, which tag matches the one we use, worked but the one pve-firmware ships didn't, seems a bit strange to me.. Possibly it only breaks on some boots?
You could also extract the compressed firmware shipped by pve-firmware and compare each files (e.g. sha256sum checksums)
 
Last edited:
t.lamprecht said:
So 6.2.16-19 also fails?

What hardware is this (CPU model, motherboard/system model?)?

What's the root file-system?

Any error when trying to boot?
Click to expand...

Sry,

6.2.16-9 also freeze
Ryzen 5 5600
Asrock B450M

/ = xfs

Freezing on first image "Load initial ramdisk"

GPU and NVME in passthrough
 
Last edited:
  • Like
Reactions: guilxxx
guilxxx said:
Hi!

Can't boot here!
View attachment 58638

Only work's in 6.2.16-14
View attachment 58639
Click to expand...

I also have this same issue (but maybe for a different reason). Linux 6.5 just doesn't like my Dell PowerEdge R240r (4 x Intel Xeon E-2224 CPU) and refuses to boot. It just gets stuck in this "Loading initial ramdisk" part and never gets past it. However, Linux 6.2 kernels work just fine without any issues. This computer doesn't have a GPU, so this isn't a passthrough issue.
 
Hi,
Same for me.
Dell PowerEdge T340 with Intel(R) Xeon(R) E-2136 refuses to boot. It just gets stuck in this "Loading initial ramdisk".
Revert to Linux 6.2 kernels work just fine. Server have no GPU.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back