Opt-in Linux 5.19 Kernel for Proxmox VE 7.x available

[t.lamprecht]

We recently uploaded a 5.19 kernel into our repositories. The 5.15 kernel will stay the default on the Proxmox VE 7.x series, 5.19 is an option.
5.19 may be useful for some (especially newer) setups, for example if there is improved hardware support that has not yet been backported to 5.15.

How to install:

• apt update
• apt install pve-kernel-5.19
• reboot

Future updates to the 5.19 kernel will now get installed automatically.

Please note:

• It's not required to enable the pvetest repository, the opt-in kernel package is available on all repositories.
• While we are trying to provide a stable experience with the opt-in 5.19 kernel, updates may appear less frequently than for the default 5.15 based kernel.
• If unsure, we recommend continuing to use the 5.15 based kernel.

Feedback is welcome!

Edit: The 5.19 opt-in based kernel has been replaced by the 6.1 opt-in kernel.

 

[spirit]

be carefull of retbleet mitigation, performance impact seem to be huge

https://lore.kernel.org/lkml/YxxCMSYaI/o3QH03@nazgul.tnic/T/

https://www.phoronix.com/review/retbleed-benchmark

 

[leesteken]

Passthrough of two AMD GPU (with vendor-reset), audio, SATA and USB controllers works well on X570S. No more need of initcall_blacklist=sysfb_init work-around for passthrough of boot GPU, which was needed after 5.11.22 (because amdgpu crashes when unloading) until 5.15.35 and after recent (Debian 11.5?) update again.
lm-sensors still does not detect it8628 on X570S AERO G but work-around still works and not related to Proxmox.
I do not notice any regressions but I also don't see the wlan device of the mt7921e (driver in use) but that probably requires kernel 5.19.8 (or 5.15.67).

 

[t.lamprecht]

be carefull of retbleet mitigation, performance impact seem to be huge

https://lore.kernel.org/lkml/YxxCMSYaI/o3QH03@nazgul.tnic/T/

https://www.phoronix.com/review/retbleed-benchmark

Those are in 5.15 too since end of July, i.e. pve-kernel-5.15.39-2.

 

[itNGO]

Those are in 5.15 too since end of July, i.e. pve-kernel-5.15.39-2.

Did anyone made measurement if and how much this slows performance down in PVE?

 

[fabian]

Did anyone made measurement if and how much this slows performance down in PVE?

well, that depends a lot on hardware and workload, like with all of these mitigations. you can disable it via the kernel cmdline and check for yourself (and then decide which tradeoff you want to make w.r.t. performance vs. security/safety, like with all optional mitigations and security features).

note that the kernel devs are already working on re-gaining a lot of the performance lost with this specific mitigation, but the changes are not trivial so likely it will take a bit before they become available in released kernel versions (upstream and in Proxmox products): https://lore.kernel.org/all/20220902130625.217071627@infradead.org/

 

[Neobin]

Finally the latest and greatest.
Successfully installed and running on:

• 5950X on X570 with nvidia-driver=470.141.03
• 5700G on X300
• 9900K on Z390 with kde-plasma-desktop

All are completely on bullseye-backports and have PBS aside.

Only the Intel throwed some possible missing firmware warnings at me:

Spoiler

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  pve-kernel-5.19.7-1-pve
The following NEW packages will be installed:
  pve-kernel-5.19 pve-kernel-5.19.7-1-pve
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 77.6 MB of archives.
After this operation, 412 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://download.proxmox.com/debian/pbs bullseye/pbs-no-subscription amd64 pve-kernel-5.19.7-1-pve amd64 5.19.7-1 [77.6 MB]
Get:2 http://download.proxmox.com/debian/pbs bullseye/pbs-no-subscription amd64 pve-kernel-5.19 all 7.2-11 [4,976 B]
Fetched 77.6 MB in 2s (31.2 MB/s)         
Selecting previously unselected package pve-kernel-5.19.7-1-pve.
(Reading database ... 182580 files and directories currently installed.)
Preparing to unpack .../pve-kernel-5.19.7-1-pve_5.19.7-1_amd64.deb ...
Unpacking pve-kernel-5.19.7-1-pve (5.19.7-1) ...
Selecting previously unselected package pve-kernel-5.19.
Preparing to unpack .../pve-kernel-5.19_7.2-11_all.deb ...
Unpacking pve-kernel-5.19 (7.2-11) ...
Setting up pve-kernel-5.19.7-1-pve (5.19.7-1) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.19.7-1-pve /boot/vmlinuz-5.19.7-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.19.7-1-pve /boot/vmlinuz-5.19.7-1-pve
update-initramfs: Generating /boot/initrd.img-5.19.7-1-pve
W: Possible missing firmware /lib/firmware/i915/dg1_huc_7.9.3.bin for module i915
W: Possible missing firmware /lib/firmware/i915/tgl_guc_69.0.3.bin for module i915
W: Possible missing firmware /lib/firmware/i915/adlp_guc_69.0.3.bin for module i915
W: Possible missing firmware /lib/firmware/i915/skl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/bxt_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/kbl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/glk_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/kbl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/kbl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/cml_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/icl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/ehl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/ehl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/tgl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/tgl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/dg1_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/tgl_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/adlp_guc_70.1.1.bin for module i915
W: Possible missing firmware /lib/firmware/i915/dg2_guc_70.1.2.bin for module i915
W: Possible missing firmware /lib/firmware/i915/adlp_dmc_ver2_16.bin for module i915
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/7048-8851
        Copying kernel and creating boot-entry for 5.15.53-1-pve
        Copying kernel and creating boot-entry for 5.19.7-1-pve
        Removing old version 5.15.39-4-pve
Couldn't find EFI system partition. It is recommended to mount it to /boot or /efi.
Alternatively, use --esp-path= to specify path to mount point.
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 5.19.7-1-pve /boot/vmlinuz-5.19.7-1-pve
run-parts: executing /etc/kernel/postinst.d/systemd-boot 5.19.7-1-pve /boot/vmlinuz-5.19.7-1-pve
Couldn't find EFI system partition. It is recommended to mount it to /boot or /efi.
Alternatively, use --esp-path= to specify path to mount point.
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 5.19.7-1-pve /boot/vmlinuz-5.19.7-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 5.19.7-1-pve /boot/vmlinuz-5.19.7-1-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
Copying and configuring kernels on /dev/disk/by-uuid/7048-8851
        Copying kernel and creating boot-entry for 5.15.53-1-pve
        Copying kernel and creating boot-entry for 5.19.7-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.19.7-1-pve /boot/vmlinuz-5.19.7-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.19.7-1-pve
Found initrd image: /boot/initrd.img-5.19.7-1-pve
Found linux image: /boot/vmlinuz-5.15.53-1-pve
Found initrd image: /boot/initrd.img-5.15.53-1-pve
Found linux image: /boot/vmlinuz-5.15.39-4-pve
Found initrd image: /boot/initrd.img-5.15.39-4-pve
Found memtest86+ image: /ROOT/pve-1@/boot/memtest86+.bin
Found memtest86+ multiboot image: /ROOT/pve-1@/boot/memtest86+_multiboot.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
Setting up pve-kernel-5.19 (7.2-11) ...

 

[mac.linux.free]

Did anyone made measurement if and how much this slows performance down in PVE?

you can always set retbleed=off in grub.

 

[celemine1gig]

Just tried the Kernel 5.19 on my Xeon E5 v4 system with passed through GTX 980 Ti under Windows.
Experienced massive lags/delays in games. Same setup runs smooth if I switch back to 5.15. I guess there still is work to do.
BTW, nothing obvious in the system log, when the delays hit. And the interrupt setup also is the same. Double-checked.

 

[t.lamprecht]

Experienced massive lags/delays in games. Same setup runs smooth if I switch back to 5.15. I guess there still is work to do.

Note that with 5.19 the schedulers preemption types are finally not compiled in anymore and can be set on boot. You could try to adapt those:

 preempt=        [KNL]
                 Select preemption mode if you have CONFIG_PREEMPT_DYNAMIC
                 none - Limited to cond_resched() calls
                 voluntary - Limited to cond_resched() and might_sleep() calls
                 full - Any section that isn't explicitly preempt disabled
                        can be preempted anytime.

-- Documentation/admin-guide/kernel-parameters.txt

none means most throughput (often best for servers), voluntary brings less latency for processes (often better for desktop/ui) at the cost of some smaller throughput impact (this is somewhat of a balance), and full has the lowest latency but also the biggest impact on throughput. It seems default is "voluntary"

You could try booting with preempt=none to see if that improves your situation. Nnote that with VMs the host OS giving more throughput to the guest OS should mean that the guest OS has more control about its guest perceived latency. Would be interesting to get feedback here.

 

[celemine1gig]

Tried all three kernel parameters with the current 5.19 pve-kernel. Sadly none brought back the previous functionality, as seen on 5.15.
I do not have measurements, but it felt like "voluntary" was the worst, followed by "full" and the most acceptable (yet not nearly as good as on 5.15) was "none".

 

[celemine1gig]

BTW, another quick question, about the upcoming kernel 5.19:
Is there a chance to solve that dreadful simplefb dilemma with the 5.19 configuration, concerning simplefb.
I have a Proxmox CoffeeLake system with passed-though iGPU. Up until 5.13 this worked (and still works) beautifully.
However, since 5.15, the simplefb claims the iGPU and, in contrast to external PCIe devices, the workaround by resetting the device, does not work.
For the internal function, this seems more or less like an unsolveable situation, as long as simplefb claims those devices by default.

 

[leesteken]

However, since 5.15, the simplefb claims the iGPU and, in contrast to external PCIe devices, the workaround by resetting the device, does not work.

The work-around with the initcall_blacklist=sysfb_init kernel parameter also does not work for you?

 

[celemine1gig]

OK, so you can scratch my request, as I obviously did not know about that option.
Just tested and works fine with 5.15 and 5.19.
Now that you mentioned it, I think I remember having read about it some time ago, but then forgot to try it out.
Thanks anyway! Much appreciated input.

 

[Giovanni]

This is perfect timing. I had a new Intel N6000 processor that wasn't working on kernel 5.15 - I was using https://github.com/fabianishere/pve-edge-kernel to fill in the gap to get proper hardware working.

 

[Dark26]

I try to install it on my N5105 server , and it crash after booting with this version of 5.19

 

[t.lamprecht]

What crashes (host, VM, ..?) and do you got any specifc error logs that show up before/during crash? Also more details about the HW could be nice to have.

 

[Dark26]

What crashes (host, VM, ..?) and do you got any specifc error logs that show up before/during crash? Also more details about the HW could be nice to have.

With the 5.15, host stable, and one vm ( debian with container ( bitwarder ) crash. No problem with oher Vm / lxc.

with 5.19, the host boot, but it seem that when it start lauching the VM /lxc , the hosts crash.

i watch quickly, the log, and maybe it's a storage problem ( Nfs ganesha over glusterfs not starting automaticly a boot) and when the VM start on the storage, everything crash.

i will watch to reboot with the secreen on when i have time

 

[Daniel Keller]

nice with version 5.19 I can again migrate my VMs between my servers without the VMs crashing

 

[Randell]

It looks like it has resolved my migration issues to/from an i7-12700K and i7-8700K machine.