Opt-in Linux Kernel 5.15 for Proxmox VE 7.x available

[t.lamprecht]

The next Proxmox VE point release 7.2 (~ Q2/2022) will presumably use the 5.15 based Linux kernel.
You can test this kernel now, install the meta-package that pulls in the latest 5.15 kernel with:

apt update && apt install pve-kernel-5.15

It's not required to enable the pvetest repository, the opt-in kernel package is available on all repositories.

We invite you to test your hardware with this kernel, and we are thankful for receiving your feedback.

Please note that while we are trying to provide a stable experience with the Opt-in kernel 5.15, updates for this kernel may appear less frequently until Proxmox projects actually switch to it as their new default.

 

[ness1602]

It is working great with Epyc, and also with some older ryzen and intel cpus.

 

[birdy]

Just installed 5.15 and rebooted. So far so good.

 

[Felix.]

Running smooth so far on a Ryzen 1700X. GPU passthrough of a GTX1060 works. No issues at all, so far.

 

[leesteken]

I just installed pve-kernel 5.15 and headers on a X470 with Ryzen 2700X. It boots fine from ZFS with UEFI. Passthrough of both AMD GPUs and onboard audio works fine. vendor-reset compiled fine (during installation of kernel) and I still see kernel: vendor_reset_hook: installed in the system log. However, I no longer see AMD_POLARIS10: performing reset etc. as before in 5.11 and 5.13. This makes me think that vendor-reset is not working on Proxmox with kernel 5.15, but stopping and restarting VM with passthrough still works fine. Therefore, I'm not sure if my test is a complete success, even though there are no issues.

Is there anything in particular that we should check with 5.15? Any new features to try?

EDIT: I found a regression for my system. I always needed reboot=efi on this system because otherwise reboots would hang, but with kernel 5.15 it stopped working and the system hangs at around Reached target Reboot. Which is a pain because the system is forced to be headless since 5.11.22-7, so I don't know when to press the reset button. This might not be a Proxmox issue once again, but it makes me want to revert to kernel 5.13.

 

[Raschen]

Hardware acceleration not working, is there a solution?

Linux pve 5.15.5-1-pve
CPU - 11th Gen Intel(R) Core(TM) i7-11700 @ 2.50GHz
---------------------------------------------------------------------------------------------------------------------------------------------------
lspci

00:02.0 VGA compatible controller: Intel Corporation Device 4c8a (rev 04) (prog-if 00 [VGA controller])
    DeviceName: Onboard - Video
    Subsystem: Gigabyte Technology Co., Ltd Device d000
    Flags: bus master, fast devsel, latency 0, IRQ 190, IOMMU group 1
    Memory at 60000000 (64-bit, non-prefetchable) [size=16M]
    Memory at 40000000 (64-bit, prefetchable) [size=512M]
    I/O ports at 4000 [size=64]
    Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
    Capabilities: [40] Vendor Specific Information: Len=0c <?>
    Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit-
    Capabilities: [d0] Power Management version 2
    Capabilities: [100] Process Address Space ID (PASID)
    Capabilities: [200] Address Translation Service (ATS)
    Capabilities: [300] Page Request Interface (PRI)
    Kernel driver in use: i915
    Kernel modules: i915
---------------------------------------------------------------------------------------------------------------------------------------------------
dmesg  | grep i915

[    6.506221] i915 0000:00:02.0: [drm] VT-d active for gfx access
[    6.506227] i915 0000:00:02.0: vgaarb: deactivate vga console
[    6.507908] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    6.508471] i915 0000:00:02.0: Direct firmware load for i915/rkl_dmc_ver2_03.bin failed with error -2
[    6.508475] i915 0000:00:02.0: [drm] Failed to load DMC firmware i915/rkl_dmc_ver2_03.bin. Disabling runtime power management.
[    6.508477] i915 0000:00:02.0: [drm] DMC firmware homepage: https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/i915
[    6.604658] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[    6.606354] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[    6.606464] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    6.606520] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    6.635900] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
---------------------------------------------------------------------------------------------------------------------------------------------------

 

[t.lamprecht]

EDIT: I found a regression for my system. I always needed reboot=efi on this system because otherwise reboots would hang, but with kernel 5.15 it stopped working and the system hangs at around Reached target Reboot. Which is a pain because the system is forced to be headless since 5.11.22-7, so I don't know when to press the reset button. This might not be a Proxmox issue once again, but it makes me want to revert to kernel 5.13.

That can well just be a misconfiguration of a service or most often mount unit where systemd is not aware that it needs to unmount it before stopping network, once the shutdown target is reached the "real fun" only starts..
What can help is turning up logging from systemd, starting out with something like the following as kernel command line:
systemd.log_target=kmsg systemd.log_level=debug printk.devkmsg=on and see where it hangs after reaching target shutdown. IIRC that also should give you a debug shell with root on tty9 or so, but not so sure anymore if that's still useable after the "reached target shutdown" stage.

 

[t.lamprecht]

Hardware acceleration not working, is there a solution?

Well was the GPU HW of that consumer HW working all well before, with an older kernel?
The only thing that's noticeable from that kernel log is that the power management firmware this kernel module version expects was not found. FWIW, a pve-firmware bump is planned soon.

 

[leesteken]

That can well just be a misconfiguration of a service or most often mount unit where systemd is not aware that it needs to unmount it before stopping network, once the shutdown target is reached the "real fun" only starts..
What can help is turning up logging from systemd, starting out with something like the following as kernel command line:
systemd.log_target=kmsg systemd.log_level=debug printk.devkmsg=on and see where it hangs after reaching target shutdown. IIRC that also should give you a debug shell with root on tty9 or so, but from not sure anymore if that's still useable after the "reached target shutdown" stage.

Thank you for the tips for debugging. It turns out that the work-around of reboot=efi (from years ago) is not necessary for either kernel version. Removing it fixed my reboot issue and I'm using 5.15 now.

 

[Raschen]

Well was the GPU HW of that consumer HW working all well before, with an older kernel?
The only thing that's noticeable from that kernel log is that the power management firmware this kernel module version expects was not found. FWIW, a pve-firmware bump is planned soon.

Kernel 5.13 is identical.
Graphics does not work under any load in intel_gpu_top

 

[t.lamprecht]

Kernel 5.13 is identical.
Graphics does not work under any load in intel_gpu_top

Would be good to note that, as your original wording sounds like that has been a regression from 5.15

You may require a newer mesa version for that to work, the 20.3 from bullseye is too old for the iris GPUs released this year.
I'm using such a CPU in my laptop with Debian Sid (unstable) and with mesa 21.x it works just fine there, even with an older kernel. There's no newer version available in a backport yet, though, so you'd need to build it yourself (or open a wishlist bug on debian mesa package asking for a backport)

 

[birdy]

Almost 24h on the way and all works ok. Only noticable is the much higher CPU usage.

Most of my VMs do not drop below 20% anymore where they were at approx. 4-5% on average before.

Host:


Guest:

 

[Kelu]

Upgrading my Ryzen 7 5700U-based Asus PN51 from Proxmox 7.0 to 7.1 broke networking. Installing this 5.15 kernel fixed the issue. Thanks...

 

[joao0]

5.15 fixed the hang boot issue I had with windows 11 on my epyc server when using hyper-v but it's still running into 100% CPU and getting stuck after about 10 minutes from boot making it unusable.

Any ideas why is that happening?

 

[vmware]

Updated my Ryzen box to 5.15 kernel. Seems to work well and nothing dramatic.

However, I still experience a long standing PVE 7.1 issue - In Debian 11.0 CT with nesting=1 , "su username" still takes 20 seconds or so to complete. Symptom is very similar to this https://forum.proxmox.com/threads/l...ullseye-slow-login-and-apparmor-errors.93064/

 

[janssensm]

Thanks for publishing this kernel.
I installed it on a old and simple system.

With 5.13.19-2-pve it detected sata hdd and ssd with, one example:

kernel: sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
kernel: sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

With 5.15.5-1-pve:

kernel: sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
kernel: sd 2:0:0:0: [sda] Asking for cache data failed
kernel: sd 2:0:0:0: [sda] Assuming drive cache: write through

but smartctl still says:

~# smartctl -x /dev/sda | grep cache
Write cache is:   Enabled

Seem like it's this bug:
https://bugzilla.kernel.org/show_bug.cgi?id=215137
I thought it was just my old system, but according the supplied dmesg from this report it was linked to a AMD Ryzen 9 5950X.
This bug appears to be fixed in kernel 5.15.6.

On another old system, but with sata drives connected through a PERC H310 hba (flashed IT mode), no issues with 5.15.5-1-pve.

 

[t.lamprecht]

This bug appears to be fixed in kernel 5.15.6.

FWIW, pve-kernel-5.15.7-1-pve (package version 5.15.7-1) is available on the pvetest repository as of now if you want to test if that fix you found was really the remedy for your issue.

 

[spirit]

Almost 24h on the way and all works ok. Only noticable is the much higher CPU usage.

Most of my VMs do not drop below 20% anymore where they were at approx. 4-5% on average before.

Host:
View attachment 32251

Guest:
View attachment 32252

can you compare host cpu frequency && governor between both kernel ?

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq

 

[birdy]

Linux PVE01 5.15.5-1-pve #1 SMP PVE 5.15.5-1 (Mon, 29 Nov 2021 18:49:57 +0100) x86_64

root@PVE01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
root@PVE01:~# cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq
4200021
4200033
4089445
4200018
root@PVE01:~#

Linux PVE01 5.13.19-2-pve #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100) x86_64

root@PVE01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
root@PVE01:~# cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq
4368934
4217306
4333556
4335916
root@PVE01:~#

And CPU usage is down again (as expected):

 

[janssensm]

FWIW, pve-kernel-5.15.7-1-pve (package version 5.15.7-1) is available on the pvetest repository as of now if you want to test if that fix you found was really the remedy for your issue.

Thanks for publishing this new kernel version.
I just installed it and can confirm that the drive sense mode reports correct cache parameters, the same as on 5.13.19-2-pve.
So I can confirm that this issue is fixed with 5.15.7-1-pve