Opt-in Linux 5.19 Kernel for Proxmox VE 7.x available

[Smooky]

Hi @ all

I just installed the 5.19 kernel on my server (MSI x399 SLI Plus with Ryzen Threadripper 1950X) and everything is running great.
The only thing I noticed is that the font size on the host console is now very large.
How can I reset them to their original size?

Kind regards

Smooky

 

[Juan Pablo Abuyeres]

It worked fine for several days, but now I have this in one node:

[324845.995993] BUG: unable to handle page fault for address: ffffffff00000008 [324845.996316] #PF: supervisor instruction fetch in kernel mode [324845.996610] #PF: error_code(0x0010) - not-present page [324845.996854] PGD 14e67214067 P4D 14e67215067 PUD 0 [324845.997089] Oops: 0010 [#1] PREEMPT SMP NOPTI [324845.997324] CPU: 99 PID: 935263 Comm: z_wr_iss Tainted: P O 5.19.7-1-pve #1 [324845.997558] Hardware name: iXsystems RS700-E10-RS12U-WOCPU005Z-IXN/Z12PP-D32 Series, BIOS 0701 10/15/2021 [324845.997795] RIP: 0010:0xffffffff00000008 [324845.998036] Code: Unable to access opcode bytes at RIP 0xfffffffeffffffde. [324845.998281] RSP: 0018:ff6cad692caf7d50 EFLAGS: 00010246 [324845.998525] RAX: 0000000000000010 RBX: 0000000000000000 RCX: ff46923f2038ddd8 [324845.998768] RDX: 0000000000000000 RSI: 0000000000001000 RDI: ff46923f2038e088 [324845.999008] RBP: ff6cad692caf7da0 R08: 0000000000001000 R09: ff46923f2038e068 [324845.999246] R10: 0000000000001000 R11: 0000000000000001 R12: ff46923f2038eb40 [324845.999484] R13: 0000000000080000 R14: ff46923f2038dca0 R15: 0000000000000000 [324845.999718] FS: 0000000000000000(0000) GS:ff469434ff6c0000(0000) knlGS:0000000000000000 [324845.999956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [324846.000191] CR2: fffffffeffffffde CR3: 0000014e67210003 CR4: 0000000000773ee0 [324846.000430] PKRU: 55555554 [324846.000668] Call Trace: [324846.000903] <TASK> [324846.001136] zio_execute+0x92/0x160 [zfs] [324846.001485] taskq_thread+0x29c/0x4d0 [spl] [324846.001726] ? wake_up_q+0x90/0x90 [324846.001961] ? zio_gang_tree_free+0x70/0x70 [zfs] [324846.002306] ? taskq_thread_spawn+0x60/0x60 [spl] [324846.002550] kthread+0xee/0x120 [324846.002796] ? kthread_complete_and_exit+0x20/0x20 [324846.003030] ret_from_fork+0x1f/0x30 [324846.003274] </TASK> [324846.003503] Modules linked in: xt_mac act_police cls_basic sch_ingress sch_htb nfsv3 nfs_acl veth rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs ebtable_filter ebtables ip6table_raw ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables iptable_raw ipt_REJECT nf_reject_ipv4 xt_mark xt_set xt_physdev xt_addrtype xt_comment xt_tcpudp xt_multiport xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bpfilter ip_set_hash_net ip_set sctp ip6_udp_tunnel udp_tunnel nf_tables 8021q garp mrp bonding softdog nfnetlink_log nfnetlink ipmi_ssif intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common i10nm_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd ast drm_vram_helper rapl drm_ttm_helper ttm drm_kms_helper i2c_algo_bit cdc_ether fb_sys_fops syscopyarea usbnet intel_cstate cmdlinepart pcspkr efi_pstore spi_nor sysfillrect input_leds [324846.003556] joydev mii isst_if_mbox_pci mei_me sysimgblt ioatdma isst_if_mmio mtd mei isst_if_common intel_vsec intel_pch_thermal acpi_ipmi dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor raid6_pq zstd_compress libcrc32c simplefb csiostor hid_generic usbmouse usbkbd usbhid hid uas usb_storage crc32_pclmul cxgb4 xhci_pci xhci_pci_renesas scsi_transport_fc tls spi_intel_pci i2c_i801 i40e ahci spi_intel i2c_smbus xhci_hcd libahci wmi [324846.007594] CR2: ffffffff00000008 [324846.007924] ---[ end trace 0000000000000000 ]--- [324846.100485] RIP: 0010:0xffffffff00000008 [324846.101045] Code: Unable to access opcode bytes at RIP 0xfffffffeffffffde. [324846.101563] RSP: 0018:ff6cad692caf7d50 EFLAGS: 00010246 [324846.102010] RAX: 0000000000000010 RBX: 0000000000000000 RCX: ff46923f2038ddd8 [324846.102478] RDX: 0000000000000000 RSI: 0000000000001000 RDI: ff46923f2038e088 [324846.102817] RBP: ff6cad692caf7da0 R08: 0000000000001000 R09: ff46923f2038e068 [324846.103166] R10: 0000000000001000 R11: 0000000000000001 R12: ff46923f2038eb40 [324846.103496] R13: 0000000000080000 R14: ff46923f2038dca0 R15: 0000000000000000 [324846.103819] FS: 0000000000000000(0000) GS:ff469434ff6c0000(0000) knlGS:0000000000000000 [324846.104151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [324846.104478] CR2: fffffffeffffffde CR3: 0000014e67210003 CR4: 0000000000773ee0 [324846.104804] PKRU: 55555554

I can't even run pveversion -v on the node... this is from another node with same versions
proxmox-ve: 7.2-1 (running kernel: 5.19.7-1-pve) pve-manager: 7.2-11 (running version: 7.2-11/b76d3178) pve-kernel-helper: 7.2-12 pve-kernel-5.19: 7.2-11 pve-kernel-5.15: 7.2-10 pve-kernel-5.4: 6.4-18 pve-kernel-5.19.7-1-pve: 5.19.7-1 pve-kernel-5.15.53-1-pve: 5.15.53-1 pve-kernel-5.15.35-2-pve: 5.15.35-5 pve-kernel-5.4.189-2-pve: 5.4.189-2 pve-kernel-5.4.34-1-pve: 5.4.34-2 ceph-fuse: 14.2.21-1 corosync: 3.1.5-pve2 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown: residual config ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.24-pve1 libproxmox-acme-perl: 1.4.2 libproxmox-backup-qemu0: 1.3.1-1 libpve-access-control: 7.2-4 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.2-2 libpve-guest-common-perl: 4.1-2 libpve-http-server-perl: 4.1-3 libpve-storage-perl: 7.2-8 libqb0: 1.0.5-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 5.0.0-3 lxcfs: 4.0.12-pve1 novnc-pve: 1.3.0-3 proxmox-backup-client: 2.2.6-1 proxmox-backup-file-restore: 2.2.6-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.5.1 pve-cluster: 7.2-2 pve-container: 4.2-2 pve-docs: 7.2-2 pve-edk2-firmware: 3.20220526-1 pve-firewall: 4.2-6 pve-firmware: 3.5-1 pve-ha-manager: 3.4.0 pve-i18n: 2.7-2 pve-qemu-kvm: 7.0.0-3 pve-xtermjs: 4.16.0-1 qemu-server: 7.2-4 smartmontools: 7.2-pve3 spiceterm: 3.2-2 swtpm: 0.7.1~bpo11+1 vncterm: 1.7-1 zfsutils-linux: 2.1.5-pve1

It was some faulty memory sticks. We removed them, and things look good so far again.

 

[Vengance]

I'm running a Ryzen 9 3900X + Asrock Rack + 128GB ECC and the system frequently crashes.
The syslog doesn't show any suspicious when the server goes down and only a hard reset helps.

Kernel version 5.19.7-2-pve is currently installed.
It doesn't seem to be hardware related as pretty much all components have been changed already.

Any ideas?

proxmox-ve: 7.2-1 (running kernel: 5.19.7-2-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-5.19: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.19.7-2-pve: 5.19.7-2
pve-kernel-5.19.7-1-pve: 5.19.7-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.60-1-pve: 5.15.60-1
pve-kernel-5.15.53-1-pve: 5.15.53-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

 

[t.lamprecht]

The syslog doesn't show any suspicious when the server goes down and only a hard reset helps.

Can you try to ssh into it and run a journalctl -f session to ensure you actually get the latest log at time of crash - often they cannot be synced to disk anymore, but still go out via a network (or, if you got one, serial console).

Without such info it's pretty hard to tell what the issue is, fwiw, we don't know of a general issue with that CPU from users and the (similar) 3700X here runs well in the workstations that got it here, albeit they don't use ECC ram. I'd also ensure that you run the newest UEFI/BIOS version.

 

[Vengance]

Can you try to ssh into it and run a journalctl -f session to ensure you actually get the latest log at time of crash - often they cannot be synced to disk anymore, but still go out via a network (or, if you got one, serial console).

Without such info it's pretty hard to tell what the issue is, fwiw, we don't know of a general issue with that CPU from users and the (similar) 3700X here runs well in the workstations that got it here, albeit they don't use ECC ram. I'd also ensure that you run the newest UEFI/BIOS version.

I can only SSH into it after a hard reset as the system becomes totally unresponsive.
I'll try to update the bios and hope that will help

 

[t.lamprecht]

I can only SSH into it after a hard reset as the system becomes totally unresponsive.

Yes, sure. Then run the journalctl and wait for the next crash and lets hope we got some additional log data exfiltrated.

 

[love4taylor]

5.19.y is already EOL

 

[ness1602]

Yes,but it will backport fixes from 6.x

 

[fiona]

5.19.y is already EOL

Our kernel is based on the one from Ubuntu and 5.19 is still supported by the Ubuntu kernel team. In fact Ubuntu 22.10 ships with 5.19 . Also, we try to backport other fixes that might be more specific to Proxmox and security fixes as needed.

 

[androme13]

I'm running a Ryzen 9 3900X + Asrock Rack + 128GB ECC and the system frequently crashes.
The syslog doesn't show any suspicious when the server goes down and only a hard reset helps.

Kernel version 5.19.7-2-pve is currently installed.
It doesn't seem to be hardware related as pretty much all components have been changed already.

Any ideas?
[/CODE]

Hi, i've somewhere same server specs as you and no problem for me. May be a PSU related problem ?
3900X + 64 ECC + asrock rack x570d4u
1 PCIE HBA+ 1 PCIE Intel x710 Nic

 

[flames]

@Vengance Ryzen and Threadripper processors have CSTATE 6 that needs to be disabled, when running Linux. Look into your BIOS/UEFI settings and find something similar like: "Power idle control" and set it to "Typical current idle" (or normal, high, something that is not an equivalent of low). You may find this settings somewhere in Power, Misc, or even in CPU settings. Different brands might name it a little different. Search how to disable CSTATE 6 for your mainboard.
Your crashes will disappear. Known issue since Kernel 5.10 or even earlier.

 

[lxrootard]

hello
I came cross this thread as I'm struggling since several days running Proxmox on my J6412 Elkhart Lake NUC.
It's almost running using pve-edge-kernel, see my posts here (debian bullseye).

Any hints or bypasses?
Thanks

 

[Vengance]

@Vengance Ryzen and Threadripper processors have CSTATE 6 that needs to be disabled, when running Linux. Look into your BIOS/UEFI settings and find something similar like: "Power idle control" and set it to "Typical current idle" (or normal, high, something that is not an equivalent of low). You may find this settings somewhere in Power, Misc, or even in CPU settings. Different brands might name it a little different. Search how to disable CSTATE 6 for your mainboard.
Your crashes will disappear. Known issue since Kernel 5.10 or even earlier.

Thank you, I think I found this option and changed it.
I really hope this will help, the system just crashed again tonight after about a week.

The bios upgrade did not help apparently

 

[Vengance]

Hi, i've somewhere same server specs as you and no problem for me. May be a PSU related problem ?
3900X + 64 ECC + asrock rack x570d4u
1 PCIE HBA+ 1 PCIE Intel x710 Nic

The PSU has been changed already so It looks like its SW related

 

[VonChair]

I run several deployments of Proxmox and in some of my production clusters, I would prefer not to update to this kernel version. Is there any news on the older kernel getting these fixes backported to it or will the older kernel be left with this bug and focus will eventually shift to 5.19 for the development?

 

[fiona]

Hi,

I run several deployments of Proxmox and in some of my production clusters, I would prefer not to update to this kernel version. Is there any news on the older kernel getting these fixes backported to it or will the older kernel be left with this bug and focus will eventually shift to 5.19 for the development?

please clarify what fixes/issues you are talking about. The issues mentioned in this thread are for the 5.19 kernel, so I'm really not sure what you mean.

 

[t.lamprecht]

Is there any news on the older kernel getting these fixes backported to it or will the older kernel be left with this bug and focus will eventually shift to 5.19 for the development?

To add on fiona's reply: We'll maintain the 5.15 kernel until end of life of the Proxmox VE 7.x series and release periodic updates.
5.19 may well be replaced with a newer one down the line, as opt-in kernel it doesn't have the same lifetime guarantees as our default stable 5.15.

 

[Neobin]

Disclaimer: The following is a wish; not (yet?) existing!

5.19 may well be replaced with a newer one down the line

pve-kernel-6.1

For those who are interested:

That then puts the Linux 6.1 stable release likely taking place either on the 4th or 11th of December.

https://www.phoronix.com/news/Linux-6.1-Likely-LTS

• https://www.phoronix.com/review/linux-60-features
• https://www.phoronix.com/review/linux-61-features

 

[bugeyedearl]

Very cool! 5.19 kernel solved my problem of live migration from intel to amd where after migration is completed the vm would hang and a reset is needed to recover.

 

[flames]

Very cool! 5.19 kernel solved my problem of live migration from intel to amd where after migration is completed the vm would hang and a reset is needed to recover.

uh, i am curious, you have kvm64 cpu on that VM, or host, or any other option?