Opt-in Linux 5.19 Kernel for Proxmox VE 7.x available

i have the kvm64 cpu setting, on 5.15 i was getting something like this in the log. after 5.19 i don't see it anymore.

[ 141.628962] ------------[ cut here ]------------
[ 141.628965] WARNING: CPU: 7 PID: 3497 at arch/x86/kvm/x86.c:10614 kvm_arch_vcpu_ioctl_run+0x1cb0/0x1df0 [kvm]
[ 141.629027] Modules linked in: veth tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter sctp ip6_udp_tunnel udp_tunnel nf_tables softdog bonding tls nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i915 drm_buddy ttm kvm drm_display_helper crct10dif_pclmul ghash_clmulni_intel cec aesni_intel rc_core crypto_simd cryptd snd_soc_rt5640 mei_hdcp drm_kms_helper mei_pxp snd_soc_rl6231 rapl i2c_algo_bit snd_soc_core fb_sys_fops mei_me syscopyarea sysfillrect intel_cstate at24 sysimgblt mei snd_compress ac97_bus pcspkr snd_pcm_dmaengine efi_pstore snd_pcm zfs(PO) snd_timer snd zunicode(PO) soundcore mac_hid zzstd(O) acpi_pad zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vfio_pci vfio_pci_core vfio_virqfd irqbypass
[ 141.629078] vfio_iommu_type1 vfio drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xhci_pci ahci crc32_pclmul r8169 i2c_i801 xhci_pci_renesas ehci_pci libahci i2c_smbus mpt3sas lpc_ich tg3 raid_class realtek xhci_hcd ehci_hcd scsi_transport_sas video
 
  • Like
Reactions: flames
After I upgraded to the 5.19 kernel, I have four network ports and can only pass through two, if I pass through three,

I got the following error on passthrough:

vifo-pci 0000:02:00:0: unable to change power state from D3cold to D0, device inaccessible
vifo-pci 0000:02:00:0: unable to change power state from D3cold to D0, device inaccessible
vifo-pci 0000:02:00:0: unable to change power state from D3cold to D0, device inaccessible


Downgrade to 5.15 and it to work！
 
fiona said:
Hi,

please clarify what fixes/issues you are talking about. The issues mentioned in this thread are for the 5.19 kernel, so I'm really not sure what you mean.
Click to expand...
I am referring to some fixes that seem to resolve issues with live migration between machines which both run Intel CPUs. I have machines running Intel E5-2660 v2 CPUs and some other machines in my stack running Intel Xeon Silver 4110 CPUs and when live migrating between these hosts, VMs running the KVM64 CPU type will have an issue where the guest agent becomes unresponsive and the VM crashes to a black screen and fails to respond over console. It can only be stopped or reset with qm stop <vmid> or qm reset <vmid> commands to resolve the issue.

Reported here: https://forum.proxmox.com/threads/virtual-machines-freeze-just-after-a-live-migration.109796/

There was another post I found reporting this fixing the live migration issue I'm having, but I can't seem to find it right now. I'll try to find it in my browser history and post it if I can.
 
VonChair said:
I am referring to some fixes that seem to resolve issues with live migration between machines which both run Intel CPUs. I have machines running Intel E5-2660 v2 CPUs and some other machines in my stack running Intel Xeon Silver 4110 CPUs and when live migrating between these hosts, VMs running the KVM64 CPU type will have an issue where the guest agent becomes unresponsive and the VM crashes to a black screen and fails to respond over console. It can only be stopped or reset with qm stop <vmid> or qm reset <vmid> commands to resolve the issue.

Reported here: https://forum.proxmox.com/threads/virtual-machines-freeze-just-after-a-live-migration.109796/

There was another post I found reporting this fixing the live migration issue I'm having, but I can't seem to find it right now. I'll try to find it in my browser history and post it if I can.
Click to expand...
Assuming this is the same as the FPU-related migration issue mentioned here, the fix is unlikely to land in 5.15. While Thomas made an initial backport of a proposed fix, it's a hairy area of the kernel and IIRC it still had some issues with migration from an older kernel to the kernel with the fix and wasn't considered safe enough to roll out. Its recommended to upgrade to 5.19 if you are affected by this bug.

@t.lamprecht please correct me if I'm wrong :)
 
  • Like
Reactions: VonChair
Hi

We plan to upgrade all of our ceph nodes to kernel-5.19 but we've hit a roadblock. After booting into kernel-5.19.7-2-pve only one out of two NVMe controllers on our Intel DC P4608 (SSDPECKE064T701) are available. This is one PCIe card that have dual controllers. During boot there is an error stating "globally duplicate IDs for nsid 1":

Bash: 
root@hk-cephnode-54:~# dmesg | grep nvme
[    2.825268] nvme nvme0: pci function 0000:18:00.0
[    2.827741] nvme nvme1: pci function 0000:64:00.0
[    2.827826] nvme nvme2: pci function 0000:65:00.0
[    2.829989] nvme nvme0: 31/0/0 default/read/poll queues
[    2.831756] nvme nvme1: Shutdown timeout set to 10 seconds
[    2.831813] nvme nvme2: Shutdown timeout set to 10 seconds
[    2.835493] nvme nvme1: 40/0/0 default/read/poll queues
[    2.835589] nvme nvme2: 40/0/0 default/read/poll queues
[    2.841982] nvme nvme2: globally duplicate IDs for nsid 1
[    2.841998] nvme nvme2: VID:DID 8086:0a54 model:7335943:ICDPC5ED2ORA6.4T firmware:QDV1RF30

root@hk-cephnode-54:~# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev 
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     CVFT7362001S800MGN   MT0800KEXUU                              1         800.17  GB / 800.17  GB    512   B +  0 B   4IFDHPK3
/dev/nvme1n1     PHLE747102LA6P4BGN-1 7335943:ICDPC5ED2ORA6.4T                 1           3.20  TB /   3.20  TB    512   B +  0 B   QDV1RF30

(The MT0800KEXUU (nvme0) is a Intel DC P3700 800GB and is working as expected)

After rebooting into previous kernel-5.13.19-6-pve or kernel-5.15.39-4-pve the second NVMe controller comes back and is functioning properly:

Bash: 
root@hk-cephnode-54:~# dmesg | grep nvme
[    2.557224] nvme nvme0: pci function 0000:18:00.0
[    2.563497] nvme nvme0: 31/0/0 default/read/poll queues
[    2.567233] nvme nvme1: pci function 0000:64:00.0
[    2.567429] nvme nvme2: pci function 0000:65:00.0
[    2.572168] nvme nvme2: Shutdown timeout set to 10 seconds
[    2.575618] nvme nvme2: 40/0/0 default/read/poll queues
[    2.581091] nvme nvme1: Shutdown timeout set to 10 seconds
[    2.584637] nvme nvme1: 40/0/0 default/read/poll queues

root@hk-cephnode-54:~# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev 
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     CVFT7362001S800MGN   MT0800KEXUU                              1         800.17  GB / 800.17  GB    512   B +  0 B   4IFDHPK3
/dev/nvme1n1     PHLE747102LA6P4BGN-1 7335943:ICDPC5ED2ORA6.4T                 1           3.20  TB /   3.20  TB    512   B +  0 B   QDV1RF30
/dev/nvme2n1     PHLE747102LA6P4BGN-2 7335943:ICDPC5ED2ORA6.4T                 1           3.20  TB /   3.20  TB    512   B +  0 B   QDV1RF30

root@hk-cephnode-54:~# lspci -nn|grep NVM
64:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]
65:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]


Any idea on how to get both NVMe controllers active in kernel-5.19?
 
https://bugzilla.kernel.org/show_bug.cgi?id=216049

it seems upstream has applied quite a few quirks for various vendor models that exhibit this broken behaviour, so likely we need to cherry-pick one of those (or wait for them to trickle down from upstream)..

edit: seems like for this particular model such a patch hasn't been written yet.. maybe you could add your model to the linked bug tracker?
 
Last edited:
fabian said:
https://bugzilla.kernel.org/show_bug.cgi?id=216049

it seems upstream has applied quite a few quirks for various vendor models that exhibit this broken behaviour, so likely we need to cherry-pick one of those (or wait for them to trickle down from upstream)..

edit: seems like for this particular model such a patch hasn't been written yet.. maybe you could add your model to the linked bug tracker?
Click to expand...

Yes, I believe the problem lies within here:
https://git.kernel.org/pub/scm/linux/kernel/git/srini/nvmem.git/tree/drivers/nvme/host/pci.c#n3406

And it looks like there have been work to mitigate the issue but it dosen't look like these changes are imminent to be merged upstream since there is no indication whether this change break other models in the same P4500/P4600 series that share the same PCI dev ID...

https://80x24.org/lore/all/CAF9DR02.../T/#mbd44365722d95b533d1d81afe77034e250d5573d

I will add my model to the bug tracker, hopefully you will cherry-pick this quirk with your next pve-kernel-5.19 release :)
 
The kernel is your smallest problem w.r.t. Venus, needs newer mesa, qemu, ... Once available we'll include and announce it just like we did with its predecessor VirGL.
 
t.lamprecht said:
The kernel is your smallest problem w.r.t. Venus, needs newer mesa, qemu, ... Once available we'll include and announce it just like we did with its predecessor VirGL.
Click to expand...
Amazing, thank you very much! Is this commit already upstream in Mesa? Will this only be made available with a future proxmox relese or could a Update of the system with a current installation use this feature as well?
 
Depends on the level of involvement and getting your hands dirty you're comfortable with. Quite some stuff is upstream, and you'd also need a newer virglrender package (which also hosts some venus stuff), but most of that needs re-compiling a not so small set of packages including newer dependencies, if you don't care for clean updatable solution you may cut some corners on using debian packages (but IMO not that much to be worth it) and may also need to go for unreleased mesa/virglrender from-git compilation - I did not tried it end to end myself for venus, so I cannot give a definitve answer, but I'd guesstimate: possible -> yes, but hard and/or lots of work.
 
Hello
I've used this kernel for a week before my debian 11 VM crashed, here's the console stack trace.

pve-manager/7.2-11/b76d3178 (running kernel: 6.0.6-edge)
Hardware is an Intel 6412 NUC with a 3165AC M.2 wifi card

Thanks

jeedom crash.png

Any hint?
 
Secure Boot not working on official 5.19 but working on official 5.15 (in an experiment environment, PVE in PVE's OVMF VM).

After signing and enabling Secure Boot, choose the kernel entry in Grub, the 5.15 kernel prints "EFI stub: UEFI Secure Boot is enabled." and starts booting but 5.19 kernel hang after printing it.

1668834350621.png

Ubuntu's 5.19.0-24 kernel image doesn't have this problem. I'm trying to check the kernel config and build the pve-kernel myself.

Update: I believe it's a kernel config issue. With Ubuntu's 5.19.0-24's kernel source (without PVE patches) and PVE's config it hangs but with Ubuntu's original config it boots.
 
Last edited:
  • Like
Reactions: eXtremeSHOk
Hi!
Migration failed - with kernel 5.19 migration still doesn't work.

Environment:
HOST OLD: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (old)
HOST NEW: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (new ;-))

Kernel (both): PVE 5.19.7-2
GUEST: VM - CPU generic kvm64

Migration from NEW to OLD works like a harm.
Migration from OLD to NEW freezes VM. All CPUs 100% utilization.
Console:
PANIC: double fault, error_code: 0x0
Click to expand...

On NEW host first migration after reboot - dmesg output:

[Sun Nov 20 16:44:33 2022] ------------[ cut here ]------------
[Sun Nov 20 16:44:33 2022] WARNING: CPU: 39 PID: 45957 at arch/x86/kvm/x86.c:10614 kvm_arch_vcpu_ioctl_run+0x1cb0/0x1df0 [kvm]
[Sun Nov 20 16:44:33 2022] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs xfs ebtable_filter ebtables ip6table_raw ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables iptable_raw xt_mac ipt_REJECT nf_reject_ipv4 xt_physdev xt_addrtype xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_multiport xt_comment xt_set xt_tcpudp xt_mark iptable_filter bpfilter ip_set_hash_net ip_set sctp ip6_udp_tunnel udp_tunnel nf_tables softdog nfnetlink_log nfnetlink ipmi_ssif intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul ghash_clmulni_intel aesni_intel mgag200 drm_shmem_helper crypto_simd drm_kms_helper cryptd i2c_algo_bit rapl fb_sys_fops cdc_ether ipmi_si mei_me syscopyarea usbnet ipmi_devintf sysfillrect intel_cstate pcspkr efi_pstore mxm_wmi sysimgblt mii mei ipmi_msghandler mac_hid acpi_power_meter acpi_pad zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO)
[Sun Nov 20 16:44:33 2022] icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c crc32_pclmul xhci_pci i2c_i801 xhci_pci_renesas i2c_smbus lpc_ich ehci_pci tg3 xhci_hcd ehci_hcd megaraid_sas wmi
[Sun Nov 20 16:44:33 2022] CPU: 39 PID: 45957 Comm: CPU 1/KVM Tainted: P O 5.19.7-2-pve #1
[Sun Nov 20 16:44:33 2022] Hardware name: LENOVO System x3650 M5: -[8871AC1]-/01GR174, BIOS -[TCE136H-2.70]- 06/13/2018
[Sun Nov 20 16:44:33 2022] RIP: 0010:kvm_arch_vcpu_ioctl_run+0x1cb0/0x1df0 [kvm]
[Sun Nov 20 16:44:33 2022] Code: 48 8b 78 08 4c 89 e6 e8 be a9 fc ff 65 ff 0d b7 0f d3 3e 0f 85 a2 ec ff ff 0f 1f 44 00 00 e9 d0 e9 ff ff 0f 0b e9 4f ed ff ff <0f> 0b e9 2c ed ff ff 0f 1f 44 00 00 e9 23 f5 ff ff 48 89 de 4c 89
[Sun Nov 20 16:44:33 2022] RSP: 0018:ffffb05f9386bd10 EFLAGS: 00010202
[Sun Nov 20 16:44:33 2022] RAX: 0000000000000001 RBX: 000000000000ae80 RCX: 0000000000000000
[Sun Nov 20 16:44:33 2022] RDX: 000044457ee1b000 RSI: 00000000fffffe01 RDI: ffff8beb2cb4c000
[Sun Nov 20 16:44:33 2022] RBP: ffffb05f9386bdb0 R08: 0000000000000000 R09: 0000000000000027
[Sun Nov 20 16:44:33 2022] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8beb2cb4c000
[Sun Nov 20 16:44:33 2022] R13: ffff8beb32e67000 R14: ffff8beb2cb4c048 R15: ffff8bea91429400
[Sun Nov 20 16:44:33 2022] FS: 00007f1980ff9700(0000) GS:ffff8c19bfcc0000(0000) knlGS:0000000000000000
[Sun Nov 20 16:44:33 2022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sun Nov 20 16:44:33 2022] CR2: 00005592bb523360 CR3: 0000003143948005 CR4: 00000000003726e0
[Sun Nov 20 16:44:33 2022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[Sun Nov 20 16:44:33 2022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[Sun Nov 20 16:44:33 2022] Call Trace:
[Sun Nov 20 16:44:33 2022] <TASK>
[Sun Nov 20 16:44:33 2022] kvm_vcpu_ioctl+0x24f/0x6d0 [kvm]
[Sun Nov 20 16:44:33 2022] ? __x64_sys_futex+0x81/0x1d0
[Sun Nov 20 16:44:33 2022] ? __fget_light.part.0+0x8c/0xd0
[Sun Nov 20 16:44:33 2022] ? __fget_light.part.0+0x8c/0xd0
[Sun Nov 20 16:44:33 2022] __x64_sys_ioctl+0x95/0xd0
[Sun Nov 20 16:44:33 2022] do_syscall_64+0x5c/0x90
[Sun Nov 20 16:44:33 2022] ? exit_to_user_mode_prepare+0x37/0x180
[Sun Nov 20 16:44:33 2022] ? irqentry_exit_to_user_mode+0x9/0x20
[Sun Nov 20 16:44:33 2022] ? irqentry_exit+0x3b/0x50
[Sun Nov 20 16:44:33 2022] ? exc_page_fault+0x87/0x180
[Sun Nov 20 16:44:33 2022] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[Sun Nov 20 16:44:33 2022] RIP: 0033:0x7f19f786b6b7
[Sun Nov 20 16:44:33 2022] Code: 00 00 00 48 8b 05 d9 c7 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a9 c7 0d 00 f7 d8 64 89 01 48
[Sun Nov 20 16:44:33 2022] RSP: 002b:00007f1980ff4288 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Sun Nov 20 16:44:33 2022] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f19f786b6b7
[Sun Nov 20 16:44:33 2022] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000042
[Sun Nov 20 16:44:33 2022] RBP: 00005592bbd5a650 R08: 00005592b9f29250 R09: 00005592ba608ec0
[Sun Nov 20 16:44:33 2022] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Sun Nov 20 16:44:33 2022] R13: 00005592ba60e140 R14: 00007f1980ff4540 R15: 0000000000802000
[Sun Nov 20 16:44:33 2022] </TASK>
[Sun Nov 20 16:44:33 2022] ---[ end trace 0000000000000000 ]---
Click to expand...
UPDATE:
I've downgraded kernel on both hosts to PVE 5.13.19-15 and migration works in both direction.
Of course on 5.15 problem exists (that's why I've tested 5.19)
 
Last edited:
QM Remote Migrate does work with 5.19 and AMD Epyc. Had some problems with 5.15 (VMs migrated with qm remote-migrate did freeze and needed a stop). After switching to 5.19 live remote-migrate works well (everythings online and responsiv no need to stop vm after successful migration).
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back