Issue with Kernel 5.15.104-2 ?

J

jordanp123

New Member
Proxmox Subscriber
Apr 18, 2023
20
6
3
Hello,
I upgraded the kernel version this morning to 5.15.104-2 but several of my windows VM's would just refuse to boot, and would hang on the EFI screen, I tried rebooting the vm's several times but no real luck. I pinned the older kernel 5.15.102-1 and everything is working normally. Is anyone else having issues ?

proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve) pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a) pve-kernel-5.15: 7.4-1 pve-kernel-5.15.104-1-pve: 5.15.104-2 pve-kernel-5.15.102-1-pve: 5.15.102-1 pve-kernel-5.15.85-1-pve: 5.15.85-1 pve-kernel-5.15.74-1-pve: 5.15.74-1 ceph-fuse: 15.2.17-pve1 corosync: 3.1.7-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.24-pve2 libproxmox-acme-perl: 1.4.4 libproxmox-backup-qemu0: 1.3.1-1 libproxmox-rs-perl: 0.2.1 libpve-access-control: 7.4-2 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.3-4 libpve-guest-common-perl: 4.2-4 libpve-http-server-perl: 4.2-3 libpve-rs-perl: 0.7.5 libpve-storage-perl: 7.4-2 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 5.0.2-2 lxcfs: 5.0.3-pve1 novnc-pve: 1.4.0-1 proxmox-backup-client: 2.4.1-1 proxmox-backup-file-restore: 2.4.1-1 proxmox-kernel-helper: 7.4-1 proxmox-mail-forward: 0.1.1-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.6.5 pve-cluster: 7.3-3 pve-container: 4.4-3 pve-docs: 7.4-2 pve-edk2-firmware: 3.20230228-2 pve-firewall: 4.3-1 pve-firmware: 3.6-4 pve-ha-manager: 3.6.0 pve-i18n: 2.12-1 pve-qemu-kvm: 7.2.0-8 pve-xtermjs: 4.16.0-1 qemu-server: 7.4-3 smartmontools: 7.2-pve3 spiceterm: 3.2-2 swtpm: 0.8.0~bpo11+3 vncterm: 1.7-1 zfsutils-linux: 2.1.9-pve1
 
Hi,
what version(s) of Windows are you using? Can you share some of the affected VM's configuration (with qm config <ID>)?
 
All of the VM's are windows 11, ( I also went ahead and tried the new 6.2 kernel and it works fine as well, its what the server is currently running).

root@neptune:~# qm config 101
agent: 1,fstrim_cloned_disks=1
audio0: device=ich9-intel-hda,driver=spice
balloon: 8192
bios: ovmf
boot: order=sata0
cores: 2
cpu: host,flags=+ibpb;+virt-ssbd;+amd-ssbd;+hv-tlbflush;+aes
cpuunits: 200
description: * Firewall configured in ProxMox NOT in VM.%0A* Willies VM. Running windows 11 with office 2019 Pro
efidisk0: local-zfs:vm-101-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
machine: pc-q35-7.2
memory: 16384
meta: creation-qemu=7.1.0,ctime=1671541357
name: Win-ThinServer
net0: virtio=2E:4B:05:74:1C:AF,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
protection: 1
sata0: local-zfs:vm-101-disk-0,cache=writeback,discard=on,size=60G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=48d76fbb-0a05-4efc-a48e-e66402df1d80
sockets: 1
spice_enhancements: videostreaming=filter
startup: order=2,up=10,down=60
tablet: 0
tags: ThinClientVM
tpmstate0: local-zfs:vm-101-disk-2,size=4M,version=v2.0
vga: qxl2
vmgenid: c68412d0-3e8a-4e35-8ef6-75ab4c10e96d
 
Same problem here, all windows machines are not booting after a system update. Looks like a Kernel bug with Proxmox.
It hangs at the same point as shown in the screenshot above from @jordanp123

VM is "Windows Server 2022"

VM config:
Code: 
# qm config 300
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 16
cpu: host,flags=+ibpb;+virt-ssbd;+amd-ssbd;+pdpe1gb;+hv-tlbflush;+aes
efidisk0: NVME-Pool:vm-300-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
machine: pc-q35-7.2
memory: 32768
meta: creation-qemu=7.0.0,ctime=1665464685
name: Server-Win2022
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr5,queues=8
numa: 0
onboot: 1
ostype: win11
scsi0: NVME-Pool:vm-300-disk-1,aio=native,discard=on,iothread=1,size=300G
scsihw: virtio-scsi-single
smbios1: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
sockets: 1
startup: order=2
tablet: 0
tpmstate0: NVME-Pool:vm-300-disk-2,size=4M,version=v2.0
vmgenid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Booting the VM raises a kernel NULL pointer error in the Proxmox syslog:
Bash: 
Apr 20 17:33:11  pvedaemon[102819]: start VM 300: UPID::000191A3:0000B743:64415B37:qmstart:300:root@pam:
Apr 20 17:33:11  pvedaemon[4325]: <root@pam> starting task UPID::000191A3:0000B743:64415B37:qmstart:300:root@pam:
Apr 20 17:33:12  systemd[1]: Started 300.scope.
Apr 20 17:33:12  systemd-udevd[102901]: Using default interface naming scheme 'v247'.
Apr 20 17:33:12  systemd-udevd[102901]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 20 17:33:12  kernel: device tap300i0 entered promiscuous mode
Apr 20 17:33:12  kernel: vmbr5: port 2(tap300i0) entered blocking state
Apr 20 17:33:12  kernel: vmbr5: port 2(tap300i0) entered disabled state
Apr 20 17:33:12  kernel: vmbr5: port 2(tap300i0) entered blocking state
Apr 20 17:33:12  kernel: vmbr5: port 2(tap300i0) entered forwarding state
Apr 20 17:33:12  pvedaemon[4325]: <root@pam> end task UPID::000191A3:0000B743:64415B37:qmstart:300:root@pam: OK
Apr 20 17:33:12  pvedaemon[4325]: <root@pam> starting task UPID::00019237:0000B794:64415B38:vncproxy:300:root@pam:
Apr 20 17:33:12  pvedaemon[102967]: starting vnc proxy UPID::00019237:0000B794:64415B38:vncproxy:300:root@pam:
Apr 20 17:33:13  pvedaemon[4325]: <root@pam> end task UPID::00019237:0000B794:64415B38:vncproxy:300:root@pam: OK

Apr 20 17:33:23  kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
Apr 20 17:33:23  kernel: #PF: supervisor read access in kernel mode
Apr 20 17:33:23  kernel: #PF: error_code(0x0000) - not-present page
Apr 20 17:33:23  kernel: PGD 0 P4D 0
Apr 20 17:33:23  kernel: Oops: 0000 [#2] SMP NOPTI
Apr 20 17:33:23  kernel: CPU: 8 PID: 102946 Comm: CPU 0/KVM Tainted: P      D    O      5.15.104-1-pve #1
Apr 20 17:33:23  kernel: Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P4.50 10/19/2022
Apr 20 17:33:23  kernel: RIP: 0010:_find_first_bit+0x19/0x40
Apr 20 17:33:23  kernel: Code: 5d 41 5e 41 5f 5d e9 26 d9 9b 00 cc cc cc cc cc cc 49 89 f0 48 85 f6 74 28 31 c0 eb 0d 48 83 c0 40 48 83 c7 08 4c 39 c0 73 17 <48> 8b 17 48 85 d2 74 eb f3 48 0f bc d2 48 01 d0 49 39 c0 4c 0f 47
Apr 20 17:33:23  kernel: RSP: 0018:ffffbaeec159f7d8 EFLAGS: 00010246
Apr 20 17:33:23  kernel: RAX: 0000000000000000 RBX: ffffbaeec5509000 RCX: 0000000000000000
Apr 20 17:33:23  kernel: RDX: 0000000000000000 RSI: 0000000000000400 RDI: 0000000000000000
Apr 20 17:33:23  kernel: RBP: ffffbaeec159f820 R08: 0000000000000400 R09: ffff995bc0ff8728
Apr 20 17:33:23  kernel: R10: ffff996b4a6b0430 R11: 0000000000000000 R12: 0000000000000000
Apr 20 17:33:23  kernel: R13: ffff995bc0ff8728 R14: 0000000000000323 R15: 0000000000000003
Apr 20 17:33:23  kernel: FS:  00007fbfd270f700(0000) GS:ffff997a7ec00000(0000) knlGS:0000000000000000
Apr 20 17:33:23  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 20 17:33:23  kernel: CR2: 0000000000000000 CR3: 000000104e2fe000 CR4: 0000000000350ee0
Apr 20 17:33:23  kernel: Call Trace:
Apr 20 17:33:23  kernel:  <TASK>
Apr 20 17:33:23  kernel:  ? kvm_make_vcpus_request_mask+0x3d/0x130 [kvm]
Apr 20 17:33:23  kernel:  kvm_hv_flush_tlb.isra.0+0x116/0x540 [kvm]
Apr 20 17:33:23  kernel:  ? follow_page_mask+0x5f1/0x820
Apr 20 17:33:23  kernel:  ? __get_user_pages+0x18c/0x690
Apr 20 17:33:23  kernel:  ? lookup_address_in_pgd+0x3a/0x1d0
Apr 20 17:33:23  kernel:  ? kvm_page_track_is_active+0x16/0x60 [kvm]
Apr 20 17:33:23  kernel:  ? mmu_try_to_unsync_pages+0x35/0x210 [kvm]
Apr 20 17:33:23  kernel:  ? make_spte+0x165/0x3e0 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_tdp_mmu_map+0x3bd/0x6a0 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_is_reserved_pfn+0x2f/0x80 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_release_pfn_clean+0x3d/0x50 [kvm]
Apr 20 17:33:23  kernel:  ? direct_page_fault+0x543/0xbd0 [kvm]
Apr 20 17:33:23  kernel:  ? svm_get_segment+0x1e/0x110 [kvm_amd]
Apr 20 17:33:23  kernel:  kvm_hv_hypercall+0x3af/0x880 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_msr_allowed+0xb1/0xe0 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_complete_insn_gp+0x4f/0x70 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_emulate_hypercall.part.0+0x6e0/0x6e0 [kvm]
Apr 20 17:33:23  kernel:  kvm_emulate_hypercall+0x51/0x60 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_emulate_hypercall+0x51/0x60 [kvm]
Apr 20 17:33:23  kernel:  svm_invoke_exit_handler+0x74/0x1b0 [kvm_amd]
Apr 20 17:33:23  kernel:  handle_exit+0x98/0x230 [kvm_amd]
Apr 20 17:33:23  kernel:  kvm_arch_vcpu_ioctl_run+0xdd6/0x1730 [kvm]
Apr 20 17:33:23  kernel:  ? do_futex+0x147/0xb90
Apr 20 17:33:23  kernel:  ? kvm_vm_ioctl+0x304/0xfc0 [kvm]
Apr 20 17:33:23  kernel:  kvm_vcpu_ioctl+0x252/0x6b0 [kvm]
Apr 20 17:33:23  kernel:  ? exit_to_user_mode_prepare+0x37/0x1b0
Apr 20 17:33:23  kernel:  ? syscall_exit_to_user_mode+0x27/0x50
Apr 20 17:33:23  kernel:  ? __fget_files+0x86/0xc0
Apr 20 17:33:23  kernel:  __x64_sys_ioctl+0x95/0xd0
Apr 20 17:33:23  kernel:  do_syscall_64+0x5c/0xc0
Apr 20 17:33:23  kernel:  ? do_syscall_64+0x69/0xc0
Apr 20 17:33:23  kernel:  ? do_syscall_64+0x69/0xc0
Apr 20 17:33:23  kernel:  ? exc_page_fault+0x89/0x170
Apr 20 17:33:23  kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Apr 20 17:33:23  kernel: RIP: 0033:0x7fbfde3595f7
Apr 20 17:33:23  kernel: Code: 00 00 00 48 8b 05 99 c8 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 69 c8 0d 00 f7 d8 64 89 01 48
Apr 20 17:33:23  kernel: RSP: 002b:00007fbfd270a288 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Apr 20 17:33:23  kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007fbfde3595f7
Apr 20 17:33:23  kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000048
Apr 20 17:33:23  kernel: RBP: 000055d896694500 R08: 000055d8944e7240 R09: 000055d894beb980
Apr 20 17:33:23  kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Apr 20 17:33:23  kernel: R13: 000055d894bf2020 R14: 00007fbfd270a540 R15: 000055d8962d7810
Apr 20 17:33:23  kernel:  </TASK>
Apr 20 17:33:23  kernel: Modules linked in: tcp_diag inet_diag vfio_pci vfio_pci_core vfio_virqfd vfio_iommu_type1 vfio ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd snd_hda_codec_hdmi nouveau kvm snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec mxm_wmi irqbypass video drm_ttm_helper ttm drm_kms_helper crct10dif_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep snd_pcm corsair_psu cec snd_timer crypto_simd cryptd rapl ccp efi_pstore pcspkr snd wmi_bmof k10temp rc_core fb_sys_fops syscopyarea sysfillrect sysimgblt soundcore vhost_net vhost vhost_iotlb tap ib_iser mac_hid rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nct6775 hwmon_vid jc42 drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs
Apr 20 17:33:23  kernel:  blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb mlx4_ib ib_uverbs mlx4_en ib_core hid_generic usbhid hid mpt3sas raid_class xhci_pci crc32_pclmul xhci_pci_renesas i2c_piix4 scsi_transport_sas igb ahci nvme libahci xhci_hcd i2c_algo_bit r8169 mlx4_core dca realtek nvme_core wmi
Apr 20 17:33:23  kernel: CR2: 0000000000000000
Apr 20 17:33:23  kernel: ---[ end trace 39b9bac6abd10222 ]---
Apr 20 17:33:23  kernel: RIP: 0010:_find_first_bit+0x19/0x40
Apr 20 17:33:23  kernel: Code: 5d 41 5e 41 5f 5d e9 26 d9 9b 00 cc cc cc cc cc cc 49 89 f0 48 85 f6 74 28 31 c0 eb 0d 48 83 c0 40 48 83 c7 08 4c 39 c0 73 17 <48> 8b 17 48 85 d2 74 eb f3 48 0f bc d2 48 01 d0 49 39 c0 4c 0f 47
Apr 20 17:33:23  kernel: RSP: 0018:ffffbaeef1083748 EFLAGS: 00010246
Apr 20 17:33:23  kernel: RAX: 0000000000000000 RBX: ffffbaeec1ffc000 RCX: 0000000000000000
Apr 20 17:33:23  kernel: RDX: 0000000000000000 RSI: 0000000000000400 RDI: 0000000000000000
Apr 20 17:33:23  kernel: RBP: ffffbaeef1083790 R08: 0000000000000400 R09: ffff995bc0c1f728
Apr 20 17:33:23  kernel: R10: ffff99658d830430 R11: 0000000000000000 R12: 0000000000000000
Apr 20 17:33:23  kernel: R13: ffff995bc0c1f728 R14: 0000000000000323 R15: 0000000000000003
Apr 20 17:33:23  kernel: FS:  00007fbfd270f700(0000) GS:ffff997a7ec00000(0000) knlGS:0000000000000000
Apr 20 17:33:23  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 20 17:33:23  kernel: CR2: 0000000000000000 CR3: 000000104e2fe000 CR4: 0000000000350ee0

System version:
Bash: 
proxmox-ve: 7.4-1 (running kernel: 5.15.104-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-5.15.104-1-pve: 5.15.104-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

Any idea how to fix this? I'm currently stuck and cannot boot any WIndows VM at all.
 
Last edited:
  • Like
Reactions: itNGO and codgician
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom