Issue with Kernel 5.15.104-2 ?

J

jordanp123

Member
Apr 18, 2023
24
7
8
Hello,
I upgraded the kernel version this morning to 5.15.104-2 but several of my windows VM's would just refuse to boot, and would hang on the EFI screen, I tried rebooting the vm's several times but no real luck. I pinned the older kernel 5.15.102-1 and everything is working normally. Is anyone else having issues ?

proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve) pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a) pve-kernel-5.15: 7.4-1 pve-kernel-5.15.104-1-pve: 5.15.104-2 pve-kernel-5.15.102-1-pve: 5.15.102-1 pve-kernel-5.15.85-1-pve: 5.15.85-1 pve-kernel-5.15.74-1-pve: 5.15.74-1 ceph-fuse: 15.2.17-pve1 corosync: 3.1.7-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.24-pve2 libproxmox-acme-perl: 1.4.4 libproxmox-backup-qemu0: 1.3.1-1 libproxmox-rs-perl: 0.2.1 libpve-access-control: 7.4-2 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.3-4 libpve-guest-common-perl: 4.2-4 libpve-http-server-perl: 4.2-3 libpve-rs-perl: 0.7.5 libpve-storage-perl: 7.4-2 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 5.0.2-2 lxcfs: 5.0.3-pve1 novnc-pve: 1.4.0-1 proxmox-backup-client: 2.4.1-1 proxmox-backup-file-restore: 2.4.1-1 proxmox-kernel-helper: 7.4-1 proxmox-mail-forward: 0.1.1-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.6.5 pve-cluster: 7.3-3 pve-container: 4.4-3 pve-docs: 7.4-2 pve-edk2-firmware: 3.20230228-2 pve-firewall: 4.3-1 pve-firmware: 3.6-4 pve-ha-manager: 3.6.0 pve-i18n: 2.12-1 pve-qemu-kvm: 7.2.0-8 pve-xtermjs: 4.16.0-1 qemu-server: 7.4-3 smartmontools: 7.2-pve3 spiceterm: 3.2-2 swtpm: 0.8.0~bpo11+3 vncterm: 1.7-1 zfsutils-linux: 2.1.9-pve1
 
Screenshot 2023-04-18 at 1.14.44 PM.png
This is as far as any of my windows VM's ever get, on the newer kernel, when I pin the older kernel they all boot with no issues.
 
Hi,
what version(s) of Windows are you using? Can you share some of the affected VM's configuration (with qm config <ID>)?
 
All of the VM's are windows 11, ( I also went ahead and tried the new 6.2 kernel and it works fine as well, its what the server is currently running).

root@neptune:~# qm config 101
agent: 1,fstrim_cloned_disks=1
audio0: device=ich9-intel-hda,driver=spice
balloon: 8192
bios: ovmf
boot: order=sata0
cores: 2
cpu: host,flags=+ibpb;+virt-ssbd;+amd-ssbd;+hv-tlbflush;+aes
cpuunits: 200
description: * Firewall configured in ProxMox NOT in VM.%0A* Willies VM. Running windows 11 with office 2019 Pro
efidisk0: local-zfs:vm-101-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
machine: pc-q35-7.2
memory: 16384
meta: creation-qemu=7.1.0,ctime=1671541357
name: Win-ThinServer
net0: virtio=2E:4B:05:74:1C:AF,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
protection: 1
sata0: local-zfs:vm-101-disk-0,cache=writeback,discard=on,size=60G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=48d76fbb-0a05-4efc-a48e-e66402df1d80
sockets: 1
spice_enhancements: videostreaming=filter
startup: order=2,up=10,down=60
tablet: 0
tags: ThinClientVM
tpmstate0: local-zfs:vm-101-disk-2,size=4M,version=v2.0
vga: qxl2
vmgenid: c68412d0-3e8a-4e35-8ef6-75ab4c10e96d
 
Same problem here, all windows machines are not booting after a system update. Looks like a Kernel bug with Proxmox.
It hangs at the same point as shown in the screenshot above from @jordanp123

VM is "Windows Server 2022"

VM config:
Code: 
# qm config 300
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 16
cpu: host,flags=+ibpb;+virt-ssbd;+amd-ssbd;+pdpe1gb;+hv-tlbflush;+aes
efidisk0: NVME-Pool:vm-300-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
machine: pc-q35-7.2
memory: 32768
meta: creation-qemu=7.0.0,ctime=1665464685
name: Server-Win2022
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr5,queues=8
numa: 0
onboot: 1
ostype: win11
scsi0: NVME-Pool:vm-300-disk-1,aio=native,discard=on,iothread=1,size=300G
scsihw: virtio-scsi-single
smbios1: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
sockets: 1
startup: order=2
tablet: 0
tpmstate0: NVME-Pool:vm-300-disk-2,size=4M,version=v2.0
vmgenid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Booting the VM raises a kernel NULL pointer error in the Proxmox syslog:
Bash: 
Apr 20 17:33:11  pvedaemon[102819]: start VM 300: UPID::000191A3:0000B743:64415B37:qmstart:300:root@pam:
Apr 20 17:33:11  pvedaemon[4325]: <root@pam> starting task UPID::000191A3:0000B743:64415B37:qmstart:300:root@pam:
Apr 20 17:33:12  systemd[1]: Started 300.scope.
Apr 20 17:33:12  systemd-udevd[102901]: Using default interface naming scheme 'v247'.
Apr 20 17:33:12  systemd-udevd[102901]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 20 17:33:12  kernel: device tap300i0 entered promiscuous mode
Apr 20 17:33:12  kernel: vmbr5: port 2(tap300i0) entered blocking state
Apr 20 17:33:12  kernel: vmbr5: port 2(tap300i0) entered disabled state
Apr 20 17:33:12  kernel: vmbr5: port 2(tap300i0) entered blocking state
Apr 20 17:33:12  kernel: vmbr5: port 2(tap300i0) entered forwarding state
Apr 20 17:33:12  pvedaemon[4325]: <root@pam> end task UPID::000191A3:0000B743:64415B37:qmstart:300:root@pam: OK
Apr 20 17:33:12  pvedaemon[4325]: <root@pam> starting task UPID::00019237:0000B794:64415B38:vncproxy:300:root@pam:
Apr 20 17:33:12  pvedaemon[102967]: starting vnc proxy UPID::00019237:0000B794:64415B38:vncproxy:300:root@pam:
Apr 20 17:33:13  pvedaemon[4325]: <root@pam> end task UPID::00019237:0000B794:64415B38:vncproxy:300:root@pam: OK

Apr 20 17:33:23  kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
Apr 20 17:33:23  kernel: #PF: supervisor read access in kernel mode
Apr 20 17:33:23  kernel: #PF: error_code(0x0000) - not-present page
Apr 20 17:33:23  kernel: PGD 0 P4D 0
Apr 20 17:33:23  kernel: Oops: 0000 [#2] SMP NOPTI
Apr 20 17:33:23  kernel: CPU: 8 PID: 102946 Comm: CPU 0/KVM Tainted: P      D    O      5.15.104-1-pve #1
Apr 20 17:33:23  kernel: Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P4.50 10/19/2022
Apr 20 17:33:23  kernel: RIP: 0010:_find_first_bit+0x19/0x40
Apr 20 17:33:23  kernel: Code: 5d 41 5e 41 5f 5d e9 26 d9 9b 00 cc cc cc cc cc cc 49 89 f0 48 85 f6 74 28 31 c0 eb 0d 48 83 c0 40 48 83 c7 08 4c 39 c0 73 17 <48> 8b 17 48 85 d2 74 eb f3 48 0f bc d2 48 01 d0 49 39 c0 4c 0f 47
Apr 20 17:33:23  kernel: RSP: 0018:ffffbaeec159f7d8 EFLAGS: 00010246
Apr 20 17:33:23  kernel: RAX: 0000000000000000 RBX: ffffbaeec5509000 RCX: 0000000000000000
Apr 20 17:33:23  kernel: RDX: 0000000000000000 RSI: 0000000000000400 RDI: 0000000000000000
Apr 20 17:33:23  kernel: RBP: ffffbaeec159f820 R08: 0000000000000400 R09: ffff995bc0ff8728
Apr 20 17:33:23  kernel: R10: ffff996b4a6b0430 R11: 0000000000000000 R12: 0000000000000000
Apr 20 17:33:23  kernel: R13: ffff995bc0ff8728 R14: 0000000000000323 R15: 0000000000000003
Apr 20 17:33:23  kernel: FS:  00007fbfd270f700(0000) GS:ffff997a7ec00000(0000) knlGS:0000000000000000
Apr 20 17:33:23  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 20 17:33:23  kernel: CR2: 0000000000000000 CR3: 000000104e2fe000 CR4: 0000000000350ee0
Apr 20 17:33:23  kernel: Call Trace:
Apr 20 17:33:23  kernel:  <TASK>
Apr 20 17:33:23  kernel:  ? kvm_make_vcpus_request_mask+0x3d/0x130 [kvm]
Apr 20 17:33:23  kernel:  kvm_hv_flush_tlb.isra.0+0x116/0x540 [kvm]
Apr 20 17:33:23  kernel:  ? follow_page_mask+0x5f1/0x820
Apr 20 17:33:23  kernel:  ? __get_user_pages+0x18c/0x690
Apr 20 17:33:23  kernel:  ? lookup_address_in_pgd+0x3a/0x1d0
Apr 20 17:33:23  kernel:  ? kvm_page_track_is_active+0x16/0x60 [kvm]
Apr 20 17:33:23  kernel:  ? mmu_try_to_unsync_pages+0x35/0x210 [kvm]
Apr 20 17:33:23  kernel:  ? make_spte+0x165/0x3e0 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_tdp_mmu_map+0x3bd/0x6a0 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_is_reserved_pfn+0x2f/0x80 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_release_pfn_clean+0x3d/0x50 [kvm]
Apr 20 17:33:23  kernel:  ? direct_page_fault+0x543/0xbd0 [kvm]
Apr 20 17:33:23  kernel:  ? svm_get_segment+0x1e/0x110 [kvm_amd]
Apr 20 17:33:23  kernel:  kvm_hv_hypercall+0x3af/0x880 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_msr_allowed+0xb1/0xe0 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_complete_insn_gp+0x4f/0x70 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_emulate_hypercall.part.0+0x6e0/0x6e0 [kvm]
Apr 20 17:33:23  kernel:  kvm_emulate_hypercall+0x51/0x60 [kvm]
Apr 20 17:33:23  kernel:  ? kvm_emulate_hypercall+0x51/0x60 [kvm]
Apr 20 17:33:23  kernel:  svm_invoke_exit_handler+0x74/0x1b0 [kvm_amd]
Apr 20 17:33:23  kernel:  handle_exit+0x98/0x230 [kvm_amd]
Apr 20 17:33:23  kernel:  kvm_arch_vcpu_ioctl_run+0xdd6/0x1730 [kvm]
Apr 20 17:33:23  kernel:  ? do_futex+0x147/0xb90
Apr 20 17:33:23  kernel:  ? kvm_vm_ioctl+0x304/0xfc0 [kvm]
Apr 20 17:33:23  kernel:  kvm_vcpu_ioctl+0x252/0x6b0 [kvm]
Apr 20 17:33:23  kernel:  ? exit_to_user_mode_prepare+0x37/0x1b0
Apr 20 17:33:23  kernel:  ? syscall_exit_to_user_mode+0x27/0x50
Apr 20 17:33:23  kernel:  ? __fget_files+0x86/0xc0
Apr 20 17:33:23  kernel:  __x64_sys_ioctl+0x95/0xd0
Apr 20 17:33:23  kernel:  do_syscall_64+0x5c/0xc0
Apr 20 17:33:23  kernel:  ? do_syscall_64+0x69/0xc0
Apr 20 17:33:23  kernel:  ? do_syscall_64+0x69/0xc0
Apr 20 17:33:23  kernel:  ? exc_page_fault+0x89/0x170
Apr 20 17:33:23  kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Apr 20 17:33:23  kernel: RIP: 0033:0x7fbfde3595f7
Apr 20 17:33:23  kernel: Code: 00 00 00 48 8b 05 99 c8 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 69 c8 0d 00 f7 d8 64 89 01 48
Apr 20 17:33:23  kernel: RSP: 002b:00007fbfd270a288 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Apr 20 17:33:23  kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007fbfde3595f7
Apr 20 17:33:23  kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000048
Apr 20 17:33:23  kernel: RBP: 000055d896694500 R08: 000055d8944e7240 R09: 000055d894beb980
Apr 20 17:33:23  kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Apr 20 17:33:23  kernel: R13: 000055d894bf2020 R14: 00007fbfd270a540 R15: 000055d8962d7810
Apr 20 17:33:23  kernel:  </TASK>
Apr 20 17:33:23  kernel: Modules linked in: tcp_diag inet_diag vfio_pci vfio_pci_core vfio_virqfd vfio_iommu_type1 vfio ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd snd_hda_codec_hdmi nouveau kvm snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec mxm_wmi irqbypass video drm_ttm_helper ttm drm_kms_helper crct10dif_pclmul snd_hda_core ghash_clmulni_intel aesni_intel snd_hwdep snd_pcm corsair_psu cec snd_timer crypto_simd cryptd rapl ccp efi_pstore pcspkr snd wmi_bmof k10temp rc_core fb_sys_fops syscopyarea sysfillrect sysimgblt soundcore vhost_net vhost vhost_iotlb tap ib_iser mac_hid rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nct6775 hwmon_vid jc42 drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs
Apr 20 17:33:23  kernel:  blake2b_generic xor zstd_compress raid6_pq libcrc32c simplefb mlx4_ib ib_uverbs mlx4_en ib_core hid_generic usbhid hid mpt3sas raid_class xhci_pci crc32_pclmul xhci_pci_renesas i2c_piix4 scsi_transport_sas igb ahci nvme libahci xhci_hcd i2c_algo_bit r8169 mlx4_core dca realtek nvme_core wmi
Apr 20 17:33:23  kernel: CR2: 0000000000000000
Apr 20 17:33:23  kernel: ---[ end trace 39b9bac6abd10222 ]---
Apr 20 17:33:23  kernel: RIP: 0010:_find_first_bit+0x19/0x40
Apr 20 17:33:23  kernel: Code: 5d 41 5e 41 5f 5d e9 26 d9 9b 00 cc cc cc cc cc cc 49 89 f0 48 85 f6 74 28 31 c0 eb 0d 48 83 c0 40 48 83 c7 08 4c 39 c0 73 17 <48> 8b 17 48 85 d2 74 eb f3 48 0f bc d2 48 01 d0 49 39 c0 4c 0f 47
Apr 20 17:33:23  kernel: RSP: 0018:ffffbaeef1083748 EFLAGS: 00010246
Apr 20 17:33:23  kernel: RAX: 0000000000000000 RBX: ffffbaeec1ffc000 RCX: 0000000000000000
Apr 20 17:33:23  kernel: RDX: 0000000000000000 RSI: 0000000000000400 RDI: 0000000000000000
Apr 20 17:33:23  kernel: RBP: ffffbaeef1083790 R08: 0000000000000400 R09: ffff995bc0c1f728
Apr 20 17:33:23  kernel: R10: ffff99658d830430 R11: 0000000000000000 R12: 0000000000000000
Apr 20 17:33:23  kernel: R13: ffff995bc0c1f728 R14: 0000000000000323 R15: 0000000000000003
Apr 20 17:33:23  kernel: FS:  00007fbfd270f700(0000) GS:ffff997a7ec00000(0000) knlGS:0000000000000000
Apr 20 17:33:23  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 20 17:33:23  kernel: CR2: 0000000000000000 CR3: 000000104e2fe000 CR4: 0000000000350ee0

System version:
Bash: 
proxmox-ve: 7.4-1 (running kernel: 5.15.104-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-5.15.104-1-pve: 5.15.104-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

Any idea how to fix this? I'm currently stuck and cannot boot any WIndows VM at all.
 
Last edited:
  • Like
Reactions: itNGO and codgician
You must log in or register to reply here.
Top Bottom
Back