Kernel Bug: Unable to start Windows VM

[Phiolin]

Sometime last year I had set up a Windows VM on my Proxmox host. It was only occasionally used and worked fine until some time ago.
Now every time I try to boot that VM, syslog shows a kernel bug and the VM fails to start. I tried setting up a completely new Win10 VM from scratch, but that encounters the same issue. I don't even get to the Windows installer, it fails to start on the very first boot with just basic settings.

Need some help here to figure out what's wrong. It only happens for Windows VMs. Linux VMs and LXC containers work perfectly fine.

VM configuration:

agent: 1
balloon: 0
boot: cdn
bootdisk: scsi0
cores: 2
cpu: host
ide2: local:iso/Win10_1909_German_x64.iso,media=cdrom
ide3: local:iso/virtio-win.iso,media=cdrom,size=363020K
memory: 2048
name: winnie
net0: virtio=6A:BA:CE:00:AD:4D,bridge=vmbr1,firewall=1,tag=10
numa: 0
ostype: win10
scsi0: tank2:vm-118-disk-0,cache=writeback,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=80d4c2e5-a83d-49d3-95b4-5146cdbdfed2
sockets: 1
vmgenid: cdeb53f7-e469-48e0-93e5-eae6d56626a1

Kernel Bug output:

Jan 27 11:41:07 pve kernel: [15734.536481] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
Jan 27 11:41:07 pve kernel: [15734.536487] #PF error: [INSTR]
Jan 27 11:41:07 pve kernel: [15734.536489] PGD 0 P4D 0
Jan 27 11:41:07 pve kernel: [15734.536491] Oops: 0010 [#2] SMP PTI
Jan 27 11:41:07 pve kernel: [15734.536493] CPU: 1 PID: 24975 Comm: kvm Tainted: P      D    O      5.0.21-3-pve #1
Jan 27 11:41:07 pve kernel: [15734.536495] Hardware name: Shuttle Inc. SZ270/FZ270, BIOS 1.11 07/20/2018
Jan 27 11:41:07 pve kernel: [15734.536496] RIP: 0010:          (null)
Jan 27 11:41:07 pve kernel: [15734.536499] Code: Bad RIP value.
Jan 27 11:41:07 pve kernel: [15734.536500] RSP: 0018:ffffb596a6333b88 EFLAGS: 00010246
Jan 27 11:41:07 pve kernel: [15734.536501] RAX: 0000000000000000 RBX: 00007ff44b00c008 RCX: 0000000000000000
Jan 27 11:41:07 pve kernel: [15734.536503] RDX: 00007ff44b00c008 RSI: ffffb596a6333d08 RDI: ffff9eda09840000
Jan 27 11:41:07 pve kernel: [15734.536504] RBP: ffffb596a6333cc8 R08: 0000000000000007 R09: 0000000000000019
Jan 27 11:41:07 pve kernel: [15734.536505] R10: ffff9ed4187d3c00 R11: 0000000000000000 R12: ffffb596a6333b90
Jan 27 11:41:07 pve kernel: [15734.536506] R13: ffffb596a6333d08 R14: ffff9eda09840058 R15: 0000000000000000
Jan 27 11:41:07 pve kernel: [15734.536507] FS:  00007ff44c57e700(0000) GS:ffff9edacea80000(0000) knlGS:0000000000000000
Jan 27 11:41:07 pve kernel: [15734.536509] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 27 11:41:07 pve kernel: [15734.536510] CR2: ffffffffffffffd6 CR3: 00000001abf98006 CR4: 00000000003626e0
Jan 27 11:41:07 pve kernel: [15734.536511] Call Trace:
Jan 27 11:41:07 pve kernel: [15734.536527]  kvm_vcpu_ioctl_get_hv_cpuid+0x44/0x220 [kvm]
Jan 27 11:41:07 pve kernel: [15734.536530]  ? _cond_resched+0x19/0x30
Jan 27 11:41:07 pve kernel: [15734.536534]  ? vmx_vcpu_load+0x22d/0x570 [kvm_intel]
Jan 27 11:41:07 pve kernel: [15734.536535]  ? _cond_resched+0x19/0x30
Jan 27 11:41:07 pve kernel: [15734.536538]  ? apparmor_file_alloc_security+0x3e/0x170
Jan 27 11:41:07 pve kernel: [15734.536540]  ? get_page_from_freelist+0xf55/0x1440
Jan 27 11:41:07 pve kernel: [15734.536550]  ? kvm_arch_vcpu_load+0x94/0x290 [kvm]
Jan 27 11:41:07 pve kernel: [15734.536553]  ? vmx_vcpu_put+0x1a/0x20 [kvm_intel]
Jan 27 11:41:07 pve kernel: [15734.536561]  kvm_arch_vcpu_ioctl+0x14b/0x11f0 [kvm]
Jan 27 11:41:07 pve kernel: [15734.536563]  ? __alloc_pages_nodemask+0x13f/0x2e0
Jan 27 11:41:07 pve kernel: [15734.536565]  ? mem_cgroup_commit_charge+0x82/0x4d0
Jan 27 11:41:07 pve kernel: [15734.536567]  ? mem_cgroup_try_charge+0x8b/0x190
Jan 27 11:41:07 pve kernel: [15734.536568]  ? mem_cgroup_throttle_swaprate+0x2c/0x154
Jan 27 11:41:07 pve kernel: [15734.536575]  kvm_vcpu_ioctl+0xe5/0x610 [kvm]
Jan 27 11:41:07 pve kernel: [15734.536578]  do_vfs_ioctl+0xa9/0x640
Jan 27 11:41:07 pve kernel: [15734.536580]  ? handle_mm_fault+0xdd/0x210
Jan 27 11:41:07 pve kernel: [15734.536581]  ksys_ioctl+0x67/0x90
Jan 27 11:41:07 pve kernel: [15734.536583]  __x64_sys_ioctl+0x1a/0x20
Jan 27 11:41:07 pve kernel: [15734.536585]  do_syscall_64+0x5a/0x110
Jan 27 11:41:07 pve kernel: [15734.536587]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jan 27 11:41:07 pve kernel: [15734.536588] RIP: 0033:0x7ff45a198427
Jan 27 11:41:07 pve kernel: [15734.536590] Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 64 89 01 48
Jan 27 11:41:07 pve kernel: [15734.536592] RSP: 002b:00007ff44c578668 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Jan 27 11:41:07 pve kernel: [15734.536594] RAX: ffffffffffffffda RBX: 00000000c008aec1 RCX: 00007ff45a198427
Jan 27 11:41:07 pve kernel: [15734.536595] RDX: 00007ff44b00c000 RSI: ffffffffc008aec1 RDI: 0000000000000019
Jan 27 11:41:07 pve kernel: [15734.536597] RBP: 00007ff44b00c000 R08: 0000000000000000 R09: 00007ff44c57afe0
Jan 27 11:41:07 pve kernel: [15734.536599] R10: 0000000000005000 R11: 0000000000000246 R12: 00007ff44ce9ee80
Jan 27 11:41:07 pve kernel: [15734.536600] R13: 00007ff44ce9ee80 R14: 00007ff44b00c000 R15: 00007ff44ce9ee80
Jan 27 11:41:07 pve kernel: [15734.536603] Modules linked in: tcp_diag inet_diag arc4 md4 cmac nls_utf8 cifs ccm fscache veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter 8021q garp mrp softdog nfnetlink_log nfnetlink intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel kvmgt vfio_mdev aes_x86_64 mdev crypto_simd vfio_iommu_type1 cryptd glue_helper snd_pcm intel_cstate snd_timer intel_rapl_perf snd vfio soundcore kvm pcspkr irqbypass drm_kms_helper cdc_acm drm fb_sys_fops mei_me syscopyarea sysfillrect mei sysimgblt mac_hid acpi_pad vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) btrfs xor zstd_compress raid6_pq libcrc32c igb i2c_algo_bit i2c_i801 ahci dca libahci video
Jan 27 11:41:07 pve kernel: [15734.536641] CR2: 0000000000000000
Jan 27 11:41:07 pve kernel: [15734.536643] ---[ end trace c3290565f193dc72 ]---
Jan 27 11:41:07 pve kernel: [15734.536644] RIP: 0010:          (null)
Jan 27 11:41:07 pve kernel: [15734.536646] Code: Bad RIP value.
Jan 27 11:41:07 pve kernel: [15734.536647] RSP: 0018:ffffb596ac5ebb88 EFLAGS: 00010246
Jan 27 11:41:07 pve kernel: [15734.536649] RAX: 0000000000000000 RBX: 00007f5282e0c008 RCX: 0000000000000000
Jan 27 11:41:07 pve kernel: [15734.536650] RDX: 00007f5282e0c008 RSI: ffffb596ac5ebd08 RDI: ffff9ed929b70000
Jan 27 11:41:07 pve kernel: [15734.536651] RBP: ffffb596ac5ebcc8 R08: 0000000000000007 R09: 0000000000000018
Jan 27 11:41:07 pve kernel: [15734.536652] R10: ffff9ed8a0406100 R11: 0000000000000000 R12: ffffb596ac5ebb90
Jan 27 11:41:07 pve kernel: [15734.536653] R13: ffffb596ac5ebd08 R14: ffff9ed929b70058 R15: 0000000000000000
Jan 27 11:41:07 pve kernel: [15734.536654] FS:  00007ff44c57e700(0000) GS:ffff9edacea80000(0000) knlGS:0000000000000000
Jan 27 11:41:07 pve kernel: [15734.536656] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 27 11:41:07 pve kernel: [15734.536657] CR2: ffffffffffffffd6 CR3: 00000001abf98006 CR4: 00000000003626e0

pveversion -v

proxmox-ve: 6.1-2 (running kernel: 5.0.21-3-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-10
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-2
pve-cluster: 6.1-3
pve-container: 3.0-16
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

uname -a

Linux pve 5.0.21-3-pve #1 SMP PVE 5.0.21-7 (Mon, 30 Sep 2019 09:11:02 +0200) x86_64 GNU/Linux

 

[Stefan_R]

You're running a rather old kernel. Considering you've seemingly already upgraded to 6.1, a reboot should load a newer kernel automatically (5.3.13-1-pve at the time of writing).

 

[Phiolin]

Uhm... let me check. I rebooted today, so it looks like for some reason it did not pick a newer kernel.

Ok...checked. A grub-install on the boot device somehow fixed it. Not sure what was wrong, but looking into it.
5.3 kernel is now running and the Windows VM is starting.
Thanks.