Windows Server 2016 frequent lockups with 100% CPU

[casalicomputers]

Hello,
since a few days we're experiencing frequent lockups on ALL Windows Server 2016 VMs (i got 3).
These vms have been running without any issue for months, but now all are experiencing the same issue.

As you can see from the attached screenshot



CPU spikes at 100% and there's no way to access the affected VM via SPICE/noVNC or even reset it. When this happens I can only STOP the VM and then restart it, waiting for the next time this happens (usually within few hours).

Here's VM configuration:

boot: cdn
bootdisk: scsi0
cores: 4
ide2: none,media=cdrom
memory: 4096
name: dea-cnc
net0: virtio=5A:0B:3C:8E:B4:C4,bridge=vmbr0
numa: 0
onboot: 1
ostype: win8
parent: new
scsi0: raid10-ssd:vm-104-disk-1,size=200G
scsi1: raid10-ssd:vm-104-disk-2,size=5G
scsihw: virtio-scsi-pci
smbios1: uuid=a835c388-2f72-4540-9b81-50c464178624
sockets: 1
startup: order=10
vga: qxl

I really, really, really have no idea of what could be the cause, considering that I have absolutely no problems with linux VMs on the same server. Here's what I tried until now:

1) Upgraded PVE to latest version via APT
2) Powered off and on the server after the system upgrade
3) Disks were set to virtio-blk (virtioN) with default LSI SCSI controller. Migrated to virtio-scsi (scsiN) with VirtIO SCSI controller.
4) Updated spice-tools with latest package from spice-space.org

I suspect this has been caused by some microsoft update because, I repeat, the systems were working properly since a few days ago, and the weird thing is that only Windows vms are affected .... randomly of course.

Hope that someone could give hints on how to troubleshoot this.

Thanks
Michele

 

[casalicomputers]

As of today the issue is still present and happens really often.
Just updated BIOS and storage controller's firmware to latest version from manufacturer.

Let's see how it goes.

 

[casalicomputers]

Looks like the system behaves better, but lockups still happen even if less frequently.
I really don't know what to do next.

Please help!

EDIT:
I exclude any hardware defect because

1. only windows server 2016 vms are affected
2. the idrac on the server does not report any issue

 

[casalicomputers]

Don't know if this could help, but I found some errors in the eventlog:

The computer has rebooted from a bugcheck. The bugcheck was: 0x00000109 (0xa39fd8da6d32975e, 0xb3b6e560bfb48ccf, 0x0000032000000000, 0x0000000000000017). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: a755cf19-7b99-4c6c-a56c-b34dcbbb1f9a.

Bugcheck 0x00000109 with 0x17 as the 4th parameter means Local APIC modifications
(from https://docs.microsoft.com/en-us/wi...g-check-0x109---critical-structure-corruption)

Memory.dmp is roughly 80MB compressed - if someone needs it, I can send.

 

[casalicomputers]

Digging into the forum I discovered a few posts which could be related to some APIC issues.

• https://forum.proxmox.com/threads/windows-2003-64-bit-random-crashes-apic-related.5317/
• https://forum.proxmox.com/threads/windows-xp-multicore-bsod.4550/

which both resolved by disabling some power saving settings in the bios.
Unfortunately this didn't work for me as they were already disabled.

Btw, I found and enabled a setting named X2APIC which was disabled.
Again, let's see how it goes ...

 

[casalicomputers]

No luck.

VMs still experience lockups and I had to create a script that runs every 5 minutes, pings the guest agent and powercycles the vm when it doesn't reply. It's not the best solution, I know, but at least this reduces the downtime.

 

[OXIB]

Hi caslicomputers,
Were you able to resolve this issue with your Windows Server 2016 vm?
I found this post as I am having the same issue like you do with a Windows Server 2016 Domain Controller. I just started 5 days ago and there seems to be two blue screens. Funny thing with me, I have another Windows Server 2016 F&P and that one seems to be OK. I am monitoring it now to make sure that there are no stop errors.
here is the bug check from the DC:

and this one:


If you were able to solve this, would you please let me know how you did it?
I'd really appreciate it.
Thanks!
OXIB.

 

[casalicomputers]

Hi Oxib,
yes I've been able to resolve via Proxmox support.
Basically I installed a package (vlan) via APT which removed the package "proxmox-ve" and prevented pve kernels to be updated.
After I reinstalled the "proxmox-ve" package, did a apt dist-upgrade and reboot, everything started working again.

Hope this helps you as well.

Michele