VM Windows server 2022 crashed randomly after update proxmox - again.

[ColinDexter]

After updating proxmox, Windows Server 2022 VM crashed randomly with a different message every time.
In my previous post it was indicated that physical memory would probably be the cause. I replaced this and after that it seemed to be fine for a while. But unfortunately it was not the solution. The server still restarts regularly. As you can see below with all different notifications. So I have now replaced the physical memory and hard disk. Have even reinstalled Windows 2022 server...

I cannot trace anything in the proxmox logs to what could cause these restarts.

I actually have no idea where to look anymore.

[edit]
It turns out that not only Windows server 2020 crashed. A regular Windows 10 Client VM also crashes spontaneously. One that runs on another host without any problem.

 

[_gabriel]

without more informations, will be hard to guess.
full hw Specs.
you should try a bare metal Windows install.

 

[fanumtax123]

you need to reset all configuration and install proxmox from the start, i had the same issue

 

[ColinDexter]

What exactly do you mean by "reset all configuration"?

 

[gfngfn256]

What does cat /etc/pve/qemu-server/VMID.conf show.

 

[pille99]

did you check your memory ?
all my VMs runs smooth and for many others as well.
you have as 1 proxmox server ?
does the VMs run on other servers without problem ?

my attantion catched: Attempted to write to readonly_memory

the readonly memory is usually a ROM.

did you do, with the ugrade change something on the hardware config ?

 

[ColinDexter]

@gfngfn256
root@pev1:~# cat /etc/pve/qemu-server/VMID.conf
cat: /etc/pve/qemu-server/VMID.conf: No such file or directory

@pille99
Yes, I checked the memory and even replaced it.
If I move the VM to another host it works without any problems.
There are always different messages that come along. The IRQL and system message occur most often.

 

[gfngfn256]

@gfngfn256
root@pev1:~# cat /etc/pve/qemu-server/VMID.conf
cat: /etc/pve/qemu-server/VMID.conf: No such file or directory

You have to replace VMID with the actual ID number of the VM.
Example; if your VM is 105, you would enter cat: /etc/pve/qemu-server/105.conf

 

[ColinDexter]

ah sorry, I misunderstood that. Below is the content of the config

agent: 1
bios: ovmf
boot: order=ide0;scsi0;net0
cores: 4
cpu: x86-64-v2-AES
efidisk0: shared-storage:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=528K
ide0: none,media=cdrom
machine: pc-q35-8.0
memory: 4096
meta: creation-qemu=8.0.2,ctime=1690360033
name: W2022
net0: e1000=9E:0E:3F:25:47:0F,bridge=vmbr0,tag=12
numa: 0
ostype: win11
scsi0: shared-storage:vm-101-disk-1,cache=writeback,discard=on,size=128G
scsihw: virtio-scsi-pci
smbios1: uuid=e6351074-a01f-484a-a75a-1ce9a36fd2e5
sockets: 1
vmgenid: 10c854d1-1107-4585-bd51-f1e33db00887

 

[cwt]

For

scsi0: shared-storage

What kind of storage do you use for your VM?

 

[ColinDexter]

SSD (Ceph)

 

[cwt]

Did you check your Ceph setup for any errors/problems or bottlenecks?

 

[emunt6]

scsi0: shared-storage:vm-101-disk-1,cache=writeback,discard=on,size=128G

Old story, simple fix:

cache=none, aio=native, discard=off

 

[gfngfn256]

cpu: x86-64-v2-AES

I would try host instead

 

[ColinDexter]

@emunt6 @gfngfn256
I have adjusted the settings and am curious if this works. Because this VM will run with this configuration on another host with exactly the same hardware setup. What is the difference in the proxmox that it no longer functions properly after an update?

 

[emunt6]

@emunt6 @gfngfn256
I have adjusted the settings and am curious if this works. Because this VM will run with this configuration on another host with exactly the same hardware setup. What is the difference in the proxmox that it no longer functions properly after an update?

New Proxmox versions adding new configs and changing the default values of the VMs qemu-kvm command line config, some of them are hardcoded. I prefer stability over performance, the suggested settings are for that. I also I do not recommend "virtio" based emulation for Windows, I had many bad experiences with it ( maybe Redhat/IBM fixed many of them over the years - but still lack of some: see https://forum.proxmox.com/threads/support-for-windows-failover-clustering.141016/ ).


You can check the "hidden" difference of the VMs config:

$> qm showcmd <vmid>
vs
$> qm config <vmid>

 

[ColinDexter]

Things seemed to go well for a few days after the changes. But unfortunately another BSOD this morning.
What do you think are the best settings in proxmox for a Windows VM? I had the current settings advised in the proxmox help.

 

[gfngfn256]

I don't know your HW setup, but I note you use Ceph, so I assume cluster. You need to provide more details. You must realize Ceph has its requirements which must be met. Your networking HW/SW is also going to play a major role in your Ceph performance/stability.


Can you try an alternate (non network-dependent) type of storage on this PVE, as a testing ground for this VM.

Because this VM will run with this configuration on another host with exactly the same hardware setup. What is the difference in the proxmox that it no longer functions properly after an update?

To be able to answer this, we would have to know ALL the differences between the 2 hosts, (HW & SW &NW). I've never encountered 2 identical hosts. Also for what period of time have you run it on the other host? I quote you:

Things seemed to go well for a few days after the changes. But unfortunately another BSOD this morning.

So obviously you need to do at least comparative run-time lengths, for this test to have any value.


Have you ever tried running this VM on bare-metal? If it has no errors, we can rule out a non-PVE/network linked problem.

 

[emunt6]

Things seemed to go well for a few days after the changes. But unfortunately another BSOD this morning.
View attachment 66135What do you think are the best settings in proxmox for a Windows VM? I had the current settings advised in the proxmox help.

Remove "virtio" based emulation from the VM,
I have seen same problem like this, thats why i never use "virtio" for Windows anymore (drivers unstable/buggy).

VM config:
balloon: 0
net0: e1000
ostype: other
scsi0: cache=none,aio=native
scsihw: pvscsi

LSI based SCSI emulation is not supported by Windows 2022, but the "vmware pvscsi" is supported.

 

[ColinDexter]

I have 3 of the same NUC running. On two of these, Windows continues to run until I restart for the monthly updates. In the past everything worked perfectly on all three. The one on which I now constantly get BSOD with Windows is the one that crashed during the promox update. I reinstalled it completely and since then have these problems.

@emunt6
Thanks, I'm going to adjust that. See if that helps.