Windows 10 VM stuck in Automatic Repair boot loop

[AlmostSerious]

Have you tried setting the CPU to MAX?
AFAIK this is kind of an "auto" setting choosing the best compatible for you. Also accross a Cluster.
Anyway changing from host to max fixed this issue for me on a single node.

 

[rcoll]

no luck for me with "cpu: max"
I am having again this automatic repair infinite loop on my NUC with an i7-10710U

For those interested to understand what's behind "max":
https://serverfault.com/questions/977643/kvm-difference-between-cpu-host-and-cpu-max

 

[AlmostSerious]

I am on the same NUC with the same CPU. Interesting that there is such a difference.
When you are in the Repair Loop, does the machine also shutdown in the middle of the process when running through it?

 

[rcoll]

I am on the same NUC with the same CPU. Interesting that there is such a difference.
When you are in the Repair Loop, does the machine also shutdown in the middle of the process when running through it?

Not 100% sure as I was playing with that issue for so many times.
At least I was able to try a win10 repair, which took quite a long time, ending with "it was unable to repair the issue".

But using same Intel NUC with different behaviors... are you also running latest Proxmox version ?

It was just updated now, showing 7.2-3, with new kernel:

Start-Date: 2022-05-06
Commandline: apt -y dist-upgrade
Install: pve-kernel-5.15.35-1-pve:amd64 (5.15.35-2, automatic)
Upgrade: pve-firmware:amd64 (3.4-1, 3.4-2), pve-kernel-5.15:amd64 (7.2-1, 7.2-3), pve-kernel-helper:amd64 (7.2-2, 7.2-3)
End-Date: 2022-05-06

I tried to revert to cpu: host without limiting up to SandyBridge compatibility. Without luck. I did try also to play with more recent CPU compatibilities... Skylake-Client, Broadwell, Haswell, IvyBridge without luck. I was getting CPU flags issues (as those are Xeon whereas the i7-10710U is not an enterprise CPU), so SandyBridge still looks to be a good choice for me.

 

[mira]

Does this only affect Win10 VMs or also other Windows versions?
I tried Windows Server 2022 on my Skylake (6700K) and it worked just fine.
One difference though to your VM config, I configured it with OVMF.

Have you tried switching to OVMF from SeaBIOS?

I'll try Win10 next, if you have any other Windows versions it doesn't work on, please tell us.

 

[rcoll]

Does this only affect Win10 VMs or also other Windows versions?
I tried Windows Server 2022 on my Skylake (6700K) and it worked just fine.
One difference though to your VM config, I configured it with OVMF.

Have you tried switching to OVMF from SeaBIOS?

I'll try Win10 next, if you have any other Windows versions it doesn't work on, please tell us.

Hello mira
Thanks for helping us.
I was curious, so I did install a new VM with Win10 21H2, using OVMF and UEFI.
Everything went fine with cpu: host, until I did install WSL (wsl --install in an admin cmd.exe)
At that very moment, after first reboot request of this install... I did end up with the automatic repair loop again.

 

[mira]

Oh, that explains it. I missed it in your previous posts.
WSL is rather hit and miss in a nested virtualization context.
My own tests with it, which was 2020 I think, also failed. Windows couldn't be booted afterwards.
I assume you mean WSL 2, right? The one requiring nested virtualization.

 

[rcoll]

Correct. I am currently playing with WSL2.
But I can assure you that it was working perfectly with Proxmox 7.1
It's only with 7.2 that it started to induce this automatic repair loop.
But again : for the moment, I am quite happy with the workaround:

cpu: host
args: -cpu SandyBridge,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+vmx

even if it's a bit slower than before.

 

[mira]

I installed WSL 2 to see if I can reproduce this, but my VM still works.
I installed from the Win10 21H2 EnglishInternational ISO.

Is there anything else you did?
Can you reproduce this issue on a fresh install?

Do you have the latest microcode installed? https://wiki.debian.org/Microcode

 

[rcoll]

Thanks for the hint about Microcode. I have to admit, I did not pay attention to this one.

Before adding intel-microcode:
[ 1.199471] microcode: sig=0xa0660, pf=0x80, revision=0xea
[ 1.199729] microcode: Microcode Update Driver: v2.2.

After:
[ 1.207097] microcode: sig=0xa0660, pf=0x80, revision=0xea
[ 1.207338] microcode: Microcode Update Driver: v2.2.

So no change. Nothing new unfortunately.

***

I did reinstall from scratch a Win10 21H2 English, using OVMF to follow your advice upper.
All was fine until I did install WSL:

C:\>wsl --install
Installing: Virtual Machine Platform
Virtual Machine Platform has been installed.
Installing: Windows Subsystem for Linux
Windows Subsystem for Linux has been installed.
Downloading: WSL Kernel
Installing: WSL Kernel
WSL Kernel has been installed.
Downloading: Ubuntu
The requested operation is successful. Changes will not be effective until the system is rebooted.

After this requested reboot -> Automatic repair loop again.

***

I think this is probably having something to do with the CPU gen10:
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 166
Model name: Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz

But again this can be corrected by editing the xxx.conf of the VM:
cpu: host
args: -cpu SandyBridge,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+vmx

Is there some debug trace I could provide to help you pinpoint what is going wrong ?

 

[mira]

If you're using an Intel CPU, why did you install the amd-microcode package?
Install the one for Intel CPUs and reboot afterwards.

 

[rcoll]

If you're using an Intel CPU, why did you install the amd-microcode package?
Install the one for Intel CPUs and reboot afterwards.

Sorry my bad. I am tired.

Here is my bash history
1178 apt install intel-microcode
1179 reboot
1180 qm list
1181 top
1185 lscpu

So I well did install intel-microcode with no improvement. Nothing new.

 

[oguz]

could you try the following on your PVE host:

echo Y > /sys/module/kvm/parameters/ignore_msrs

and try running your win10 VM with the host cpu.

this fixed the BSOD at boot for me, would be interesting to know if it works as well.

the workaround is documented on our wiki [0] (if it works you can make it persistent by following the instructions there)

[0]: https://pve.proxmox.com/wiki/Nested_Virtualization#Bluescreen_at_boot_since_Windows_10_1803

 

[rcoll]

hello oguz
I saw that one and tried it in the past.
I did another try right now for you, to be sure.
I am again ending with this "automatic repair loop".
I never had a BSOD in fact.
It's just looking like Win10 cannot enable hyper-v - which is required for WSL - and because of that it's doing an endless automatic repair.

 

[oguz]

I am again ending with this "automatic repair loop".
I never had a BSOD in fact.

when installing a win10 VM with the host CPU i was getting a BSOD, the above fixed it for me.

the first boot after installation was successful, the following boots were stuck in the automatic repair like you've described.

we'll let you know if we find a solution

 

[oguz]

It's just looking like Win10 cannot enable hyper-v - which is required for WSL - and because of that it's doing an endless automatic repair.

can you please try this on your PVE host:

echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf
update-initramfs -k all -u
reboot

here it works after the reboot, with Windows 10 using host CPU (running a 1st gen Ryzen) and SeaBIOS.

here's my VM configuration for reference:

boot: order=ide2;scsi0;ide0
cores: 4
cpu: host
ide0: ISO:iso/virtio-win-0.1.215.iso,media=cdrom,size=528322K
ide2: ISO:iso/Win10_21H2_EnglishInternational_x64.iso,media=cdrom,size=5748118K
kvm: 1
machine: pc-q35-6.2
memory: 4096
meta: creation-qemu=6.2.0,ctime=1651749465
name: win10-host-seabios
net0: virtio=B2:43:F1:5B:79:19,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: guests:vm-422-disk-1,cache=writeback,discard=on,size=32G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=46fc806d-b1fe-4746-ab1d-4a151da6ded2
sockets: 1
vmgenid: 54c6df43-399f-473c-a052-0fc0394452c1

installing WSL2 and running ubuntu also works.

no errors and no automatic repair at consequent boots either

As the title says the machine was fine and running but at any point Im not able to boot the machine. It Stucks on this blue screen saying "Automatic Repair mode". And does not go through

Im really stuck here. I com into the "secured" mode of windows. I found the the Memory.dmp but unfortunately I had not installed the "Windows debug utils" to view it with dumpchk.

if the above does not work for you, i would suggest trying to do a clean installation of windows 10 (after the workaround has been applied) and try it with that, as there might be actually something wrong with your VM disk or windows partitions if you didn't do a clean shutdown (we can rule that out if the clean installation works normally).

hope this helps

 

[rcoll]

No luck on my side with my NUC i7-10710U

I configured like you asked:

echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf
update-initramfs -k all -u
reboot

My current Win10 installed VM was still stuck in automatic repair loop.
I tried to set up another Win10 VM from scratch to be sure.
After trying to install WSL
wsl --install
it ended up again with an endless automatic repair loop, unfortunately. :-(

 

[oguz]

No luck on my side with my NUC i7-10710U

tested this on our (older) NUC here and it works fine so far with WSL2 (with kernel 5.15.35-1-pve )

But as my setup was perfectly working in Proxmox 7.1 and as it's suddenly broken in Proxmox 7.2... I am quite confident it's a bug related to linux kernel 5.15

so i'm not quite sure if it's kernel related.. though to be certain you could rollback to the older kernel.

otherwise i'm not sure what you've changed in your machine that could cause the issue.

After trying to install WSL
wsl --install
it ended up again with an endless automatic repair loop, unfortunately. :-(

can you also post your VM configuration? qm config VMID

 

[rcoll]

I tried also yesterday to boot on previous kernel with:

 checked the list with:
 pve-efiboot-tool kernel list
 
 then I tried to revert to kernel of Proxmox 7.1:
 pve-efiboot-tool kernel pin 5.13.19-6-pve --next-boot

without luck. So I wonder if it's due to newer qemu and other packages coming with Proxmox 7.2

As you requested here is my qm config:

# qm config 101
bios: ovmf
boot: order=ide2;sata0;net0
cores: 1
cpu: host
description: args%3A -cpu SandyBridge,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+vmx
efidisk0: local-zfs:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide2: local:iso/virtio-win.iso,media=cdrom,size=519096K
machine: pc-q35-6.2
memory: 4096
meta: creation-qemu=6.2.0,ctime=1651843844
net0: virtio=A2:4D:2B:83:4E:98,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata0: local-zfs:vm-101-disk-1,size=60G
scsihw: virtio-scsi-pci
smbios1: uuid=ab428338-b3be-4bf4-bc05-de16c971359b
sockets: 2
vmgenid: 27a5aa3a-04de-4619-806a-912d81118fff

description: is my commented out cpu arg line which is always solving my issue for the time being.

Edit:
it has to be noted this one is running with UEFI (ovmf) as I was asked in this thread to test like this. But I am encountering same issue if chosing Seabios.

 

[oguz]

without luck. So I wonder if it's due to newer qemu and other packages coming with Proxmox 7.2

you can also test older qemu versions by changing the machine property in the "Hardware" menu for your VM and pin an older qemu version.
though in my tests the qemu version has made no difference.

it has to be noted this one is running with UEFI (ovmf) as I was asked in this thread to test like this. But I am encountering same issue if chosing Seabios.

same setup works here using OVMF as well:

bios: ovmf
boot: order=scsi0;ide2
cores: 4
cpu: host
efidisk0: guests:vm-424-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
ide0: ISO:iso/virtio-win-0.1.215.iso,media=cdrom,size=528322K
ide2: ISO:iso/Win10_21H2_EnglishInternational_x64.iso,media=cdrom,size=5748118K
kvm: 1
machine: pc-q35-6.2
memory: 4096
meta: creation-qemu=6.2.0,ctime=1651749465
name: win10-host-ovmf
net0: virtio=8A:36:B8:50:9C:04,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: guests:vm-424-disk-0,cache=writeback,discard=on,size=32G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=80446a4e-16ed-4a8a-82c9-9ac42599bfa3
sockets: 1
vmgenid: 3a09285c-36e1-4793-89df-0b23a0305d56

* do you have intel-microcode package installed? (for amd would be amd64-microcode)

* have you checked for possible BIOS upgrades?