Windows Server update causing boot issues with VirtIO SCSI

[nnraymond]

Summary: I had to detach Windows Server boot drives in our Server 2012 R2 and 2019 VMs to change their configuration from SCSI to IDE so that Windows Server would no longer get stuck endless booting this morning (spinning circle, no error, no crash, but no progress in the boot process). Windows updates had applied in the early AM and caused the situation. Our VMs have VirtO SCSI drivers installed from the virtio-win-0.1.189.iso.

Question: Is this a known issue? Do more recent VirtIO SCSI drivers fix this?

More detail: When this problem happened, our Proxmox installations were running 7.2-7, and thinking it might be a bug in that version, I tried updating to 7.3-3 but that did not improve the situation at all. The Windows Server 2019 update that triggered this was KB5021237:

https://support.microsoft.com/en-us...763-3770-8c1506cc-e030-4cf1-8cd6-774091f46f34

The Windows Server 2012 R2 update that triggered this was either KB5021294 or KB5021296 (both had been installed):

https://support.microsoft.com/en-gb...y-rollup-aa4f93d3-883c-4918-85af-76abbe1d5586
https://support.microsoft.com/en-us...y-update-a7796022-c0e2-4e3f-8630-c987d4e4b3bb

When I first encountered the problem, I tried using the Windows Server install ISO to boot into the recovery console after detaching the boot drive and changing it from SCSI to IDE (I had a stock Windows Server install ISO, and for reasons of expediency I didn't want to go through the third-party driver loading process from inside the install environment to access the drive as a VirtIO SCSI drive). In the recovery console with the drive as an IDE drive I could browse it just fine, all files seemed to be there just fine, partitions looked normal. I issues the commands bootrec /fixmbr and bootrec /fixboot, shut down the VM, detached the boot drive and changing it back from IDE to SCSI, then booted the VM back up. The OS still was stuck in the infinite spinning circle on boot, so no improvement. On a hunch I stopped the VM, detached the boot drive and changed it to IDE, started it back up and it booted normally. Windows Server had been in the middle of applying the Windows update, it finished that process without incident, and I could then log into Windows Server where I verified everything was running fine (but my boot drives are now configured as IDE, not VirtIO SCSI devices, which is not ideal).

 

[rason]

Please make sure that you have all the required drivers installed. Here's the official Proxmox guide: https://pve.proxmox.com/wiki/Windows_2012_guest_best_practices

 

[nnraymond]

Please make sure that you have all the required drivers installed. Here's the official Proxmox guide: https://pve.proxmox.com/wiki/Windows_2012_guest_best_practices

All drivers have been installed from day 1 when all the VMs were set up about two years ago, and the Windows Device Manager has been checked with the DVMGR_SHOW_NONPRESENT_DEVICES environment variable set to 1 and show hidden devices. VMs have been running fine through hundreds of Windows updates and a Proxmox 6.x to 7.x upgrade, this is the first time I have run into an issue like this. We also have VMs in VMWare, and they are not experiencing these problems with the same updates.

 

[nnraymond]

So I thought I'd test out the latest VirtIO SCSI drivers, but I can't even install the latest from virtio-win-0.1.225.iso which is what is linked to as the current stable drivers. I mounted the ISO in the VM, and first tried running virtio-win-gt-x64.msi but that results in this error as soon as the install starts:


So then I tried updating just the VirtIO SCSI driver manually. Here is what is installed:

I chose "Update Driver..." and said I had my own disk and manually navigated to the driver on the ISO:

Then it complains about the driver not being signed, which I would think is an issue:

I clicked on "Next" and that results in this error:

I'm guessing that's because the latest driver isn't signed? Is there a better ISO I should be using other than the current stable virtio-win-0.1.225.iso?

 

[sherminator]

I'm guessing that's because the latest driver isn't signed? Is there a better ISO I should be using other than the current stable virtio-win-0.1.225.iso?

We ran into this recently. Our workaround goes like this: During the installation wizard, when all components are listed, select "Feature will be installed when required" for every component.
We did a quick research and found hints that one of the components, which is required almost never, is responsible for the issue.

Greets
Stephan

 

[VictorSTS]

As far as I can remember, VirtIO drivers have always been digitally signed with a valid CA and signature but for some reason when you try to update the drivers, Windows is telling you that the driver isn't signed . Maybe that certificate has expired or the CA has been removed?

I did install a couple of Win2K19 like a month ago and it was all ok... Don't remember which version of VirtIO did I use at the time, probably something like 0.1.217.

Regarding the VirtIO ISO, there are currently two iso files for v0.1.225, check that you are using the latest. Also, you may try with any other iso file:

https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/?C=M;O=D

 

[nnraymond]

I did download the ISO from https://fedorapeople.org/groups/vir...ownloads/archive-virtio/virtio-win-0.1.225-2/ which by all accounts is the latest. I ran the installation wizard again and tried selecting "Feature will be installed when required" and while it did finish the installation without error, it was done in about a second and didn't suggest rebooting or anything, and I'm not convinced it installed anything. I checked the driver version number of the Red Hat VirtIO SCSI pass-through controller before and after running the 0.1.225 installer and it remained the same. If I get properties on vioscsi.sys in the directory \vioscsi\2k12R2\amd64 on the install ISO it says the version number is 62.91.104.22500 which is later than what is installed. It would be nice if there was a way to actually install that driver update.

Last night I did some testing where I shut down one of the VMs affected by all this, changed the boot drive in one of the systems from IDE back to SCSI, and it didn't have any trouble booting up. That indicates that this issue with non-IDE boot drives may be limited to just during the application of these Windows updates.

I also looked back into the history of our VMs and this problem may be further limited to Windows Server systems that went through an upgrade-in-place since it appears that our Windows Server VMs that were installed fresh with Server 2012r2 did not encounter this boot issue during the Windows updates in question. This may explain why there aren't more widespread reports of this issue going on right now.

 

[VictorSTS]

You should be able to update the drivers manually using device manager, on the driver tab of the storage controller (or any other virtIO device), and manually looking for the driver in the VirtIO CD.

 

[nnraymond]

You should be able to update the drivers manually using device manager, on the driver tab of the storage controller (or any other virtIO device), and manually looking for the driver in the VirtIO CD.

I agree that I should be able to do that, but that method is throwing an error, which looks like it might be related to driver signing (see above).

 

[sherminator]

I ran the installation wizard again and tried selecting "Feature will be installed when required" and while it did finish the installation without error, it was done in about a second and didn't suggest rebooting or anything, and I'm not convinced it installed anything.

wow, your're right! "... when required" seems to do nothing, when there is already some driver installed. Thanks for your research.
I did some further trial & error and figured out that if I deselect "Qemupciserial" ("Entire feature will be unavailable") the installation wizard installs the current drivers and ends successfully.

 

[nnraymond]

wow, your're right! "... when required" seems to do nothing, when there is already some driver installed. Thanks for your research.
I did some further trial & error and figured out that if I deselect "Qemupciserial" ("Entire feature will be unavailable") the installation wizard installs the current drivers and ends successfully.

I will test that out as soon as I have a chance.

Meanwhile, further digging reveals that driver signing changed at some point. Comparing the vioscsi.sys driver from virtio-win-0.1.189.iso vs. virtio-win-0.1.225.iso one can immediately see a difference: the older driver was issued by "Symantec Class 3 SHA256 Code Signing CA - G2" with a certification path that goes back to "VeriSign Universal Root Certification Authority" while the newer driver has a certificate issued by "Red Hat Inc." that traces back to nothing. This is why the older VirtIO drivers currently installed in my VMs installed without issue and why the new ones won't without taking additional steps. This is a big change to make in driver signing that I haven't seen documented or mentioned on the Proxmox side of things, i.e. no mention of any of this at https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers#Manual_Installation where it is a very relevant issue. At this point I am also unsure if resolving this is simply a matter of installing the new certificate issued by Red Hat Inc. into the Trusted Root Certification Store, or if it is also necessary to use Bcdedit.exe to enable the TESTSIGNING boot configuration option as documented here: https://github.com/virtio-win/virtio-win-pkg-scripts/blob/master/README.md

 

[Cactus]

Hi, I am new to the proxmox and faced the same issue that Windows 2019 got stuck at logo after Windows update.

Could you please let me know how to change the system from SCSI to IDE in order to bring the system back online?

Thanks!

is this settings you are referring?



or


that i have to detach scsi Hard Disk and add a new disk using IDE?

 

[ChiquiFornari]

figured out that if I deselect "Qemupciserial" ("Entire feature will be unavailable") the installation wizard installs the current drivers and ends successfully.

THIS and installing Qemu agente solved the issue for me.
Thank you a lot!

 

[IDLOG]

Hello, I have the same problem today on a proxmox 7.2 - I clone an operational Windows 2019 VM and at startup: recovery blue screen - Like you, I switched the SCSI to IDE but at startup, I have after the windows logo and the hourglass an error stop code 0xc00002e2 - the virtio drivers are in v215 but as I no longer have access to windows, I cannot upgrade them to v229 - I did all the bootrec, chkdsk, . ..: impossible to restart - does anyone have a solution please? Thanks in advance

 

[ChiquiFornari]

Hello, I have the same problem today on a proxmox 7.2 - I clone an operational Windows 2019 VM and at startup: recovery blue screen - Like you, I switched the SCSI to IDE but at startup, I have after the windows logo and the hourglass an error stop code 0xc00002e2 - the virtio drivers are in v215 but as I no longer have access to windows, I cannot upgrade them to v229 - I did all the bootrec, chkdsk, . ..: impossible to restart - does anyone have a solution please? Thanks in advance

I understand you are moving a W2019 VM from one proxmox to a different proxmox server, is that right?

In the original VM before cloning it, which bus device were you using: IDE , SCSI or VirtioSCSI?

 

[Maximiliano]

Hello, I would suggest you take a look at [1, 2]. Unfortunately Window has to have seen a device that requires the driver before it can boot from a device that requires the driver, normally this would result in a blue screen at boot. The standard procedure is to attach a SCSI drive to your machine (it can be of any size) and let Window recognize it, reboot, and then set your other drivers at SCSI. You can find more details at [1].

[1] https://pve.proxmox.com/wiki/Additi...rt_Windows_to_use_.28VirtIO.29_SCSI_.28KVM.29
[2] https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers

 

[IDLOG]

I understand you are moving a W2019 VM from one proxmox to a different proxmox server, is that right?

In the original VM before cloning it, which bus device were you using: IDE , SCSI or VirtioSCSI?

Hello, thank you for the response. I just found the problem: after detaching the SCSI disk and attaching it in IDE, I was able to enter the advanced recovery mode and choose the restart option for directory services repair. Indeed it is a domain controller and several log files were corrupted. I was able to repair them and reinstall version 229 of the Virtio drivers and everything was back to normal. It seems that saving on PBK is the cause...

 

[IDLOG]

Hello, I would suggest you take a look at [1, 2]. Unfortunately Window has to have seen a device that requires the driver before it can boot from a device that requires the driver, normally this would result in a blue screen at boot. The standard procedure is to attach a SCSI drive to your machine (it can be of any size) and let Window recognize it, reboot, and then set your other drivers at SCSI. You can find more details at [1].

[1] https://pve.proxmox.com/wiki/Additi...rt_Windows_to_use_.28VirtIO.29_SCSI_.28KVM.29
[2] https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers

Hello, thank you for the response. I just found the problem: after detaching the SCSI disk and attaching it in IDE, I was able to enter the advanced recovery mode and choose the restart option for directory services repair. Indeed it is a domain controller and several log files were corrupted. I was able to repair them and reinstall version 229 of the Virtio drivers and everything was back to normal. It seems that saving on PBK is the cause...