Redhat VirtIO developers would like to coordinate with Proxmox devs re: "[vioscsi] Reset to device ... system unresponsive"

With the master branch driver, My windows VM has the "blue screen of death" problem.
Are you sure you have successfully installed the driver from master? It sounds like you might still be using the v262 package driver.

In Device Manager:

Right-click: Red Hat VirtIO SCSI pass-through controller
Select: Properties
Select: Driver tab

The Driver Version should be 100.6.101.58000 and not 100.95.104.26200.
 
Are you using the following process?

In Device Manager:

Right-click: Red Hat VirtIO SCSI pass-through controller
Select: Update driver
Select: --> Browse my computer for driver software
Select: --> Let me pick from a list of available drivers on my computer
Select: Have Disk...
Browse
and select the vioscsi.inf file, then choose OK
Select: Next
Select: Close

The version will be 100.6.101.58000.
This do update the driver (I miss the "Select: Have Disk..." opertaion before).

BUT, I pretty sure that the current working driver is the one I built when "blue screen of death" happened, by checking the driver version info in Device Manager (I have changed the version info in the source code, and the version info matched ).
 
BUT, I pretty sure that the current working driver is the one I built when "blue screen of death" happened, by checking the driver version info in Device Manager (I have changed the version info in the source code, and the version info matched ).
I'm not seeing a BSOD myself, but I'm only installing the vioscsi driver from master, the rest is v248 (I only did brief testing aside v262).

Did you change anything other than the version number in the source? It's possible there are unintended side effects from making this change.

Also, what is your target...?
I have only built and tested for Win10.
IIRC, you built with the Win11 WDK. That's ok, and should work, but I have not performed extensive testing with such a build myself (yet).
Could you perhaps do a fresh build from master with no mods, and give it a try?
 
Hi team

Is this issue still active? No response from the parties involved?

I am experiencing this bug with Windows Server 2022 and SQL Server 2019.

The virtual disk that hosts the databases is the one that suddenly loses access, I also see in the log the records mentioned above.

We use the virtio-win-0.1.240 driver

Is using the virtio-win-0.1.208 driver still the best option?
 
Hi, as they already mentioned, @benyamin's pull request [1] was merged upstream a couple of weeks ago. With this patch, the reproducer I was using did not trigger the issue anymore [2]. So this looks quite promising, thanks again @benyamin for your great work!

While the patch looks promising, it might still be too early to conclude it fixes the issues reported here completely. For this, more testing with realistic workloads would be necessary. Until a new release of virtio-win including the patch is available, for testing one would need to build virtio-win from git manually, see [3] for instructions.

So for the time being until a new virtio-win release is available, the workarounds reported e.g. by @RoCE-geek here [4] still apply.

The host qemu log show that libvirt hit the fd limit due to lots of connections to ceph OSD, this must be cause of the BSOD. I will handle it and and test the driver again.
I'm confused -- Proxmox VE does not ship libvirt. Can you elaborate on your setup? Can you provide the output of pveversion -v?
The issue you're describing (hitting the fd limit due to OSD connections) sounds like [5] which has been fixed since earlier this year.

[1] https://github.com/virtio-win/kvm-guest-drivers-windows/pull/1150
[2] https://github.com/virtio-win/kvm-guest-drivers-windows/pull/1150#issuecomment-2363879224
[3] https://virtio-win.github.io/Development/
[4] https://forum.proxmox.com/threads/139160/post-691337
[5] https://bugzilla.proxmox.com/show_bug.cgi?id=4507
 
I think the biggest hold up is because .248 of the guest tools installer screwed up installations, and for many they cannot be upgraded/downgraded or even uninstalled - the MSI installer aborts and rolls back. You can update via each component directly e.g. NIC/Storage drivers, but for the balloon driver there's an extra component, the service, that using device manager doesn't take into account. It's a bit of a mess.
 
for me v266 fixed the hang, I'm unable to hang on my two reproducer in a VM test.
with v240 , I'm easy able reproduce the hang in VM + windows log event id 129 vioscsi reset device with CrystalDiskMark test NVMe Profile 5x8GB Read&Write.
Will update production VM this week-end.
 
Last edited:
This might aid in your upgrades:

Bash:
#!/bin/bash
for vmid in $(qm list | awk '{print $1}' | grep -v VMID)
do
  if qm guest cmd $vmid ping > /dev/null 2>&1 ; then
    if qm guest cmd $vmid get-osinfo | grep mswindows > /dev/null ; then
      PROBE_VER="$(qm guest exec \
       $vmid --timeout 0 -- powershell \
       "Get-ItemProperty HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\* | Where-Object -Property DisplayName -contains Virtio-win-guest-tools | Select-Object DisplayVersion | fl" | \
       grep out-data | sed 's/["\\r\\n]//g' | sed 's/   out-data : DisplayVesio : //g')"
      if ! echo VM $vmid - VirtIO Tools version: "${PROBE_VER}" ; then
        echo Could not get VirtIO Tools Version for VM $vmid.
      fi
    else
      echo VM $vmid - NOT running Windows.
    fi
  else
    echo VM $vmid - OFFLINE or no QEMU Agent.
  fi
done

Copy that to a file on your Proxmox host and then issue a chmod +x <filename> and then execute it with sh ./filename.

Enjoy..! 8^d
 
  • Like
Reactions: Whatever and itNGO
On half of my servers I can't install new Virtio Tools. Setup breaks and reverts with error 0x80070643
Any clue what could be wrong?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!