Redhat VirtIO developers would like to coordinate with Proxmox devs re: "[vioscsi] Reset to device ... system unresponsive"

benyamin said:
Well everyone, you can see Vadim's (Red Hat) comment here: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2283909521

If, as Vadim suggests, it is "[l]ikely to Nutanix and us (RH) we don't see any problem if the IO transfer exceeds the virtual queue size. Others might have problem with that", I guess the question for the Proxmox / Debian side would be whether or not we can do anything about that...

@fiona and @fweber, would you both be the right staff to ask that question of?
Click to expand...
Please coordinate your steps with @bbgeek17 as well. He/they have some deeper findings to ev. confirm this hypothesis (Post 1 + Post 2).

And as I see now, Friedrich (@fweber / frwbr) concluded to the same results: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2220730292

So I'm quite sure we're very close to the final step.

Many thanks to anyone interested.
 
  • Like
Reactions: carles89
benyamin said:
Well everyone, you can see Vadim's (Red Hat) comment here: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2283909521

If, as Vadim suggests, it is "[l]ikely to Nutanix and us (RH) we don't see any problem if the IO transfer exceeds the virtual queue size. Others might have problem with that", I guess the question for the Proxmox / Debian side would be whether or not we can do anything about that...
Click to expand...
Do you know which kernel the Redhat people are testing on? I know that Nutanix is a bit behind.
 
  • Like
Reactions: Taomyn, carles89 and Falk R.
Hi, thanks for all your extensive research in this issue. @RoCE-geek's [1] and @bbgeek17's [2] results indeed look consistent with my [3] results.

benyamin said:
If, as Vadim suggests, it is "[l]ikely to Nutanix and us (RH) we don't see any problem if the IO transfer exceeds the virtual queue size. Others might have problem with that", I guess the question for the Proxmox / Debian side would be whether or not we can do anything about that...

@fiona and @fweber, would you both be the right staff to ask that question of?
Click to expand...
Vadim's latest response [4] suggests that there might be a possible fix on the virtio-win side. While a large portion of the users affected by this issue seem to be Proxmox VE users, there is at least one user who recently reported seeing this issue with libvirt on Debian 12 [5]. According to the github thread, the issue is not present on RH or Nutanix. So there may to be some factor at play here which makes the issue more likely to trigger on Proxmox VE (or Debian in general?), but it's not clear yet what that might be. It's unlikely to be our downstream QEMU, because the issue also triggers with upstream QEMU [3].

[1] https://forum.proxmox.com/threads/r...device-system-unresponsive.139160/post-691337
[2] https://forum.proxmox.com/threads/r...device-system-unresponsive.139160/post-692230
[3] https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2220730292
[4] https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2284118004
[5] https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2212543280
 
  • Like
Reactions: benyamin, Moayad, carles89 and 2 others
fweber said:
Hi, thanks for all your extensive research in this issue. @RoCE-geek's [1] and @bbgeek17's [2] results indeed look consistent with my [3] results.


Vadim's latest response [4] suggests that there might be a possible fix on the virtio-win side. While a large portion of the users affected by this issue seem to be Proxmox VE users, there is at least one user who recently reported seeing this issue with libvirt on Debian 12 [5]. According to the github thread, the issue is not present on RH or Nutanix. So there may to be some factor at play here which makes the issue more likely to trigger on Proxmox VE (or Debian in general?), but it's not clear yet what that might be. It's unlikely to be our downstream QEMU, because the issue also triggers with upstream QEMU [3].

[1] https://forum.proxmox.com/threads/r...device-system-unresponsive.139160/post-691337
[2] https://forum.proxmox.com/threads/r...device-system-unresponsive.139160/post-692230
[3] https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2220730292
[4] https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2284118004
[5] https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2212543280
Click to expand...
Hi @fweber, agreed.

My only little worries are about whether Vadim's conclusion is not too "simple", i.e. limited in scope.

As of me, this issue is a stochastic problem, i.e. based on probability of incidence and here the @bbgeek17 's comment regarding "race condition" is absolutely correct. But I'm afraid if we will limit our focus to the "virtual queue size" only, still there may persist something wrong between 0.1.208 and 0.1.215.
 
  • Like
Reactions: carles89
fweber said:
According to the github thread, the issue is not present on RH or Nutanix. So there may to be some factor at play here which makes the issue more likely to trigger on Proxmox VE (or Debian in general?), but it's not clear yet what that might be. It's unlikely to be our downstream QEMU, because the issue also triggers with upstream QEMU [3].
Click to expand...
If I'm not mistaken, both RH and Nutanix use older kernels, so we might be bleeding a bit here. If unresolved, the problem is likely to become more apparent for them at some stage too. So I now think the propensity for the fault to occur on Debian is just an exacerbation of a driver issue that doesn't show until properly load tested.

RoCE-geek said:
My only little worries are about whether Vadim's conclusion is not too "simple", i.e. limited in scope.

As of me, this issue is a stochastic problem, i.e. based on probability of incidence and here the @bbgeek17 's comment regarding "race condition" is absolutely correct. But I'm afraid if we will limit our focus to the "virtual queue size" only, still there may persist something wrong between 0.1.208 and 0.1.215.
Click to expand...
There is definitely a problem with the driver. I just posted some of my findings on GitHub, and I think we are getting close to the root cause.

The fault is easily reproducible but presents in different ways depending on platform and configuration. Looking at I/O plots whilst reproducing the fault has been quite telling and informative, especially whilst making changes to the driver code. I thought I had cracked it a couple of times with logically plausible solutions that made it past one or two reproducer runs, only to see it fail horribly on a third run. The interval between reproducer runs also had an impact.

Anyway, I think we're getting close to solving it... 8^d
 
Last edited:
  • Like
Reactions: RoCE-geek and carles89
benyamin said:
I so now think the propensity for the fault to occur on Debian is just an exacerbation of a driver issue that doesn't show until properly load tested.
Click to expand...
This is also my feeling. It'd be good to post a link to this thread for others on github to see what are our findings and experiences here.

There is definitely a problem with the driver.
Click to expand...
For sure, but I mean to keep focus on all related changes in 0.1.215, not only to the one Vadim has commented.
It's still needed to have both eyes open, because the first suspect may not yet be the main culprit.

The fault is easily reproducible but presents in different ways depending on platform and configuration.
Click to expand...
Exactly, but I'm not sure if everyone on github is aware of.
 
  • Like
Reactions: carles89
RoCE-geek said:
Hi @fweber, agreed.

My only little worries are about whether Vadim's conclusion is not too "simple", i.e. limited in scope.

As of me, this issue is a stochastic problem, i.e. based on probability of incidence and here the @bbgeek17 's comment regarding "race condition" is absolutely correct. But I'm afraid if we will limit our focus to the "virtual queue size" only, still there may persist something wrong between 0.1.208 and 0.1.215.
Click to expand...
benyamin said:
If I'm not mistaken, both RH and Nutanix use older kernels, so we might be bleeding a bit here. If unresolved, the problem is likely to become more apparent for them at some stage too. I so now think the propensity for the fault to occur on Debian is just an exacerbation of a driver issue that doesn't show until properly load tested.
Click to expand...
It is certainly possible that the host kernel is a factor. Would be interesting to check whether older Proxmox VE kernels (e.g. 5.15) show the same behavior.

I tested some virtio-scsi multiqueue settings today, see my post [1] on the github issue. So far, I wasn't able to reproduce the issue with virtio-win 0.1.262 if I limit the number of VirtIO SCSI queues to 3 by setting num_queues=1 for the virtio-scsi-pci device. On Proxmox VE, this can either be done by dumping the QEMU command line via qm showcmd VMID --pretty and editing it. The same effect can be currently achieved by activating CPU hotplug (though this is not intentional and likely to change in the future [2]).

To everyone who can reliably reproduce the issue: Can you try to shut down the VM, activate "CPU hotplug" in the VM Options, start the VM and check whether you can still reproduce the issue?

While the VM is running, you can check the number of queues using a QEMU monitor command (Monitor tab in the GUI):
First, find the correct device:
Code: 
# info virtio
/machine/peripheral/virtioscsi0/virtio-backend [virtio-scsi]
Then, query that device:
Code: 
# info virtio-status /machine/peripheral/virtioscsi0/virtio-backend
And check for num_vq.

With default settings, num_vq should be equal to #vcpus + 2. With num_queues=1 or CPU hotplug enabled, it should be 3.

[1] https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2289086055
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=4295#c5
 
  • Like
Reactions: _gabriel and RoCE-geek
fweber said:
On Proxmox VE, this can either be done by dumping the QEMU command line via qm showcmd VMID --pretty and editing it.
Click to expand...
You just ended my days of using ps aux | grep <VMID>/<VM name>/<VM path to image>. 8^D


So I checked mine after enabling CPU hotplug and confirmed with info virtio-status ... that my num_vqs decreased from 6 to 3.
On v262 and virtio-scsi master I was unable to reproduce the fault using simultaneous 4K and 64K blocks.

I also checked with v208 to compare I/O plots. They were very similar as seen below, but did not exceed 3.5GiB. The plots are from Process Explorer's System Information I/O tab, so the top one is I/O (considered all I/O except console), the middle one is Network and the bottom one is Disk.

v208:

io_4k+64k_v208_3vqs.jpg

master:

io_4k+64k_master_3vqs.jpg

However, turning off CPU hotplug and returning to 6 VQs on v208 gives 4.1GiB+ and a much smoother ride on disk:

io_4k+64k_v208_6vqs.jpg

So whilst informative as to what's going on, it is important to get this fixed in the driver.
 
  • Like
Reactions: fweber
fweber said:
To everyone who can reliably reproduce the issue: Can you try to shut down the VM, activate "CPU hotplug" in the VM Options, start the VM and check whether you can still reproduce the issue?
Click to expand...
unable to reproduce when CPU is ticked from Hotplug section of VM Options. ( virtio-win version = 0.1.240 ).
 
  • Like
Reactions: fweber
For anyone interested, I've dropped a patch in the GitHub issue:
https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2290163018

Until a release is forthcoming (presuming it passes muster), you will need to build it from source and enable TESTSIGNING in Windows. This might be suitable for some test and development environments.

Please note the post immediately below that one re performance and adapter / queue multiplexing.
 
  • Like
Reactions: _gabriel and Falk R.
@fweber, FWIW, I decided to check the system+boot disk with my reproducer with num_vqs at 3 via the CPU hotplug method. My previous tests were on a number of other connected disks.

Performance was bad enough to shorten the test to 15 seconds. Both v208 and master were demonstrably bad. The patched version wasn't great either but it was noticeably better.

I think this has sorted the issue out, but note I haven't checked with disks under control of the QEMU global mutex. It will be interesting to see how things perform under aio=io_uring and aio=native now.
 
  • Like
Reactions: bbgeek17 and quanto11
benyamin said:
@bbgeek17, maybe you and the team at Blockbridge can spin this up in the lab and see if things have improved...?
Click to expand...
Hi @benyamin , to clarify you are looking for a test with modifications only from this comment: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2291200058 ?

I think we have a functional build environment and might be able to give it a spin in spare time.

Thanks for your efforts!

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Thanks for the testing and reporting the results!

benyamin said:
@fweber, FWIW, I decided to check the system+boot disk with my reproducer with num_vqs at 3 via the CPU hotplug method. My previous tests were on a number of other connected disks.

Performance was bad enough to shorten the test to 15 seconds. [...]
Click to expand...
Just so I understand correctly: The performance with num_vqs=3 is bad when running the reproducer on the boot disk, but it is still the case that you haven't seen the issue (vioscsi warnings, IO lockups, ...) with num_vqs=3, correct?

I'll also look at the recent posts in the github issue today.
 
fweber said:
Just so I understand correctly: The performance with num_vqs=3 is bad when running the reproducer on the boot disk, but it is still the case that you haven't seen the issue (vioscsi warnings, IO lockups, ...) with num_vqs=3, correct?
Click to expand...
Yes that is correct. There are no vioscsi or kvm warnings.

That being said, the performance is so bad the VM appears to hang. I would think that if the reproducer ran for a time far exceeding TimeoutValue or IoTimeoutValue, there might be vioscsi warnings, maybe even a bus reset. The difference here would be that the VM would perhaps recover after the bus reset. The diskspd process(es) would likely be stuck with missing filesystem handles.

Again, there was no issue running on disks in separate iothreads, i.e. on different adapters. I think it is just queue saturation whilst guest system processes are also trying to access the boot disk.
 
  • Like
Reactions: fweber
fweber said:
Thanks for the testing and reporting the results!


Just so I understand correctly: The performance with num_vqs=3 is bad when running the reproducer on the boot disk, but it is still the case that you haven't seen the issue (vioscsi warnings, IO lockups, ...) with num_vqs=3, correct?

I'll also look at the recent posts in the github issue today.
Click to expand...
Hi @fweber,

I've similar findings as @benyamin, but not exactly the same.

Tests done on my VM102 (https://forum.proxmox.com/threads/r...device-system-unresponsive.139160/post-691337, but the corresponding PVE node is no longer empty)

Both scsi drives on aio=io_uring.

1. VirtIO 0.1.208 (vgs=6): RDP is stable, IO stable/max, no issues
2. VirtIO 0.1.208 + CPU hotplug (vgs=3): huge RDP hangs, IO works, but lower Q32T16 Read performance
3. VirtIO 0.1.240 + CPU hotplug (vgs=3): medium RDP hangs, IO works, but lower Q32T16 Read performance
4. VirtIO 0.1.240 (vgs=6): buggy as reported

As always, using RND 4K Q32T16 test was super-easy to invoke the bug.

So, there are no IO hangs / scsi alerts (with CPU hotplug / vgs = 3) on 0.1.240, but while the VM works, it appears to be unresponsive for up to 10s (for both 0.1.208 and 0.1.240), but data flow continues. So from user perspective it's buggy, but in reality it works (with lower IO performance) and the virtio problem is mitigated. "RDP hang" is not the exact definition - just the session seemed to be hanged, although this could be caused by multiple factors in this setup (some general resource congestion is evident here).

Long story short: This is a good case for debug, but not suitable setup for production.

Update: to be clear, iothread=1, SCSI Single and the tested drive was scsi1.
 
Last edited:
  • Like
Reactions: carles89 and _gabriel
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back