Redhat VirtIO developers would like to coordinate with Proxmox devs re: "[vioscsi] Reset to device ... system unresponsive"

[benyamin]

The PhysicalBreaks registry entry actually needs to be set to less than the max_segments size for each backing block device.
This can be determined by issuing the command grep "" /sys/block/*/queue/max_segments.
If, for example, your block device has a max_segments of 60, it should be set to no more than 59, but one might consider setting it to 32 instead.
More info: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/827

This could be programmatically determined and applied to the qemu command line, per this mention.

 

[Max2048]

IIRC the proxmox developer @fweber is in contact with Redhat developer vrozenfe (https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756). Is there any progress you can share with us?

 

[benyamin]

IIRC the proxmox developer @fweber is in contact with Redhat developer vrozenfe (https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756). Is there any progress you can share with us?

Also @fiona in https://github.com/virtio-win/kvm-guest-drivers-windows/issues/623.

In that issue, a mention of downgrading to version 0.1.204 sometimes resolved the problem.

It does appear that the problem is universally solved by using aio=threads and iothreads=1 per this bug report.

 

[Max2048]

The "universal workaround" seems to only apply to local storage backends, but NOT CEPH. Thanks for the link though.

 

[benyamin]

The "universal workaround" seems to only apply to local storage backends, but NOT CEPH. Thanks for the link though.

lulz, yes, universally solved for local storage. 8^d
For CEPH, I guess a workaround could be to use LSI or maybe even virtio-block.
I think there is also a CEPH HA iSCSI implementation. I'm not sure if it does active/active so YMMV. I use tgt in an active/active configuration, and it works ok. In my experience, LIO is usually only compiled in kernel as active/passive. VMs are then configured for iscsi-boot. Using virtio-scsi, et al., would be A LOT simpler and permit you to retain Proxmox disk management integration, including PBS I imagine.

 

[benyamin]

In my environment, It seemed to me that VMs were more likely to fail with each additional iothread, i.e. each additional disk iothread=1 and a VirtIO SCSI single adapter. There seems to be a proportional, perhaps even a linear relationship, between the number of iothreads and the probability of failure. I also, had one VM using a PVSCSI adapter, which was even more likely to fail.

Further reference (cross post): https://forum.proxmox.com/threads/h...-all-vms-using-virtio-scsi.124298/post-659717

 

[fiona]

For those people that run into this error message during/after backup, Proxmox VE 8.2 has the possibility to do backup fleecing, which can be configured node-wide in /etc/vzdump.conf or for a given backup job in the Advanced tab in the UI, when editing the job. With that, there should be no requests that will complete after the VirtIO driver expects.

If the issue occurs independently of backups, it might be the other kind of issue @fweber could reproduce https://github.com/virtio-win/kvm-guest-drivers-windows/issues/756#issuecomment-2012089151

 

[Max2048]

Yes. for me it's indepdently of backups. Interestingly the VM can hang when there is "medium" I/O like 50 MB/s, but during backups with 400-500 MB/s it's fine.