Passed-through GPU stuttering

[stewart]

Hello.

I've got an instance of VE running 7.2-3 and, amongst other VMs, run a Windows 11 virtual machine as my desktop. I have passed through my GPU and it works to 95%. However, I find that when I run higher-spec games, I suffer with parallel video and audio stuttering. I know my machine can run these without problem as I have an SSD with a Windows OS on to test and there are no issues when not running Proxmox. I use GTA V to test as it's an old game and I can be sure it's not due to a lack of resources. Checking performance stats on the VM and hypervisor show nothing being highly utilised.

My hardware:

Intel i7-10700KF @ 3.80GHz, 16 cores
32GB DDR4 Memory
500GB Kingston SSD
4 x 4TB WD Red HDD
NVIDIA GeForce GTX 1050 Ti (GigaByte) - Passed-through GPU
NVIDIA GeFroce GTX 750 Ti (GigaByte) - VE GPU
MSI Z590 PRO WiFi

The Windows 11 VM:


I've run 'MSI_util_v3' and updated both the audio controller and GPU to use 'msi' with no effect.



I'm not an expert on hypervisors or Linux (I'm a network guy) so I'm not completely sure what other information would be needed to help troubleshoot this, but I greatly appreciate any help you can offer.

 

[leesteken]

Try 4 cores and see if it stutters less. If it is smoother (but slower) try 6 cores. Maybe try half the memory too. If you want a smoother experience, you want to make sure to not stress any part of the system.

 

[stewart]

Try 4 cores and see if it stutters less. If it is smoother (but slower) try 6 cores. Maybe try half the memory too. If you want a smoother experience, you want to make sure to not stress any part of the system.

I've decreased the memory to 12 GB, but I'm having problems changing the Processors on the hardware screen while the VM is stopped -- the pop-up box does nothing when I press the OK button. It won't let me change the number of cores. Any suggestions on that front?

 

[leesteken]

I've decreased the memory to 12 GB, but I'm having problems changing the Processors on the hardware screen while the VM is stopped -- the pop-up box does nothing when I press the OK button. It won't let me change the number of cores. Any suggestions on that front?

That's really weird. You can change the VM configuration file with nano /etc/pve/qermu-server/100.conf.

 

[mr44er]

Try the vbios-update from nvidia first: https://support.nvidia.eu/hc/de/art...e-Update-für-DisplayPort-1-3-und-1-4-Displays

It does a lot under the hood, not only for DisplayPort. Complete changelog would be nice, but haven't found it. On my 1050Ti it reduced flickering on HDMI and it is overall more stable on passthrough. (sometimes no screen at all when rebooting vm, now it just works)
I have my GPU passthroughed to my FreeBSD desktop, sometimes audio stuttering on 4k youtube-videos, but this is clearly the resource limit of the machine.

Bad thing is, that it only runs on Windows. If you have some spare machine, I would put the GPUs there just to flash it, one after another.
I would not try to flash this with running passthrough.

 

[stewart]

That's really weird. You can change the VM configuration file with nano /etc/pve/qermu-server/100.conf.

Thanks for the help there. I've changed the memory down to 12 GB and put it down to 4 cores, but without much noticeable improvement. Out of curiousity: why would having more resources available to a VM cause it to stutter?

Try the vbios-update from nvidia first: https://support.nvidia.eu/hc/de/articles/360000024200-Grafik-Firmware-Update-für-DisplayPort-1-3-und-1-4-Displays

It does a lot under the hood, not only for DisplayPort. Complete changelog would be nice, but haven't found it. On my 1050Ti it reduced flickering on HDMI and it is overall more stable on passthrough. (sometimes no screen at all when rebooting vm, now it just works)
I have my GPU passthroughed to my FreeBSD desktop, sometimes audio stuttering on 4k youtube-videos, but this is clearly the resource limit of the machine.

Bad thing is, that it only runs on Windows. If you have some spare machine, I would put the GPUs there just to flash it, one after another.
I would not try to flash this with running passthrough.

Before I did the above I booted into a spare SSD with Win10 on and ran the update you recommended. My 1050Ti was already up to date, but it did update the 750 Ti with no noticeable improvement sadly.

 

[mr44er]

On your cpu the fix43 against error code 43 shouldn't be necessary anymore. AFAIK nvidia doesn't block passthrough on consumer gpus since last year.
Which brings me to the question which nvidia-driver you use. The one that windows pulled automatically (mostly months old) or directly the newest from https://www.nvidia.de/Download/index.aspx?lang=us (better option) ?
Besides that you could try without any cpu flags, only host and numa=0, maybe this gives another result.

 

[leesteken]

Thanks for the help there. I've changed the memory down to 12 GB and put it down to 4 cores, but without much noticeable improvement. Out of curiousity: why would having more resources available to a VM cause it to stutter?

If all your physical cores are busy (hyper-threads are not really additional cores) to the max then the system might be too busy to run your gaming VM immediately, causing delays. This can happen for any part of the system: CPU, memory (when swapping), disk and network I/O. Remember that other VMs and Proxmox also need processing power. Sometimes people expect to be able to pass 100% of everything to a single VM and run into this, but in your case this does not appear to be the root cause of the stuttering.

EDIT: This thread and the thread(s) it references also mention disk and network I/O, when run in the same QEMU thread, can cause scheduling issues and maybe stuttering.

 

[stewart]

I've been paying more attention to the issue lately, and I've noticed that when I'm torrenting or downloading pretty much anything, I get heavy stuttering to the point it's hard to keep track of my mouse pointer.

On your cpu the fix43 against error code 43 shouldn't be necessary anymore. AFAIK nvidia doesn't block passthrough on consumer gpus since last year.
Which brings me to the question which nvidia-driver you use. The one that windows pulled automatically (mostly months old) or directly the newest from https://www.nvidia.de/Download/index.aspx?lang=us (better option) ?
Besides that you could try without any cpu flags, only host and numa=0, maybe this gives another result.

I was using a driver from Oct 22, but I updated it to the latest as you recommended but without any improvement.

The below is my Win11 VM config. Are you suggesting I get rid of just hv-vendor-id=Fix43,flags=+pcid?

agent: 1,fstrim_cloned_disks=1
args: -machine type=pc-q35-6.2,kernel_irqchip=on,virtio-scsi-single,iothread=1,aio=threads
balloon: 0
bios: ovmf
boot: order=virtio0
cores: 4
cpu: host,hidden=1,hv-vendor-id=Fix43,flags=+pcid
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00,pcie=1,x-vga=1
hostpci1: 0000:00:14.0,pcie=1
hotplug: disk,network,usb,cpu
ide2: none,media=cdrom
machine: pc-q35-6.2
memory: 12288
meta: creation-qemu=6.2.0,ctime=1661523193
name: Win11
net0: virtio=0E:96:FE:D2:E1:B6,bridge=vmbr0
numa: 1
onboot: 1
ostype: win11
smbios1: uuid=91e9e5d8-af8c-4631-ab95-acd5a6998865
sockets: 1
tpmstate0: local-lvm:vm-100-disk-1,size=4M,version=v2.0
usb0: host=1-4.2.1,usb3=1
usb1: host=1-4.2.2,usb3=1
usb2: host=1-4.2.3,usb3=1
usb3: host=1-2,usb3=1
usb4: host=1-14,usb3=1
vga: none
virtio0: local-lvm:vm-100-disk-2,size=128G
virtio1: ZFSData01:100/vm-100-disk-0.qcow2,size=5588G
vmgenid: 366c6917-db04-419e-aebb-b559fd9c4ccb

If all your physical cores are busy (hyper-threads are not really additional cores) to the max then the system might be too busy to run your gaming VM immediately, causing delays. This can happen for any part of the system: CPU, memory (when swapping), disk and network I/O. Remember that other VMs and Proxmox also need processing power. Sometimes people expect to be able to pass 100% of everything to a single VM and run into this, but in your case this does not appear to be the root cause of the stuttering.

EDIT: This thread and the thread(s) it references also mention disk and network I/O, when run in the same QEMU thread, can cause scheduling issues and maybe stuttering.

Thank you for that explanation, it's helpful to know. Regarding your edit: I tried changing my args to args: -machine type=pc-q35-6.2,kernel_irqchip=on,virtio-scsi-single,iothread=1,aio=threads to include the last three arguments. I had to guess where they went, but on restart it doesn't seem any better (worse if possible).

 

[mr44er]

Are you suggesting I get rid of just hv-vendor-id=Fix43,flags=+pcid?

Yes and also switching numa off. This should only be necessary on multisocket-boards and maybe this could generate a bottleneck 'coz you only have one socket. I know it's time consuming to test after every single bit of setting.

Can you post a lspci -vv ?

Is the taskmgr in windows showing any suspicious activity when the vm is idling? Thinking about some coinmining-trojan malware shit. Just to be sure, just to rule out.

 

[stewart]

Yes and also switching numa off. This should only be necessary on multisocket-boards and maybe this could generate a bottleneck 'coz you only have one socket. I know it's time consuming to test after every single bit of setting.

Can you post a lspci -vv ?

Is the taskmgr in windows showing any suspicious activity when the vm is idling? Thinking about some coinmining-trojan malware shit. Just to be sure, just to rule out.

I removed hv-vendor-id=Fix43,flags=+pcid and changed numa to numa=0. Rebooted, and no noticeable effect.

The output is too long to post here, but see below:
https://pastebin.com/AzhywNLN

I have been variably keeping an eye on the taskmgr but the results look normal from my experience. Fluctuations are minimal. I'll check the other machines I have on here and let you know if I see anything unusual, but I'm doubtful.

 

[mr44er]

I removed hv-vendor-id=Fix43,flags=+pcid and changed numa to numa=0. Rebooted, and no noticeable effect.

Ok, thats good. And your nvidia shows up good without error 43 in device manager?

Also the taskmgr looks good.

The output is too long to post here, but see below:

I wondered what devid 00:14.0 was. It shows thats an usb3-controller. Double passthrough of already passthroughed things could be the problem here. Either you choose to passthrough single chosen USB devices OR the whole controller.

Edit: something more -> devid 00:14.2 RAM memory: Intel Corporation Device 43ef (rev 11) + Kernel driver in use: vfio-pci
This should at any point and always stay on the host. Looks the iommu-groups are suboptimal at this time.

Pls remove the whole device 00:14.0 from the vm, reboot the node and try again. I think this it it.

 

[stewart]

Ok, thats good. And your nvidia shows up good without error 43 in device manager?

Also the taskmgr looks good.


I wondered what devid 00:14.0 was. It shows thats an usb3-controller. Double passthrough of already passthroughed things could be the problem here. Either you choose to passthrough single chosen USB devices OR the whole controller.

Edit: something more -> devid 00:14.2 RAM memory: Intel Corporation Device 43ef (rev 11) + Kernel driver in use: vfio-pci
This should at any point and always stay on the host. Looks the iommu-groups are suboptimal at this time.

Pls remove the whole device 00:14.0 from the vm, reboot the node and try again. I think this it it.

When first setting up the passthrough a few months ago, I did have a persistent issue with code 43 errors. I ended up paying a contractor to help fix the issue -- I can't tell you what he did, though, I've lost contact. But no current code 43 error.

As far as I can remember, 00:14.0 was to passthrough my USB devices attached to the machine. Now I've removed that, I can't use my peripherals on the VM and also my audio as it was USB. What is the best way to pass through my various USB peripherals?

I've been unable to test if this has fixed the issue because half of the problem was audio which I no longer have access to.

 

[mr44er]

But no current code 43 error.

Perfect. Some background: This was just a software lock (driver detected if its running in vm), so nvidia wanted to sell you pricey quadro cards if you wanted to use passthrough. From my point it was complete bullshit, but anyway they decided last year to remove that limitation and drivers from that should just work, without any workarounds and fixes for the fix to hide vm usage to the driver.

As far as I can remember, 00:14.0 was to passthrough my USB devices attached to the machine. Now I've removed that, I can't use my peripherals on the VM and also my audio as it was USB.

Uh...ok, shouldn't happen with what you have configured.

What is the best way to pass through my various USB peripherals?

Preferred from my point, performancewise and if you don't need these USB slots (wired to the USB3-controller) on the host, you just passthrough the whole controller. Every USB device then physically plugged in is 'in' the vm automatically.
And here comes the problem (I think it is): you only have one controller and this is suboptimal wired with 00:14.2 RAM memory: Intel Corporation Device 43ef and also in the same iommu-group 3. That is always bad for passthrough, because it pulls both devices into the vm. And I strongly suggest to leave 00:14.2 RAM memory: Intel Corporation Device 43ef on the host (I think this is the root cause for the stuttering)

Check the attachment, it should you get back on track. Also try both variants of usb.png.

For now I'm out of ideas, maybe some other member has another idea.

 

[stewart]

Perfect. Some background: This was just a software lock (driver detected if its running in vm), so nvidia wanted to sell you pricey quadro cards if you wanted to use passthrough. From my point it was complete bullshit, but anyway they decided last year to remove that limitation and drivers from that should just work, without any workarounds and fixes for the fix to hide vm usage to the driver.

I heard about that -- thieving scoundrel.

Uh...ok, shouldn't happen with what you have configured.

My life...

Preferred from my point, performancewise and if you don't need these USB slots (wired to the USB3-controller) on the host, you just passthrough the whole controller. Every USB device then physically plugged in is 'in' the vm automatically.
And here comes the problem (I think it is): you only have one controller and this is suboptimal wired with 00:14.2 RAM memory: Intel Corporation Device 43ef and also in the same iommu-group 3. That is always bad for passthrough, because it pulls both devices into the vm. And I strongly suggest to leave 00:14.2 RAM memory: Intel Corporation Device 43ef on the host (I think this is the root cause for the stuttering)

Check the attachment, it should you get back on track. Also try both variants of usb.png.

For now I'm out of ideas, maybe some other member has another idea.

I really appreciate your help on this so far. I'll have a play with getting the USB devices working again tomorrow. I can then test to see if this has had any effect. If I remember correctly, however, I think I did this to minimise similar stuttering before...

I'll let you know when I've had a chance to test.

Merry Xmas.

 

[mr44er]

I heard about that -- thieving scoundrel.

And with this info...you paid your contractor for hv-vendor-id=Fix43,flags=+pcid

If I remember correctly, however, I think I did this to minimise similar stuttering before...

If it is how I think, there are three problems. 1. double-passthrough in general (wondering that it's working anyway and should be avoided, regardless of the stuttering) and 2. the dependency/wiring of 00:14.0 <-> 00:14.2 and 3. the same iommu-group 3

really appreciate your help on this so far.

Knowing how to fix problems helps me and others in the future

Happy holidays!

Edit:
Just another idea came up: Try to output the audio over hdmi/dp from your nvidia. If this is without stuttering then it could be that your usb audio device just doesn't work good with passthrough.

 

[stewart]

And with this info...you paid your contractor for hv-vendor-id=Fix43,flags=+pcid

Wouldn't surprise me. Took him 2 days of faffing, but did get it working to his credit.

If it is how I think, there are three problems. 1. double-passthrough in general (wondering that it's working anyway and should be avoided, regardless of the stuttering) and 2. the dependency/wiring of 00:14.0 <-> 00:14.2 and 3. the same iommu-group 3

Edit:
Just another idea came up: Try to output the audio over hdmi/dp from your nvidia. If this is without stuttering then it could be that your usb audio device just doesn't work good with passthrough.

Did some testing this morning per your recommendations. Removing the PCI passthrough and just relying on the USB devices (both passed through as device and port separately) makes the issue 10x worse. Removing all USB devices and just reverting to passing the PCI through, the issue remains at initial stuttering levels +/-.

Things are starting to make sense in my head now and the problem is becoming clearer thanks to your help. My RAM and USB controller are both in IOMMU group 3 it seems, which I can understand now is causing this problem. Doing some very brief research, I've found that you can technically split devices into other IOMMU groups with ACS (it seems) but I suppose only to a certain extent. This is still not 100% clear to me. Either that, or do you know if it's possible to have a secondary USB controller... somehow?

Thinking of alternatives: I use Mouse Without Borders to share mouse/keyboard between my work laptop and this machine, and I've found a way to isolate the mouse into a window so it doesn't travel to other screens. My only problem then comes down to audio. You mentioned I might be able to output from HDMI/DP but I'm not sure if that'll help me using a USB headset?

This is the price we pay for relying on USB for everything.

 

[mr44er]

Removing the PCI passthrough and just relying on the USB devices (both passed through as device and port separately) makes the issue 10x worse. Removing all USB devices and just reverting to passing the PCI through, the issue remains at initial stuttering levels +/-.

Ok, this narrows down things a bit.

I've found that you can technically split devices into other IOMMU groups with ACS (it seems) but I suppose only to a certain extent. This is still not 100% clear to me.

Same boat here, I read that here and there and not always leads to success. There is another knob with the kernel (pci=realloc=off / on) you can try, but pls be careful and read twice for full understanding how it is done. I don't want to be responsible for a non booting node.
https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_edit_kernel_cmdline
https://forum.proxmox.com/threads/issue-with-bcm57840-on-proxmox-5-2.47063/
https://forums.servethehome.com/index.php?threads/zfs-native-on-ubuntu-16-04-lts.9117/#post-83992

Either that, or do you know if it's possible to have a secondary USB controller... somehow?

Yes, I have one of these addon cards. It's somewhat older but it just works. It's important that the card is minimum USB3/xhci (USB2/ehci does not work at all), other chips than VIA VL805 (sold under different rebrands) may work, but better to search around here and there.

05:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01) (prog-if 30 [XHCI])
    Subsystem: VIA Technologies, Inc. VL805 USB 3.0 Host Controller
    Flags: bus master, fast devsel, latency 0, IRQ 157, NUMA node 0, IOMMU group 15
    Memory at fceff000 (64-bit, non-prefetchable) [size=4K]
    Capabilities: [80] Power Management version 3
    Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
    Capabilities: [c4] Express Endpoint, MSI 00
    Capabilities: [100] Advanced Error Reporting
    Kernel driver in use: vfio-pci
    Kernel modules: xhci_pci

I have one from CSL, looks like this: https://www.amazon.de/CSL-Express-Controllerkarte-Schnittstellenkarte-Treiber/dp/B00F9XGPTI
Dunno if it's just a stock photo, because the chip says VL123

Here somebody is using Inateck card: https://www.nicksherlock.com/2018/11/my-macos-vm-proxmox-setup/

My only problem then comes down to audio. You mentioned I might be able to output from HDMI/DP but I'm not sure if that'll help me using a USB headset?

Was meant for testing, this primarily helps to pinpoint where the stuttering comes from exactly. If you can run your game without audio stuttering over the nvidia-output, you know for sure, that your machine with this vmconfig is good overall. If your mouse and keyboard are going smooth also, this would be good so far.
Keep an eye also on the behaviour of keyboard and mouse with and without the usb headset plugged in. I'm thinking of not only audio stuttering, but also slightly stuttering or lags on keyboard/mouse input. That's not directly noticable, because keyboard and mouse have small bandwidth needs and are mostly only USB2. USB audio devices like a headset needs more bandwidth than keyboard and mouse together.

So at this point I don't know really what to recommend. Spending money on a USB3 card could be senseless if it does not help. Also it could be, that everything works as expected and just the USB headset disturbs everything.

 

[leesteken]

If you want to split (all) IOMMU groups (and don't mind the security/isolation issues or just for testing), you could try the pcie_acs_override=downstream,multifunction kernel parameter to make Proxmox ignore the actual groups. No guarantees through.

 

[stewart]

If you want to split (all) IOMMU groups (and don't mind the security/isolation issues or just for testing), you could try the pcie_acs_override=downstream,multifunction kernel parameter to make Proxmox ignore the actual groups. No guarantees through.

Thanks for that suggestion. I went to try it today, and it seems that (as per an earlier message) my contractor already implemented the patch (without mentioning security concerns!)

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on amd_iommu=on modprobe.blacklist=xhci_hcd modprobe.blacklist=nvidia-gpu modprobe.blacklist=nouveau modprobe.blackli>
GRUB_CMDLINE_LINUX="intel_iommu=on amd_iommu=on iommu=pt rd.driver.pre=vfio-pci video=efifb:off kvm.ignore_msrs=1 kvm.allow_unsafe_assigned_interrupts=1 pcie_acs_override=downstream,multifunction vfio-pci.ids=10de:1c82,10de:0fb9,10de:0fbc vfio-pci.disable_vga=1 video=simplefb:off nofb nomodeset"

But still I have bundled IOMMU groups. Shouldn't these be split with the ACS patch applied?

IOMMU Group 0:
        00:00.0 Host bridge [0600]: Intel Corporation Device [8086:9b43] (rev 05)
IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
IOMMU Group 2:
        00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU Group 3:
        00:14.0 USB controller [0c03]: Intel Corporation Device [8086:43ed] (rev 11)
        00:14.2 RAM memory [0500]: Intel Corporation Device [8086:43ef] (rev 11)
IOMMU Group 4:
        00:16.0 Communication controller [0780]: Intel Corporation Device [8086:43e0] (rev 11)
IOMMU Group 5:
        00:17.0 SATA controller [0106]: Intel Corporation Device [8086:43d2] (rev 11)
IOMMU Group 6:
        00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:43c0] (rev 11)
IOMMU Group 7:
        00:1b.4 PCI bridge [0604]: Intel Corporation Device [8086:43c4] (rev 11)
IOMMU Group 8:
        00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:43b8] (rev 11)
IOMMU Group 9:
        00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:43bc] (rev 11)
IOMMU Group 10:
        00:1c.5 PCI bridge [0604]: Intel Corporation Device [8086:43bd] (rev 11)
IOMMU Group 11:
        00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:43be] (rev 11)
IOMMU Group 12:
        00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:4385] (rev 11)
        00:1f.3 Audio device [0403]: Intel Corporation Device [8086:f0c8] (rev 11)
        00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:43a3] (rev 11)
        00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:43a4] (rev 11)
IOMMU Group 13:
        01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
IOMMU Group 14:
        01:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 15:
        03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2)
IOMMU Group 16:
        03:00.1 Audio device [0403]: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] [10de:0fbc] (rev a1)
IOMMU Group 17:
        05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
IOMMU Group 18:
        06:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 19:
        06:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 20:
        07:00.0 Network controller [0280]: Intel Corporation Device [8086:2725] (rev 1a)