IOMMU Passthrough Glitches Occasionally

[codejp3]

I know this not a Proxmox issue and beyond the scope of Proxmox support, but I'm hoping to get some advice on this.

I setup a Win11 QEMU VM and installed some music recording software. I have a direct-input box I use to plug microphones and instruments into to capture high quality audio. It is a USB unit. It did not work at all just adding the USB device to the VM, so I setup the PCIe USB card as an IOMMU passthrough device to the VM instead.

It works great most of the time, but every so often it just glitches out right in the middle of recording audio, or playing back audio while mixing. It is the only audio interface the VM has and is used for both input/output. The glitch seems to be related to the USB bus timeclock syncing. When it glitches, I just change the bitrate for the direct input box to a different setting and then switch it back and it works great again until the next glitch.

When it glitches, the Windows VM does not report any issues. There are no errors or warnings in Event Viewer and Device Manager does not show any device/driver issues for either the PCIe USB card, or the attached direct input audio interface.

The only issue to speak of with this setup is that once I enabled passthrough, I started getting kernel warnings for ignored/unhandled RDMSR and unhandled WRMS. It changed based on the CPU type I set for the VM:

kvm: kvm ignored rdmsr: 0x1c9 data 0x0
kvm: kvm ignored rdmsr: 0x680 data 0x0
kvm: kvm ignored rdmsr: 0x6c0 data 0x0

or

kvm_intel: kvm guest rIP 0xfffff80631e17102 Unhandled WRMSR(0x1d9) = 0x1

I was not able to find a CPU type that didn't have RDMSP/WRMS warnings, so I just set this to prevent the log from getting filled up with those:

# cat /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1 report_ignored_msrs=0

Other than that, there are no warning/errors to speak of on the host, or in the guest VM.


I know it has to do with the USB card PCIe passthrough, not the USB audio interface since the USB audio interface does not have this issue when plugged directly into a computer.



Setup Info:

The server is an older Dell R510
The USB card is ASMedia 3142 USB 3.1 card
The direct input box is a Focusrite Scarlett 2i2 Gen 4

I followed the instructions at https://pve.proxmox.com/wiki/PCI(e)_Passthrough to setup the PCIe card passthrough and blacklisted it from host system.

# lspci | grep ASMedia
08:00.0 USB controller: ASMedia Technology Inc. ASM2142/ASM3142 USB 3.1 Host Controller

# lspci -nn | grep ASMedia
08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142/ASM3142 USB 3.1 Host Controller [1b21:2142]

# lspci -k | grep -A 3 "ASMedia"
08:00.0 USB controller: ASMedia Technology Inc. ASM2142/ASM3142 USB 3.1 Host Controller
    Subsystem: Post Impression Systems. ASM2142/ASM3142 USB 3.1 Host Controller
    Kernel driver in use: vfio-pci
    Kernel modules: xhci_pci

# lsmod | grep vfio
vfio_pci               16384  1
vfio_pci_core          86016  1 vfio_pci
irqbypass              12288  10 vfio_pci_core,kvm
vfio_iommu_type1       49152  1
vfio                   57344  7 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                77824  1 vfio

# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.013746] ACPI: DMAR 0x00000000BF3B3668 0001D0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.013805] ACPI: Reserving DMAR table memory at [mem 0xbf3b3668-0xbf3b3837]
[    0.422646] DMAR: IOMMU enabled
[    0.980710] DMAR-IR: This system BIOS has enabled interrupt remapping
[    1.324546] DMAR: Host address width 40
[    1.324548] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    1.324559] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c90780106f0462 ecap f020fe
[    1.324562] DMAR: RMRR base: 0x000000bf4c8000 end: 0x000000bf4dffff
[    1.324564] DMAR: RMRR base: 0x000000bf4b1000 end: 0x000000bf4bffff
[    1.324565] DMAR: RMRR base: 0x000000bf4a1000 end: 0x000000bf4a1fff
[    1.324566] DMAR: RMRR base: 0x000000bf4a3000 end: 0x000000bf4a3fff
[    1.324568] DMAR: RMRR base: 0x000000bf4a5000 end: 0x000000bf4a5fff
[    1.324569] DMAR: RMRR base: 0x000000bf4a7000 end: 0x000000bf4a7fff
[    1.324570] DMAR: RMRR base: 0x000000bf4a9000 end: 0x000000bf4a9fff
[    1.324571] DMAR: RMRR base: 0x000000bf4c0000 end: 0x000000bf4c0fff
[    1.324573] DMAR: RMRR base: 0x000000bf4c2000 end: 0x000000bf4c2fff
[    1.324574] DMAR: ATSR flags: 0x0
[    1.324611] DMAR: No SATC found
[    1.324617] DMAR: dmar0: Using Queued invalidation
[    1.327947] DMAR: Intel(R) Virtualization Technology for Directed I/O

# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

# cat /etc/modprobe.d/vfio-pci.conf
options vfio-pci ids=1b21:2142

# cat /etc/modprobe.d/vfio-blacklist.conf
blacklist xhci_pci

# cat /etc/pve/local/qemu-server/222.conf
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 8
cpu: host
efidisk0: systems:vm-222-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:08:00,pcie=1
machine: pc-q35-8.1
memory: 94208
meta: creation-qemu=8.1.2,ctime=1705193241
name: DAW-Win11
net0: rtl8139=BC:24:11:9D:D2:9A,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: data:222/vm-222-disk-0.qcow2,iothread=1,size=256G
scsi1: data:222/vm-222-disk-1.qcow2,iothread=1,size=64G
scsi2: data:222/vm-222-disk-2.qcow2,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=4d86de7e-4bf7-4fd7-bcd8-5743c92581f5
sockets: 2
tpmstate0: systems:vm-222-disk-1,size=4M,version=v2.0
vga: virtio,memory=256
vmgenid: 93bf2666-5982-49c1-b66b-ee93702cc485

Like I said, it works great MOST of the time, but glitches out every now and then, and changing the audio bitrate setting for the USB audio interface seems to "reset it" and make it work again.

Any suggestions for eliminating the glitchy behavior?

 

[codejp3]

Possible culprit....

the dmesg dump above is misleading because it only shows the first line of DMAR-IR. Gave the impression that IR was enabled without any issues:

[    0.980710] DMAR-IR: This system BIOS has enabled interrupt remapping

The full message shows a possible problem:

# dmesg | grep -A 4 "DMAR-IR"
[    0.980710] DMAR-IR: This system BIOS has enabled interrupt remapping
               on a chipset that contains an erratum making that
               feature unstable.  To maintain system stability
               interrupt remapping is being disabled.  Please
               contact your BIOS vendor for an update

The BIOS is the latest version Dell offers for this server.


Digging in some more to see what else I can find, but I could see how having IR issues could cause the glitchy USB bus timeclock syn issue.

 

[codejp3]

Found a relevant link:
https://www.suse.com/support/kb/doc/?id=000018235

This is due to malfunctional firmware which flags to the operating system that it supports interrupt remapping despite the fact that is unable to handle it.

...and confirmed that this server suffers from the firmware issue:

# lspci -nn | grep -qE '8086:(340[36].*rev 13|3405.*rev (12|13|22))' && echo "Interrupt remapping is broken"
Interrupt remapping is broken

I think this may actually be the cause. It makes sense for the glitch I experience to be related to IR's not being handled properly.


The article I linked suggests:

Interrupt remapping should be disabled on systems with the above mentioned chipsets.

But then also says:

However, customers using PCI pass-through in KVM will additionally have to update their KVM configuration to no longer make use of PCI pass-through.

That would break the PCIe passthrough completely. Not really an option here.


The actual fix: contact the vendor and request a patch.
The problem with this fix: Dell isn't going to issue a patch for an out of service 13+ year old chipset.


I'm open to other suggestions (that don't include purchasing new hardware).

For now, I'm just going to keep rocking it as-is and deal with the glitches. Afterall, it works MOST of the time, and only glitches occasionally.