[SOLVED] Odd Passthrough Behavior( Audio )

[kromberg]

Hi everyone. I am new to proxmox and I think I have definitely drunk the Koolaid after someone recommended it over 'XXware'. I am trying to passthrough a PCIe soundcard to a linux guest. Basically things look like they successfully passthrough and are working fine with the guest on initial bootup. After a little bit of activity with the guest, the sound on it becomes choppy and/or stuck like it is having interrupt issues. I have done a bit of research, but nothing really seems to fix the issue. Here are the details of the setup:

Host
Hardware:

• Supermicro X10DRH-iT
• (2) E5-2680v3 CPUs
• 128GB DDR4-2133
• Sound Blaster X-Fi Titanium
• Quadro M2000

BIOS Settings:
Optimized defaults with these changes:

• Direct Cache Access "enable"
• Enable IOAH "enable"
• No Snoop "enable"
• Relaxed Ordering "enable"
• Above 4GB Decoding "enable"
• Maximum Payload "4096"
• Maximum Read Request "4096"

Proxmox: pve-manager/7.1-4/ca457116
/etc/default/grub file:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream"

/etc/modules file:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/iommu_unsafe_interrupts.conf file:
options vfio_iommu_type1 allow_unsafe_interrupts=1

/etc/modprobe.d/kvm.conf file:
options kvm ignore_msrs=1

/etc/modprobe.d/blacklist.conf file:
blacklist nouveau
blacklist nvidia
blacklist snd_ctxfi

/etc/modprobe.d/vfio.conf file:
options vfio-pci ids=10de:1430,10de:0fba,1102:000b disable_vga=1 # first two ids are the nvidia gpu and third is the sb card

lspci output( narrowed down ):
03:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)
04:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
05:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
08:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
09:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
81:00.0 Audio device: Creative Labs EMU20k2 [Sound Blaster X-Fi Titanium Series] (rev 04)
82:00.0 PCI bridge: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch (rev 0e)
86:00.0 VGA compatible controller: NVIDIA Corporation GM206GL [Quadro M2000] (rev a1)
86:00.1 Audio device: NVIDIA Corporation GM206 High Definition Audio Controller (rev a1)


Guest
Fedora 33

lspci output:
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
00:05.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
00:10.0 Audio device: Creative Labs EMU20k2 [Sound Blaster X-Fi Titanium Series] (rev 04)
00:11.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
00:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:13.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:14.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:15.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:1b.0 VGA compatible controller: NVIDIA Corporation GM206GL [Quadro M2000] (rev a1)
00:1c.0 Audio device: NVIDIA Corporation GM206 High Definition Audio Controller (rev a1)
00:1e.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
00:1f.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
01:1b.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)

The kernel does have the CONFIG_PCI_MSI flags enabled and/or set to 'Y'.

I am a little stumped on things here. The GPU and USB3 cards work perfectly fine with the passthrough behavior. Are there any other settings that might need to be set? Or is it just the card?

 

[kromberg]

OK, for good measure I added these two lines to /etc/modprobe.d/blacklist:

blacklist snd_pcm
blacklist snd_hda_intel

Got much better results. Got in about 5 minutes of usage with the sound card and then things broke again. Looking at the /var/log/messages file I saw this chuck here as the last entry right after the guest sound crashed:

Nov 25 08:23:12 odin kernel: [   78.368774] kvm_msr_ignored_check: 164 callbacks suppressed
Nov 25 08:26:56 odin kernel: [  302.997553] CPU: 39 PID: 8231 Comm: kvm Tainted: P           O      5.13.19-1-pve #1
Nov 25 08:26:56 odin kernel: [  302.997556] Hardware name: Supermicro SSG-6028R-E1CR12L/X10DRH-iT, BIOS 3.2 11/21/2019
Nov 25 08:26:56 odin kernel: [  302.997557] Call Trace:
Nov 25 08:26:56 odin kernel: [  302.997559]  <IRQ>
Nov 25 08:26:56 odin kernel: [  302.997562]  dump_stack+0x7d/0x9c
Nov 25 08:26:56 odin kernel: [  302.997571]  __report_bad_irq+0x3a/0xaf
Nov 25 08:26:56 odin kernel: [  302.997579]  note_interrupt.cold+0xb/0x60
Nov 25 08:26:56 odin kernel: [  302.997582]  handle_irq_event+0xaa/0xc0
Nov 25 08:26:56 odin kernel: [  302.997588]  handle_fasteoi_irq+0x7d/0x1c0
Nov 25 08:26:56 odin kernel: [  302.997591]  __common_interrupt+0x43/0xa0
Nov 25 08:26:56 odin kernel: [  302.997597]  common_interrupt+0x88/0xa0
Nov 25 08:26:56 odin kernel: [  302.997605]  </IRQ>
Nov 25 08:26:56 odin kernel: [  302.997606]  asm_common_interrupt+0x1e/0x40
Nov 25 08:26:56 odin kernel: [  302.997611] RIP: 0010:kvm_mmu_notifier_invalidate_range_start+0x19b/0x2b0 [kvm]
Nov 25 08:26:56 odin kernel: [  302.997674] Code: 4c 89 75 a8 48 89 45 b8 e8 72 d1 04 00 08 45 a0 48 8b 45 98 44 8b 90 0c 00 01 00 49 63 c2 49 83 c6 58 48 8d 0c 80 48 8d 04 48 <49> 8d 04 c4 49 39 c6 0f 82 65 ff ff ff 4d 89 fe 49 89 df 48 83 7d
Nov 25 08:26:56 odin kernel: [  302.997677] RSP: 0018:ffffb2b1c9b07a88 EFLAGS: 00000286
Nov 25 08:26:56 odin kernel: [  302.997679] RAX: 00000000000000e7 RBX: ffffb2b1cb1dd000 RCX: 0000000000000069
Nov 25 08:26:56 odin kernel: [  302.997680] RDX: ffff891111e82400 RSI: 00007f627d000000 RDI: ffffb2b1c9b07a08
Nov 25 08:26:56 odin kernel: [  302.997681] RBP: ffffb2b1c9b07b18 R08: ffff891111e82400 R09: 0000000000000004
Nov 25 08:26:56 odin kernel: [  302.997682] R10: 0000000000000015 R11: 0000000000000016 R12: ffff8917d4370010
Nov 25 08:26:56 odin kernel: [  302.997683] R13: 00007f78ca400000 R14: ffff8917d4370488 R15: 00007f78da400000
Nov 25 08:26:56 odin kernel: [  302.997685]  ? kvm_mmu_notifier_invalidate_range_start+0x17e/0x2b0 [kvm]
Nov 25 08:26:56 odin kernel: [  302.997713]  __mmu_notifier_invalidate_range_start+0x81/0x1a0
Nov 25 08:26:56 odin kernel: [  302.997717]  change_protection+0xc58/0xde0
Nov 25 08:26:56 odin kernel: [  302.997720]  ? common_interrupt+0x5a/0xa0
Nov 25 08:26:56 odin kernel: [  302.997724]  change_prot_numa+0x19/0x40
Nov 25 08:26:56 odin kernel: [  302.997729]  task_numa_work+0x1bf/0x310
Nov 25 08:26:56 odin kernel: [  302.997734]  task_work_run+0x6d/0xa0
Nov 25 08:26:56 odin kernel: [  302.997739]  xfer_to_guest_mode_handle_work+0x10b/0x110
Nov 25 08:26:56 odin kernel: [  302.997745]  kvm_arch_vcpu_ioctl_run+0x70d/0x1780 [kvm]
Nov 25 08:26:56 odin kernel: [  302.997780]  ? kvm_arch_vcpu_ioctl_run+0x5f2/0x1780 [kvm]
Nov 25 08:26:56 odin kernel: [  302.997812]  ? kvm_on_user_return+0x63/0xa0 [kvm]
Nov 25 08:26:56 odin kernel: [  302.997841]  kvm_vcpu_ioctl+0x247/0x5f0 [kvm]
Nov 25 08:26:56 odin kernel: [  302.997866]  ? exit_to_user_mode_prepare+0x37/0x1b0
Nov 25 08:26:56 odin kernel: [  302.997870]  ? syscall_exit_to_user_mode+0x27/0x50
Nov 25 08:26:56 odin kernel: [  302.997872]  ? do_syscall_64+0x6e/0xb0
Nov 25 08:26:56 odin kernel: [  302.997874]  __x64_sys_ioctl+0x91/0xc0
Nov 25 08:26:56 odin kernel: [  302.997878]  do_syscall_64+0x61/0xb0
Nov 25 08:26:56 odin kernel: [  302.997880]  ? syscall_exit_to_user_mode+0x27/0x50
Nov 25 08:26:56 odin kernel: [  302.997881]  ? do_syscall_64+0x6e/0xb0
Nov 25 08:26:56 odin kernel: [  302.997883]  ? do_syscall_64+0x6e/0xb0
Nov 25 08:26:56 odin kernel: [  302.997885]  ? syscall_exit_to_user_mode+0x27/0x50
Nov 25 08:26:56 odin kernel: [  302.997887]  ? do_syscall_64+0x6e/0xb0
Nov 25 08:26:56 odin kernel: [  302.997889]  ? do_syscall_64+0x6e/0xb0
Nov 25 08:26:56 odin kernel: [  302.997891]  entry_SYSCALL_64_after_hwframe+0x44/0xae
Nov 25 08:26:56 odin kernel: [  302.997893] RIP: 0033:0x7f7b3abadcc7
Nov 25 08:26:56 odin kernel: [  302.997895] Code: 00 00 00 48 8b 05 c9 91 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 99 91 0c 00 f7 d8 64 89 01 48
Nov 25 08:26:56 odin kernel: [  302.997896] RSP: 002b:00007f62f57f5288 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 25 08:26:56 odin kernel: [  302.997898] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f7b3abadcc7
Nov 25 08:26:56 odin kernel: [  302.997900] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000035
Nov 25 08:26:56 odin kernel: [  302.997901] RBP: 0000561f01553f90 R08: 0000561f00236878 R09: 00000000000000ff
Nov 25 08:26:56 odin kernel: [  302.997902] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
Nov 25 08:26:56 odin kernel: [  302.997903] R13: 0000561f00690c00 R14: 0000000000000000 R15: 0000000000000000

Looks like core/thread 39 died. At least in the call stack it looks like it is/was handing MMU & IRQ functionality. Which definitely seems like the behavior the guest is/was experiencing.

Now the $42 question...... anyone know something that might be able to be done about it?

 

[leesteken]

It could indeed very well be related to interrupts. Have you tried enabling MSI in the guest? Try adding snd_hda_intel.enable_msi=1 to the kernel parameters (or add options snd-hda-intel enable_msi=1 to a file in /etc/modprobe.d/ and run the Fedora equivalent of update-initramfs).
Have you read about people having passthrough success with this particular audio device? Maybe you are the first to try and run into a bug or incompatibility.

EDIT: Is this audio device supported by snd_hda_intel? Or what driver is it using (lspci -nnk)? Maybe binding it to vfio-pci early helps?

 

[kromberg]

OK, adding in the snd-hda-intel option doesnt seem to make any difference. Here is the output from lscpi -vv on the guest side:

00:10.0 Audio device: Creative Labs EMU20k2 [Sound Blaster X-Fi Titanium Series] (rev 04)
Subsystem: Creative Labs Device 0043
Physical Slot: 16
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at fd790000 (64-bit, non-prefetchable) [size=64K]
Region 2: Memory at fd000000 (64-bit, non-prefetchable) [size=2M]
Region 4: Memory at f8000000 (64-bit, non-prefetchable) [size=64M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [58] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W

You can see the MSI Capabilities is not enabled. Its the same way with the snd-hda-intel option not set. Currently on the proxmox host I am not loading any sound modules.

On both the proxmox and guest/fedora side, they are using the 'snd_ctxfi' driver. If you remove the snd_ctxfi module, then the card defaults to the snd-hda-intel module. Looking real quickly it does not look like there are any options for the module, just include it or not.

From what I have been seeing, reading, researching, 99.999999435% of people are using a USB based audio card and connecting it to a USB passthrough controller.

The card works fine on a native Fedora install. I will try a debian live cd to see if it works there. If it does, then it must be an issue with the passthrough functionality.

 

[leesteken]

Indeed snd_hda_intel appears not to be involved, sorry. Can you please output lspci -nnks '00:10.0' on the host? This is to double check driver and to get the numeric ID for early binding to vfio-pci, which can help prevent the host from touching the device before the VM starts.

 

[kromberg]

On the proxmox host it is 81:00.0 and here is the output of that:

root@odin:~# lspci -nnks '81:00.0' 81:00.0 Audio device [0403]: Creative Labs EMU20k2 [Sound Blaster X-Fi Titanium Series] [1102:000b] (rev 04) Subsystem: Creative Labs EMU20k2 [Sound Blaster X-Fi Titanium Series] [1102:0043] Kernel driver in use: vfio-pci Kernel modules: snd_ctxfi

Based on the blacklist modules, the snd_ctxfi module is not being loaded( or any sound module ):

root@odin:~# lsmod | grep snd_ctxfi root@odin:~# lsmod | grep snd root@odin:~#