PCI Passthrough Crashing Host

[RobJonHol]

I recently passthrough my GPU but the HDMI audio cuts out randomly. I tried to add the audio devices from the web GUI but failed to get any audio from those. I am now trying to passthrough the audio controller from the motherboard.

I got everything setup and I think it looks good but when I start the VM the network latency starts going up and the host gets really slow till it crashes about a minuet later.

The device is in its own IOMMU group (IOMMU Group 23)
IOMMU Group 2: 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] IOMMU Group 20: 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] IOMMU Group 21: 29:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486] IOMMU Group 22: 29:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] IOMMU Group 23: 29:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

I made sure to put the audio controller ID into /etc/modprobe.d/vfio.conf and black listed the driver so the host wouldn't use it.
options vfio-pci ids=10de:1c02,10de:10f1,1022:1487 disable_vga=1 root@pm2:~# cat /etc/modprobe.d/blacklist.conf blacklist radeon blacklist nouveau blacklist nvidia blacklist snd_hda_intel root@pm2:~# lspci -v 29:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller Subsystem: Micro-Star International Co., Ltd. [MSI] Starship/Matisse HD Audio Controller Flags: bus master, fast devsel, latency 0, IRQ 4 Memory at f7800000 (32-bit, non-prefetchable) [size=32K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [64] Express Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [2a0] Access Control Services Capabilities: [370] Transaction Processing Hints Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

VM Config
root@pm2:~# cat /etc/pve/qemu-server/103.conf agent: 1 args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' bios: ovmf boot: dcn bootdisk: virtio0 cores: 6 cpu: host,hidden=1,flags=+pcid efidisk0: holstein-ur1:103/vm-103-disk-1.qcow2,size=128K hostpci0: 27:00,pcie=1 hostpci1: 29:00,pcie=1 ide0: holstein-ur1:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K machine: q35 memory: 16300 name: holstein-w10a net0: e1000=F2:CD:A1:73:6F:2B,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsihw: virtio-scsi-single smbios1: uuid=e23dc67e-716f-4ab8-85bc-6b78c287986c sockets: 1 usb0: host=04d9:a0cd usb1: host=046d:c531 usb2: host=0d8c:0005 virtio0: local-lvm:vm-103-disk-0,cache=writeback,size=70G virtio1: /dev/disk/by-id/ata-WDC_WD10EZEX-60ZF5A0_WD-WCC1S1390455,size=976762584K virtio2: /dev/disk/by-id/ata-MTFDDAV256TBN-1AR15ABHA_UGXVR01D1AM34V,size=250059096K vmgenid: fa3836ab-526b-440f-a1bb-7532e89f2bbe

Host Specs
CPU - Ryzen 3700x
MB - MSI x470 Gaming Pro Carbon
RAM - 32GB DDR4 OLOy
GPU - GTX 1060 3GB

Any ideas? Is passing the onboard audio even a thing? Maybe I'm hitting my head on the wall for no reason.

 

[RobJonHol]

hostpci1: 29:00,pcie=1

This is actually "hostpci1: 29:00.4,pcie=1". Correcting this did not fix the issue.

 

[lhorace]

You haven't mentioned what you are running inside the VM

 

[RobJonHol]

You haven't mentioned what you are running inside the VM

My apologies. The operating system would be of note. I am running Windows 10 Pro.

 

[dcsapak]

hostpci0: 27:00,pcie=1

what device is this? and is it in its own iommu ?

 

[RobJonHol]

what device is this? and is it in its own iommu ?

This is the GPU. Yes, it is in its own iommu group.

IOMMU Group 18: 27:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1) 27:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)

 

[dcsapak]

did you enable the acs override patch? what does

cat /proc/cmdline

say?

 

[RobJonHol]

did you enable the acs override patch? what does

cat /proc/cmdline

say?

I don't think so.

root@pm2:~# cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.3.10-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on

I will try that now.

 

[RobJonHol]

I don't think so.

root@pm2:~# cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.3.10-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on

I will try that now.

Unfortunately, that did not work.

root@holstein-pm2:~# cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.3.10-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on pcie_acs_override=downstream,multifunction

If I let the host crash and reboot itself it doesn't come back with the PCI devices 29:00.3 or 29:00.4. I have to manually reboot the system for those devices to show back up again.

 

[dcsapak]

mhmm... dmesg/journal output from a crash would be interesting..

 

[RobJonHol]

mhmm... dmesg/journal output from a crash would be interesting..

I hope this is what you are looking for.

 

[dcsapak]

mhmm... those do not contain valuable info about the crash...
maybe the syslog? /var/log/syslog* ?

 

[unionaire]

Maybe this can help you:
https://pve.proxmox.com/wiki/Pci_passthrough

HDMI Audio crackling/broken
Some digital audio devices (usually added via GPU functions) may require MSI (Message Signaled Interrupts) to be enabled to function correctly. If you experience any issues, try changing MSI settings in the guest and rebooting the guest.

A Windows-Tool to simplify this is available here: https://github.com/CHEF-KOCH/MSI-utility/releases/latest

Linux guests usually enable MSI by themselves. To force use of MSI for GPU audio devices, use the following command and reboot:

echo "options snd-hda-intel enable_msi=1" >> /etc/modprobe.d/snd-hda-intel.conf

Use 'lspci -vv' and check for the following line on your device to see if MSI is enabled:

Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+

If it says 'Enable+', MSI is working, 'Enable-' means it is supported but disabled, and if the line is missing, MSI is not supported by the PCIe hardware.

This can potentially also improve performance for other passthrough devices, including GPUs, but that depends on the hardware being used.