PCI passthrough of onboard audio disables network card

[sinykkskizm]

I have a Windows 10 VM I'm trying to passthrough the audio card to but I'm running into a some issues.

I'm trying to pass my onboard audio card to my Windows 10 VM but every time I try and do a PCI Passthrough, my network card stops working on both the host and the guest.

Here's my setup.

My HW Info

System:   
Host: sy Kernel: 5.4.103-1-pve x86_64
Machine:   Type: Desktop Mobo: ASRock model: Z390 Phantom Gaming-ITX/ac 
CPU:       Topology: 8-Core model: Intel Core i7-9700K 
Graphics:  Device-1: Intel vendor: ASRock driver: N/A bus ID: 00:02.0
           Device-2: NVIDIA TU102 [GeForce RTX 2080 Ti] vendor: eVga.com. driver: vfio-pci v: 0.2 bus ID: 01:00.0
Audio:     Device-1: Intel Cannon Lake PCH cAVS vendor: ASRock driver: vfio-pci v: 0.2 bus ID: 00:1f.3
           Device-2: NVIDIA TU102 High Definition Audio vendor: eVga.com. driver: vfio-pci v: 0.2 bus ID: 01:00.1
           Sound Server: ALSA v: k5.4.103-1-pve
Network:   Device-1: Intel Wireless-AC 9560 [Jefferson Peak] driver: vfio-pci v: 0.2 port: 4000 bus ID: 00:14.3
           Device-2: Intel Ethernet I219-V vendor: ASRock driver: e1000e v: 3.2.6-k port: efa0 bus ID: 00:1f.6
Drives:    Local Storage: total: 1.84 TiB used: 302.26 GiB (16.0%)
           ID-1: /dev/nvme0n1 vendor: Seagate model: XPG GAMMIX S11 Pro size: 953.87 GiB
           ID-2: /dev/nvme1n1 vendor: A-Data model: SX8200PNP size: 953.87 GiB
           ID-3: /dev/sda vendor: Western Digital model: WDS100T2B0A-00SM50 size: 931.51 GiB temp: 33 C
Info:      Processes: 210 Uptime: 9m Memory: 30.85 GiB used: 29.37 GiB (95.2%) Init: systemd runlevel: 5 Compilers: gcc: 8.3.0
           clang: 7.0.1-8+deb10u2 Shell: bash v: 5.0.3 inxi: 3.0.32

Here are the IOMMU groups

root@sy:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e30] (rev 0d)
IOMMU group 10 00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port [8086:a330] (rev f0)
IOMMU group 11 00:1f.0 ISA bridge [0601]: Intel Corporation Z390 Chipset LPC/eSPI Controller [8086:a305] (rev 10)
IOMMU group 11 00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
IOMMU group 11 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU group 11 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU group 11 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
IOMMU group 12 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)
IOMMU group 13 01:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)
IOMMU group 14 01:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Controller [10de:1ad6] (rev a1)
IOMMU group 15 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 UCSI Controller [10de:1ad7] (rev a1)
IOMMU group 16 02:00.0 Non-Volatile memory controller [0108]: Device [1cc1:8201] (rev 03)
IOMMU group 17 3c:00.0 Non-Volatile memory controller [0108]: Device [1cc1:8201] (rev 03)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 0d)
IOMMU group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:3e98] (rev 02)
IOMMU group 3 00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU group 4 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU group 4 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU group 5 00:14.3 Network controller [0280]: Intel Corporation Wireless-AC 9560 [Jefferson Peak] [8086:a370] (rev 10)
IOMMU group 6 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU group 7 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
IOMMU group 8 00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port [8086:a340] (rev f0)
IOMMU group 9 00:1b.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port [8086:a32c] (rev f0)

I can see that the Ethernet Controller is in the same 00:1f group as the audio device. I'm wondering if there is a way to work around that? I can't pass the whole 00:1f group because that disables the Ethernet card on the host (still works on the VM)

Here is my /etc/default/grub line GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off rd.driver.blacklist=snd_hda_intel module.blacklist=snd_hda_intel vfio-pci.ids=8086:a348"

and here is my vm.conf

agent: 1
bios: ovmf
boot: order=virtio0;net0
cores: 8
cpu: host,hidden=1,flags=+md-clear;+pcid;+spec-ctrl;+ssbd;+pdpe1gb;+hv-tlbflush;+aes
efidisk0: efi:100/vm-100-disk-0.qcow2,size=128K
hostpci0: 01:00,pcie=1,x-vga=on
hostpci1: 00:14
hostpci2: 00:1f.3

machine: q35
memory: 28672
name: win10
net0: virtio=B4:1F:4F:41:A6:BC,bridge=vmbr0
numa: 1
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=bd448c84-3921-4ae2-a3f5-4b45d3f5b774
sockets: 1
usb1: host=1-8,usb3=1
usb2: host=1-9,usb3=1
usb3: host=1-5,usb3=1
usb4: host=2-8,usb3=1
vga: none
virtio0: other:100/vm-100-disk-0.qcow2,backup=0,cache=writeback,iothread=1,size=500G
virtio1: /dev/disk/by-id/nvme-ADATA_SX8200PNP_2J0920080265,backup=0,cache=writeback,iothread=1,size=1000204632K
virtio2: /dev/disk/by-id/ata-WDC_WDS100T2B0A-00SM50_1942B1802643,backup=0,cache=writeback,iothread=1,size=976762584K
vmgenid: 3e764e9d-cd78-4292-b401-d3b866de2377

 

[Stefan_R]

All devices in an IOMMU group have to be passed through together. There is no way around that, that's how the IOMMU works. Since your NIC and audio card are in the same group, they either get passed both, or not at all (if the device is not specified, it will get "passed through" but since QEMU doesn't know about it, it will probably be lost as long as the VM is running).

You're already using the pcie_acs_override switch, and the grouping is still not split - this usually means that the devices are tied together strongly in hardware, i.e. share a bus or similar. I don't think there is a way around that.

 

[sinykkskizm]

All devices in an IOMMU group have to be passed through together. There is no way around that, that's how the IOMMU works. Since your NIC and audio card are in the same group, they either get passed both, or not at all (if the device is not specified, it will get "passed through" but since QEMU doesn't know about it, it will probably be lost as long as the VM is running).

You're already using the pcie_acs_override switch, and the grouping is still not split - this usually means that the devices are tied together strongly in hardware, i.e. share a bus or similar. I don't think there is a way around that.

Thanks for the explanation. I appreciate it.