Adding a secondary GPU to host breaks passthrough

[jcsawyer]

Howdy,

Have been trying to find a solution to this for over a month (on and off) - Hoping someone can point me in the right direction.

I have my proxmox host setup with one VM that is a dedicated game machine for my partner.

The system is:

- Gigabyte B550 AORUS Elite V2
- RYZEN 5950x
- 5900XT

It has been working flawlessly for some time. However, I now want to add a 1050 to the system just to share amongst other VMs for acceleration on the desktop and things like PLEX - To whatever extent that is possible.

The problem is, when I add the gpu to the host machine it gets recognized but its existence breaks the game VM. It will start in proxmox without error but isn't connecting to the network so presumably it fails to boot successfully. If I change vm settings for the display so it shows in VNC, it is just a black screen.

Using `lspci -v` I can see that the 5700 xt id gets changed from 07:00 to 08:00, this is because the new card gets added into 04:00.

I update the game conf to reflect this and it makes no difference.


Game VM Config

args: -cpu 'host,-hypervisor,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_tim>balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 6
cpu: host,hidden=1
efidisk0: local-lvm:vm-600-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:07:00,pcie=1,romfile=5700xt.bin,x-vga=1
machine: pc-q35-7.2
memory: 12288
meta: creation-qemu=7.2.0,ctime=1680302718
name: game-vm
net0: virtio=C6:68:7A:32:8D:FE,bridge=vmbr0
numa: 0
ostype: win11
scsi0: local-lvm:vm-600-disk-1,cache=writeback,iothread=1,replicate=0,size=512G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=ddb7af44-574b-4de3-b0a1-f00b8a012e6b
sockets: 1
tpmstate0: local-lvm:vm-600-disk-2,size=4M,version=v2.0
vmgenid: dcabdc61-59c5-4970-8b02-fdf239b09879

Grub (has this added)

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt initcall_blacklist=sysfb_init"
GRUB_CMDLINE_LINUX=""

Blacklist

blacklist radeon
blacklist amdgpu

Modules (/etc/modules)

vendor-reset
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Other checks
- 5700xt is added to vfio.conf
- Am using https://github.com/gnif/vendor-reset

Any help would be greatly appreciated

 

[leesteken]

Adding or removing PCI(e) devices can change the PCI ID (check with lspci) of other PCI(e) devices. This can cause problems with other VMs that use passthrough because they select devices by the PCI ID. The network controller name also depends on the PCI ID, which can break networking (check with ip a and adjust accordingly in /etc/network/interfaces). Also always check the IOMMU groups to make sure you don't disconnect (all other) devices from the Proxmox host when you passthrough one device of a group.
EDIT: There are several threads about losing network when doing passthrough and about IOMMU groups. Hope my reply gets you started.

 

[jcsawyer]

Adding or removing PCI(e) devices can change the PCI ID (check with lspci) of other PCI(e) devices. This can cause problems with other VMs that use passthrough because they select devices by the PCI ID. The network controller name also depends on the PCI ID, which can break networking (check with ip a and adjust accordingly in /etc/network/interfaces). Also always check the IOMMU groups to make sure you don't disconnect (all other) devices from the Proxmox host when you passthrough one device of a group.
EDIT: There are several threads about losing network when doing passthrough and about IOMMU groups. Hope my reply gets you started.

Thanks for the reply.

All of my vms make use if the same vmbr bridge. All the others one are working other than this 1 vm with a gpu passthrough, no other VM is using passthrough. I have a feeling it is not an internet issue on the vm, but not completely sure.

The old card is in its own IOMMU group, however, I have now seen the new card (which is not been configured for anything yet) is in the same group as a bunch of stuff. Would that in of itself be an issue?


Note: the only passthrough that exists in my entire setup is the 5700xt (old card) passthrough into that one VM, for which I have updated the PCI ID for

EDIT: Uploaded lspci and find for reference (in case anyone wants to look)

EDIT2: Managed to RDP into the VM but only after I disabled "rom-bar", the GPU now throws error 43 doing this

 

[Astraea]

Have you tried using a different PCIe Slot for the GPUs. I have a hypervisor run Desktop build using the 5950x but I am on the x570 chipset and I have both a 1050 and a 4070ti installed into the system and both get passed through to different VMs. I am wondering if the PCIe slot you are using for the 1050 is coming from the chipset and so it is grouped with other chipset related items.

I know when I was building my machine I had to be very cognisant of the PCIe lanes and also the breakout/down of those lanes when it came to PCIe slots on the motherboard. So I pulled the motherboard manual for your board and noticed these:

• Your first slot at the top of the board is physically a 16x slot and runs electrically at x16.
• The second slot is physically a 16x slot and runs electrically at x2 and the third slot is physically x16 but electrically operates at 1x.
• Both the second and third x16 slots are run by the chipset and so would be within that grouping so you would not be able to pass the GPU through as it would also pull the rest of the items with in that group. However you could try adding

  pcie_acs_override=downstream

  to your boot options to see if that is able to split the chipset items apart.

 

[DocMAX]

i have the same board and want to passthrough the intel arc card. my iommus look terrible.

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]IOMMU Group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]IOMMU Group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]IOMMU Group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)IOMMU Group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)IOMMU Group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]IOMMU Group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]IOMMU Group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]IOMMU Group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]IOMMU Group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]IOMMU Group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]IOMMU Group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]IOMMU Group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]IOMMU Group 14 01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD [15b7:5006]IOMMU Group 15 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset USB 3.1 XHCI Controller [1022:43ee]IOMMU Group 15 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller [1022:43eb]IOMMU Group 15 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset Switch Upstream Port [1022:43e9]IOMMU Group 15 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]IOMMU Group 15 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]IOMMU Group 15 04:00.0 PCI bridge [0604]: Intel Corporation Device [8086:4fa1] (rev 01)IOMMU Group 15 05:01.0 PCI bridge [0604]: Intel Corporation Device [8086:4fa4]IOMMU Group 15 05:04.0 PCI bridge [0604]: Intel Corporation Device [8086:4fa4]IOMMU Group 15 06:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A380] [8086:56a5] (rev 05)IOMMU Group 15 07:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]IOMMU Group 15 08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)IOMMU Group 16 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)IOMMU Group 16 09:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)IOMMU Group 17 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]IOMMU Group 18 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]IOMMU Group 19 0b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]IOMMU Group 20 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]IOMMU Group 21 0b:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]IOMMU Group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]IOMMU Group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]

 

[DocMAX]

i have the same board and want to passthrough the intel arc card. my iommus look terrible.
nvidia passthrough works, but not intel.

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU Group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
IOMMU Group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
IOMMU Group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
IOMMU Group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
IOMMU Group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
IOMMU Group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
IOMMU Group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
IOMMU Group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU Group 14 01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD [15b7:5006]
IOMMU Group 15 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset USB 3.1 XHCI Controller [1022:43ee]
IOMMU Group 15 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller [1022:43eb]
IOMMU Group 15 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset Switch Upstream Port [1022:43e9]
IOMMU Group 15 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 15 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 15 04:00.0 PCI bridge [0604]: Intel Corporation Device [8086:4fa1] (rev 01)
IOMMU Group 15 05:01.0 PCI bridge [0604]: Intel Corporation Device [8086:4fa4]
IOMMU Group 15 05:04.0 PCI bridge [0604]: Intel Corporation Device [8086:4fa4]
IOMMU Group 15 06:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc A380] [8086:56a5] (rev 05)
IOMMU Group 15 07:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller [8086:4f92]
IOMMU Group 15 08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU Group 16 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
IOMMU Group 16 09:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
IOMMU Group 17 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 18 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 19 0b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 20 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 21 0b:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]