Adding a secondary GPU to host breaks passthrough

jcsawyer

New Member
Jan 17, 2022
2
0
1
23
Howdy,

Have been trying to find a solution to this for over a month (on and off) - Hoping someone can point me in the right direction.

I have my proxmox host setup with one VM that is a dedicated game machine for my partner.

The system is:

- Gigabyte B550 AORUS Elite V2
- RYZEN 5950x
- 5900XT

It has been working flawlessly for some time. However, I now want to add a 1050 to the system just to share amongst other VMs for acceleration on the desktop and things like PLEX - To whatever extent that is possible.

The problem is, when I add the gpu to the host machine it gets recognized but its existence breaks the game VM. It will start in proxmox without error but isn't connecting to the network so presumably it fails to boot successfully. If I change vm settings for the display so it shows in VNC, it is just a black screen.

Using `lspci -v` I can see that the 5700 xt id gets changed from 07:00 to 08:00, this is because the new card gets added into 04:00.

I update the game conf to reflect this and it makes no difference.


Game VM Config
Code:
args: -cpu 'host,-hypervisor,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_tim>balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 6
cpu: host,hidden=1
efidisk0: local-lvm:vm-600-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:07:00,pcie=1,romfile=5700xt.bin,x-vga=1
machine: pc-q35-7.2
memory: 12288
meta: creation-qemu=7.2.0,ctime=1680302718
name: game-vm
net0: virtio=C6:68:7A:32:8D:FE,bridge=vmbr0
numa: 0
ostype: win11
scsi0: local-lvm:vm-600-disk-1,cache=writeback,iothread=1,replicate=0,size=512G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=ddb7af44-574b-4de3-b0a1-f00b8a012e6b
sockets: 1
tpmstate0: local-lvm:vm-600-disk-2,size=4M,version=v2.0
vmgenid: dcabdc61-59c5-4970-8b02-fdf239b09879

Grub (has this added)
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt initcall_blacklist=sysfb_init"
GRUB_CMDLINE_LINUX=""

Blacklist
Code:
blacklist radeon
blacklist amdgpu

Modules (/etc/modules)
Code:
vendor-reset
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Other checks
- 5700xt is added to vfio.conf
- Am using https://github.com/gnif/vendor-reset

Any help would be greatly appreciated :)
 
Adding or removing PCI(e) devices can change the PCI ID (check with lspci) of other PCI(e) devices. This can cause problems with other VMs that use passthrough because they select devices by the PCI ID. The network controller name also depends on the PCI ID, which can break networking (check with ip a and adjust accordingly in /etc/network/interfaces). Also always check the IOMMU groups to make sure you don't disconnect (all other) devices from the Proxmox host when you passthrough one device of a group.
EDIT: There are several threads about losing network when doing passthrough and about IOMMU groups. Hope my reply gets you started.
 
Last edited:
Adding or removing PCI(e) devices can change the PCI ID (check with lspci) of other PCI(e) devices. This can cause problems with other VMs that use passthrough because they select devices by the PCI ID. The network controller name also depends on the PCI ID, which can break networking (check with ip a and adjust accordingly in /etc/network/interfaces). Also always check the IOMMU groups to make sure you don't disconnect (all other) devices from the Proxmox host when you passthrough one device of a group.
EDIT: There are several threads about losing network when doing passthrough and about IOMMU groups. Hope my reply gets you started.

Thanks for the reply.

All of my vms make use if the same vmbr bridge. All the others one are working other than this 1 vm with a gpu passthrough, no other VM is using passthrough. I have a feeling it is not an internet issue on the vm, but not completely sure.

The old card is in its own IOMMU group, however, I have now seen the new card (which is not been configured for anything yet) is in the same group as a bunch of stuff. Would that in of itself be an issue?


Note: the only passthrough that exists in my entire setup is the 5700xt (old card) passthrough into that one VM, for which I have updated the PCI ID for

EDIT: Uploaded lspci and find for reference (in case anyone wants to look)

EDIT2: Managed to RDP into the VM but only after I disabled "rom-bar", the GPU now throws error 43 doing this
 

Attachments

  • find.txt
    1.9 KB · Views: 1
  • lspci.txt
    3.8 KB · Views: 1
Last edited:
Have you tried using a different PCIe Slot for the GPUs. I have a hypervisor run Desktop build using the 5950x but I am on the x570 chipset and I have both a 1050 and a 4070ti installed into the system and both get passed through to different VMs. I am wondering if the PCIe slot you are using for the 1050 is coming from the chipset and so it is grouped with other chipset related items.

I know when I was building my machine I had to be very cognisant of the PCIe lanes and also the breakout/down of those lanes when it came to PCIe slots on the motherboard. So I pulled the motherboard manual for your board and noticed these:

  • Your first slot at the top of the board is physically a 16x slot and runs electrically at x16.
  • The second slot is physically a 16x slot and runs electrically at x2 and the third slot is physically x16 but electrically operates at 1x.
  • Both the second and third x16 slots are run by the chipset and so would be within that grouping so you would not be able to pass the GPU through as it would also pull the rest of the items with in that group. However you could try adding
    Code:
    pcie_acs_override=downstream
    to your boot options to see if that is able to split the chipset items apart.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!