GPU passthrough on a server with 2 GPU

[PPPCInc]

Hello,
I have a problem with the GPU passthrough.
I have a server running with a processor without integrated graphics chipset. I have an old GT210 graphics card to run the server.
I have recently added a GTX660 on a PCIE slot.
I have on this proxmox server a virtual machine of Ubuntu Server, I want to add this GTX660 in PCI passthrough. But as soon as I add it all my proxmox server crashes.

I think this is because there are two graphics cards and the server without chipset integrated in the processor needs a card to work. But I can't go any further.
Do you have a solution?
Thanks in advance
Sorry for my English I am French and I don't speak English perfectly.

 

[leesteken]

Please show us your IOMMU groups using this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
It will show whether VFIO is set-up correctly and if the GTX660 is in a isolated group by itself. Also, show the /etc/pve/qermu-server/VM.conf file of the VM.

 

[PPPCInc]

Please show us your IOMMU groups using this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
It will show whether VFIO is set-up correctly and if the GTX660 is in a isolated group by itself. Also, show the /etc/pve/qermu-server/VM.conf file of the VM.

For the first command :
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 10 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 11 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 12 00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 13 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 13 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 14 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU group 14 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU group 14 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU group 14 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU group 14 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU group 14 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU group 14 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU group 14 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 15 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43bc] (rev 02)
IOMMU group 15 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b8] (rev 02)
IOMMU group 15 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b3] (rev 02)
IOMMU group 15 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 15 02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 15 02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 15 02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 15 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 [GeForce GTX 660] [10de:11c0] (rev a1)
IOMMU group 15 03:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio Controller [10de:0e0b] (rev a1)
IOMMU group 15 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 15 06:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)
IOMMU group 16 07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2)
IOMMU group 16 07:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)
IOMMU group 17 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 18 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 19 09:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 1 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 20 09:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 21 09:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU group 22 0a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 23 0b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 5 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 6 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]

For the second :

GNU nano 3.2 100.conf

#NAS
agent: 1
boot: order=scsi0;net0
cores: 5
hostpci0: 03:00.0,pcie=1
machine: q35
memory: 30720
name: Nas-Ubuntu
net0: virtio=32:E7:75:BF:E1:02,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=90G
scsi1: /dev/disk/by-label/Series-2,backup=0,size=5723165M
scsi2: /dev/disk/by-label/Series,backup=0,size=1907728M
scsi3: /dev/disk/by-label/Films,backup=0,size=1953514584K
scsi4: /dev/disk/by-label/Drive,backup=0,size=1907728M
scsi5: /dev/disk/by-label/Films-Divers,backup=0,size=1953514584K
scsihw: virtio-scsi-pci
smbios1: uuid=1e56ea47-2423-44f9-bd92-2248b0b53c40
sockets: 2
vmgenid: 49e991d3-fab2-4d3e-a915-51872b40db8c

 

[Dunuin]

I have a server running with a processor without integrated graphics chipset. I have an old GT210 graphics card to run the server.
I have recently added a GTX660 on a PCIE slot.

That could be problematic. If you need to blacklist the Nvidia GPU driver to be able to passthrough the GTX660 your GT210 won't work either.

Did you enable IOMMU Interrupt Remapping? Your GeForce GTX 660 is sharing the same group with alot of motherboard stuff.

 

[leesteken]

Does the host boot from the GT210 (and not show output on the GTX660)? Have you tried hostpci0: 03:00,pcie=1 (without the .0)? The IOMMU group of the GTX660 also contains a USB controller, two SATA controllers, and a network device. Are you sure the Proxmox host does not use and does not need those devices? Did you add a line starting with options vfio-pci to a file in /etc/modprobe.d/ ? You might want to add 10de:11c0,10de:0e0b to the ids option of vfio-pci (instead of blacklisting drivers).
EDIT: Maybe try swapping the two cards, because the GT210 is in a separate IOMMU group, but make sure the system boots from the GT210.

 

[PPPCInc]

That could be problematic. If you need to blacklist the Nvidia GPU driver to be able to passthrough the GTX660 your GT210 won't work either.

Did you enable IOMMU Interrupt Remapping? Your GeForce GTX 660 is sharing the same group with alot of motherboard stuff.

Yes the IOMMU Interrupt Remapping is enabled..
How can I isolate the graphics card only?

 

[PPPCInc]

Does the host boot from the GT210 (and not show output on the GTX660)? Have you tried hostpci0: 03:00,pcie=1 (without the .0)? The IOMMU group of the GTX660 also contains a USB controller, two SATA controllers, and a network device. Are you sure the Proxmox host does not use and does not need those devices? Did you add a line starting with options vfio-pci to a file in /etc/modprobe.d/ ? You might want to add 10de:11c0,10de:0e0b to the ids option of vfio-pci (instead of blacklisting drivers).
EDIT: Maybe try swapping the two cards, because the GT210 is in a separate IOMMU group, but make sure the system boots from the GT210.

" Did you add a line starting with options vfio-pci to a file in /etc/modprobe.d/ ? You might want to add 10de:11c0,10de:0e0b to the ids option of vfio-pci (instead of blacklisting drivers). "
I did not add a line starting with options vfio-pci to a file in /etc/modprobe.d/ ?

I cant switch the cards because I don't have enough space on the motherboard, if this is the only solution I can buy a riser.

And finally yes the proxmox host needs sata controllers.

 

[leesteken]

" Did you add a line starting with options vfio-pci to a file in /etc/modprobe.d/ ? You might want to add 10de:11c0,10de:0e0b to the ids option of vfio-pci (instead of blacklisting drivers). "
I did not add a line starting with options vfio-pci to a file in /etc/modprobe.d/ ?

I cant switch the cards because I don't have enough space on the motherboard, if this is the only solution I can buy a riser.

And finally yes the proxmox host needs sata controllers.

Then you will have to move the GTX660 to another PCIe slot, until it shows that it is in a separate IOMMU group and the system does not use it when booting. The BIOS of the motherboard determines the IOMMU groups You can use the pcie_acs_override kernel parameter, but there is no guarantee that is will work (and always introduces a possible security risk). Can you please first try moving the video card to another slot. If that works, you can try the override or a riser later.
Moving the card to another slot might change the PCI-number, so change that also in the VM configuration file (remove the last .0) (or enable All Functions) and also add ,x-vga=1 (or enable Primary GPU) for NVidia cards.
If there are no files in /etc/modprobe.d/ that have a vfio-pci line, please add a file that contains the line options vfio-pci ids=10de:11c0,10de:0e0b and run update-initramfs -u before rebooting. This prevents the host from touching the video card, which makes it usable for a VM (if it is also in a separate IOMMU group).