Confirming Passthrough Compatibility for a Custom Server Build with 6 Dedicated GPUs for 6 VMs

[Ramin]

Hello, and thank you for reading this.

I've taken on a challenge to build a server using consumer-grade components to run 6 VMs, each with its own dedicated GPU. The number 6 is only because the motherboard has 7 PCIE slots which one will be used for a GPU dedicated to the host.

However, my research has not provided absolute certainty regarding the IOMMU group isolation of my motherboard, nor whether the following parts are compatible with such a setup:

Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI II
CPU: Ryzen Threadripper Pro 5995WX
GPU: Radeon RX 6400 (1 slot form factor)

I have successfully tested one RX 6400 for passthrough capabilities on a B550 motherboard equipped with two PCIe slots—one for the host's dedicated GPU and the other for the RX 6400 passed through to the VM—and it functioned properly.

I want to ensure this configuration will work as intended before investing in the components.

Any advice is much appreciated.

 

[LnxBil]

I want to ensure this configuration will work as intended before investing in the components.

With PCIe, there are no guarantees, yet I hope you find someone that has already setup with this exact components. Good luck.

 

[aghadjip]

Im going down this path shortly, same board and processor but with Nvidia
2 x RTX A5000 for LXC native docker GPU work (not passed through)
1x 3060ti passed through to windows gaming VM
1x 1660ti passed through to plex
1 LSI Hba passed through to trunas connected to a Jbod

using the m.2 pcie x4 carrier that came with the board for zfs to host the vm and container storage.

Going to be an interesting journey....

 

[aghadjip]

Well, passthrough works for HBA and the Asus quad PCIE x16 card, but ive got a weird rebooting issue where the host reboots after about 20 min, Trying to dig deeper, turned off all power managemnt

 

[aghadjip]

The asus PCIE storage card was causing the issue. Removed it and no more reboots, im going to wipe the server in case its the pass through edits i did is causing the issue.


2023-11-20T17:56:02.111071-05:00 compressor kernel: [ 396.945570] {7}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 512 2023-11-20T17:56:02.111096-05:00 compressor kernel: [ 396.945580] {7}[Hardware Error]: It has been corrected by h/w and requires no further action 2023-11-20T17:56:02.111098-05:00 compressor kernel: [ 396.945582] {7}[Hardware Error]: event severity: corrected 2023-11-20T17:56:02.111099-05:00 compressor kernel: [ 396.945585] {7}[Hardware Error]: Error 0, type: corrected 2023-11-20T17:56:02.111100-05:00 compressor kernel: [ 396.945589] {7}[Hardware Error]: section_type: PCIe error 2023-11-20T17:56:02.111102-05:00 compressor kernel: [ 396.945591] {7}[Hardware Error]: port_type: 4, root port 2023-11-20T17:56:02.111103-05:00 compressor kernel: [ 396.945593] {7}[Hardware Error]: version: 0.2 2023-11-20T17:56:02.111104-05:00 compressor kernel: [ 396.945595] {7}[Hardware Error]: command: 0x0407, status: 0x0010 2023-11-20T17:56:02.111105-05:00 compressor kernel: [ 396.945599] {7}[Hardware Error]: device_id: 0000:00:01.3 2023-11-20T17:56:02.111106-05:00 compressor kernel: [ 396.945602] {7}[Hardware Error]: slot: 0 2023-11-20T17:56:02.111107-05:00 compressor kernel: [ 396.945604] {7}[Hardware Error]: secondary_bus: 0x03 2023-11-20T17:56:02.111108-05:00 compressor kernel: [ 396.945606] {7}[Hardware Error]: vendor_id: 0x1022, device_id: 0x1483 2023-11-20T17:56:02.111110-05:00 compressor kernel: [ 396.945609] {7}[Hardware Error]: class_code: 060400 2023-11-20T17:56:02.111111-05:00 compressor kernel: [ 396.945611] {7}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0012

 

[aghadjip]

OK, reimaged and started over. Something about my setup didnt like the proxmox 8 tutorial.
https://forum.proxmox.com/threads/p...x-ve-8-installation-and-configuration.130218/

I dont know what yet, but here is what worked for me and the WRX80E-SAGE SE to passthrough the RTXA4000

vi /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on" vi /etc/modules vfio vfio_iommu_type1 vfio_pci update-grub reboot # Find devices and group ID lspci -nn | grep -i nvidia 41:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1) 41:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1) #41.... so now look for 41 in #lspci -v lspci -v <SNIP> 41:00.0 VGA compatible controller: NVIDIA Corporation GA104GL [RTX A4000] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation GA104GL [RTX A4000] Flags: fast devsel, IRQ 230, IOMMU group 21 <SNIP> #IOMMU group 21, make sure there is ONLY the GPU in that group find /sys/kernel/iommu_groups/ -type l | grep 21 /sys/kernel/iommu_groups/35/devices/0000:21:00.0 /sys/kernel/iommu_groups/21/devices/0000:41:00.0 /sys/kernel/iommu_groups/21/devices/0000:41:00.1 #ignore the first line since it caught the device number in the grep, groups 21 just has nv GPU and sound device # put it in the vfio.conf echo "options vfio-pci ids=0000:41:00.0,0000:41:00.1 disable vga=1" >> /etc/modprobe.d/vfio.conf update-initramfs -u -k all reboot

Now the fun part
Create a windows VM Use VirtIO where possible (disk, network)
I dont attach the card until after the VM is setup with the generic graphics driver. As soon as you add the PCIE device, novnc ceases to work

To prep, have a mouse and keyboard passed through via a usb.

Once you are good with windows, pass through the mouse and keyboard via sub port
Connect your LCD to the output of the card

Add PCI device --> Raw device --> ID of the card (above for me is 41:00.0
ALL FUNCTIONS to pull in the .1 audio device
Primary GPU

Boot to windows, install nvidia driver. and enjoy. Ran Furmark for 20 min, no issues.