Confirming Passthrough Compatibility for a Custom Server Build with 6 Dedicated GPUs for 6 VMs


New Member
Oct 24, 2023
Hello, and thank you for reading this.

I've taken on a challenge to build a server using consumer-grade components to run 6 VMs, each with its own dedicated GPU. The number 6 is only because the motherboard has 7 PCIE slots which one will be used for a GPU dedicated to the host.

However, my research has not provided absolute certainty regarding the IOMMU group isolation of my motherboard, nor whether the following parts are compatible with such a setup:

CPU: Ryzen Threadripper Pro 5995WX
GPU: Radeon RX 6400 (1 slot form factor)

I have successfully tested one RX 6400 for passthrough capabilities on a B550 motherboard equipped with two PCIe slots—one for the host's dedicated GPU and the other for the RX 6400 passed through to the VM—and it functioned properly.

I want to ensure this configuration will work as intended before investing in the components.

Any advice is much appreciated.
Im going down this path shortly, same board and processor but with Nvidia
2 x RTX A5000 for LXC native docker GPU work (not passed through)
1x 3060ti passed through to windows gaming VM
1x 1660ti passed through to plex
1 LSI Hba passed through to trunas connected to a Jbod

using the m.2 pcie x4 carrier that came with the board for zfs to host the vm and container storage.

Going to be an interesting journey....
  • Like
Reactions: Ramin
Well, passthrough works for HBA and the Asus quad PCIE x16 card, but ive got a weird rebooting issue where the host reboots after about 20 min, Trying to dig deeper, turned off all power managemnt
The asus PCIE storage card was causing the issue. Removed it and no more reboots, im going to wipe the server in case its the pass through edits i did is causing the issue.

2023-11-20T17:56:02.111071-05:00 compressor kernel: [ 396.945570] {7}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 512 2023-11-20T17:56:02.111096-05:00 compressor kernel: [ 396.945580] {7}[Hardware Error]: It has been corrected by h/w and requires no further action 2023-11-20T17:56:02.111098-05:00 compressor kernel: [ 396.945582] {7}[Hardware Error]: event severity: corrected 2023-11-20T17:56:02.111099-05:00 compressor kernel: [ 396.945585] {7}[Hardware Error]: Error 0, type: corrected 2023-11-20T17:56:02.111100-05:00 compressor kernel: [ 396.945589] {7}[Hardware Error]: section_type: PCIe error 2023-11-20T17:56:02.111102-05:00 compressor kernel: [ 396.945591] {7}[Hardware Error]: port_type: 4, root port 2023-11-20T17:56:02.111103-05:00 compressor kernel: [ 396.945593] {7}[Hardware Error]: version: 0.2 2023-11-20T17:56:02.111104-05:00 compressor kernel: [ 396.945595] {7}[Hardware Error]: command: 0x0407, status: 0x0010 2023-11-20T17:56:02.111105-05:00 compressor kernel: [ 396.945599] {7}[Hardware Error]: device_id: 0000:00:01.3 2023-11-20T17:56:02.111106-05:00 compressor kernel: [ 396.945602] {7}[Hardware Error]: slot: 0 2023-11-20T17:56:02.111107-05:00 compressor kernel: [ 396.945604] {7}[Hardware Error]: secondary_bus: 0x03 2023-11-20T17:56:02.111108-05:00 compressor kernel: [ 396.945606] {7}[Hardware Error]: vendor_id: 0x1022, device_id: 0x1483 2023-11-20T17:56:02.111110-05:00 compressor kernel: [ 396.945609] {7}[Hardware Error]: class_code: 060400 2023-11-20T17:56:02.111111-05:00 compressor kernel: [ 396.945611] {7}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0012
OK, reimaged and started over. Something about my setup didnt like the proxmox 8 tutorial.

I dont know what yet, but here is what worked for me and the WRX80E-SAGE SE to passthrough the RTXA4000

vi /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on" vi /etc/modules vfio vfio_iommu_type1 vfio_pci update-grub reboot # Find devices and group ID lspci -nn | grep -i nvidia 41:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1) 41:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1) #41.... so now look for 41 in #lspci -v lspci -v <SNIP> 41:00.0 VGA compatible controller: NVIDIA Corporation GA104GL [RTX A4000] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation GA104GL [RTX A4000] Flags: fast devsel, IRQ 230, IOMMU group 21 <SNIP> #IOMMU group 21, make sure there is ONLY the GPU in that group find /sys/kernel/iommu_groups/ -type l | grep 21 /sys/kernel/iommu_groups/35/devices/0000:21:00.0 /sys/kernel/iommu_groups/21/devices/0000:41:00.0 /sys/kernel/iommu_groups/21/devices/0000:41:00.1 #ignore the first line since it caught the device number in the grep, groups 21 just has nv GPU and sound device # put it in the vfio.conf echo "options vfio-pci ids=0000:41:00.0,0000:41:00.1 disable vga=1" >> /etc/modprobe.d/vfio.conf update-initramfs -u -k all reboot

Now the fun part
Create a windows VM Use VirtIO where possible (disk, network)
I dont attach the card until after the VM is setup with the generic graphics driver. As soon as you add the PCIE device, novnc ceases to work

To prep, have a mouse and keyboard passed through via a usb.

Once you are good with windows, pass through the mouse and keyboard via sub port
Connect your LCD to the output of the card

Add PCI device --> Raw device --> ID of the card (above for me is 41:00.0
ALL FUNCTIONS to pull in the .1 audio device
Primary GPU

Boot to windows, install nvidia driver. and enjoy. Ran Furmark for 20 min, no issues.

Last edited:
  • Like
Reactions: Ramin


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!