IOMMU Groups

[bentoooooo]

Hi there guys, new to proxmox...
So yesterday tryed to passthrough (in the interface) a NIC to a windows 10 VM. Result, full reinstall of proxmox...
So, my motherboard IOMMU i found ou later, put every PCI on the same group, incluiding sata and NVME.

Kernel Version Linux 5.4.73-1-pve #1 SMP PVE 5.4.73-1 (Mon, 16 Nov 2020 10:52:16 +0100)

PVE Manager Version pve-manager/6.3-2/22f57405

Motherboard: msi x470 gaming pro max
CPU: X2700
RAM: 4x 8GB DDR4 2666Mhz
HDD: 2x 4TB (Local Backups) + 2x 2TB (1 VM)
NVME: 1TB (2 VM)
SSD: 2x 120GB (Proxmox)
1x GT710 (and i want to place a RX560 on the 2nd slot with PCI passthrough

My IOMMU grouping:


IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 0 01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E12 NVMe Controller [1987:5012] (rev 01)
IOMMU Group 0 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
IOMMU Group 0 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
IOMMU Group 0 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
IOMMU Group 0 20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 0 20:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 0 20:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 0 20:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 0 20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 0 20:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 0 22:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 0 24:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Device [10ec:8161] (rev 15)
IOMMU Group 0 26:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 2 27:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)
IOMMU Group 2 27:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU Group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 4 28:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU Group 4 28:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 4 28:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
IOMMU Group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 5 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU Group 5 29:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 5 29:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU Group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU Group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]


The motherboard has 3 PCI-E, but also share same IOMMU... So, do you guys think i can add that PCI card to the system (even being on the same group as all rest), and even more important... Can i add the RX 560 to a VM Windows even it sharing group with the GT 710? I've AMD-V and IOMMU activated in the BIOS.

Thank you all in advance.

 

[leesteken]

Most X470 motherboards separate both x16 PCIe slots (with each actually being x8). Your NVidia card (VGA+audio) is in a IOMMU group without anything else (ignore bridges). I would expect the Radeon card to be in a similar group on its own. Just try it by inserting the card, and have a look at the IOMMU groups.
Please note that some devices might shift in PCI id because of the additional device: your network interface might have a different name etc. Maybe you can also passthrough 28:00.3 from group 4 to a VM to connect your keyboard and mouse.
Sometimes the IOMMU group improve with newer BIOS versions (be careful, you cannot always flash an earlier version), and sometime a newer BIOS makes things worse.

 

[bentoooooo]

Most X470 motherboards separate both x16 PCIe slots (with each actually being x8). Your NVidia card (VGA+audio) is in a IOMMU group without anything else (ignore bridges). I would expect the Radeon card to be in a similar group on its own. Just try it by inserting the card, and have a look at the IOMMU groups.
Please note that some devices might shift in PCI id because of the additional device: your network interface might have a different name etc. Maybe you can also passthrough 28:00.3 from group 4 to a VM to connect your keyboard and mouse.
Sometimes the IOMMU group improve with newer BIOS versions (be careful, you cannot always flash an earlier version), and sometime a newer BIOS makes things worse.

I will try to put the NIC i have on the 2nd PCI-E, the strange fact is that the 3d PCI-E its on the same group (tested with the NIC), i've updated to the latest BIOS without knowing that... So maybe i'm screwed on that mather. (Don't have the RX560, one week to arrive).
What about the NIC? I've tryed every single PCI and the PCI-E number 3, every PCI goes to group 0 and the PCI-E number 3 goes to the group 2 of the GT710.
Thank you for your time. .

 

[leesteken]

As is common to the AM4 platform: one x16 or two x8 (each look like x16) slots are connected to the CPU and usually each in a separate group available for passthrough. 4 PCIe lanes are for a SSD/NVME connected to the CPU (and in a separate group available for passthrough). The other 4 PCIe lanes are connected to the X470 chipset. The CPU also provides a single USB controller and an audio device, often available for passthrough. Everything connected to the chipset (except for a X570 chipset) are in a single group and not available for passthrough. This usually includes the network controller(s), the SATA controller(s), additional SSD/NVME connectors, additional USB controllers, and all (devices in the) x1 PCIe slots. I hope this give you an overview of the possibilities of your motherboard and helps set expectations regarding passthrough.

 

[bentoooooo]

As is common to the AM4 platform: one x16 or two x8 (each look like x16) slots are connected to the CPU and usually each in a separate group available for passthrough. 4 PCIe lanes are for a SSD/NVME connected to the CPU (and in a separate group available for passthrough). The other 4 PCIe lanes are connected to the X470 chipset. The CPU also provides a single USB controller and an audio device, often available for passthrough. Everything connected to the chipset (except for a X570 chipset) are in a single group and not available for passthrough. This usually includes the network controller(s), the SATA controller(s), additional SSD/NVME connectors, additional USB controllers, and all (devices in the) x1 PCIe slots. I hope this give you an overview of the possibilities of your motherboard and helps set expectations regarding passthrough.

Thank you so much for the inside...
All i need is 1x NIC 1x RX560 1x USB.

I will not need a screen for proxmox, if i place the RX560 on the first PCI-E slot + NIC on the 2nd PCI-E slot + USB from CPU group it's possible to do it?

I don't know how proxmox will handle not having a gpu to himself, and if this setup can be stable...

Thank you for the help. .

 

[leesteken]

Thank you so much for the inside...
All i need is 1x NIC 1x RX560 1x USB.

I will not need a screen for proxmox, if i place the RX560 on the first PCI-E slot + NIC on the 2nd PCI-E slot + USB from CPU group it's possible to do it?

I don't know how proxmox will handle not having a gpu to himself, and if this setup can be stable...

Thank you for the help. .

I expect that this "single GPU passthrough" will work with a RX560. If not, then have a look at vendor-reset. Don't enable autostart on the VM until you are sure it works, because you cannot see error messages or login to the Proxmox console.
Why do you need NIC passthrough? Can't you use a virtual network card (virtio) and bridge? Or a USB network dongle attached to the passed through USB controller?

 

[bentoooooo]

I expect that this "single GPU passthrough" will work with a RX560. If not, then have a look at vendor-reset. Don't enable autostart on the VM until you are sure it works, because you cannot see error messages or login to the Proxmox console.
Why do you need NIC passthrough? Can't you use a virtual network card (virtio) and bridge? Or a USB network dongle attached to the passed through USB controller?

i'm testing with this NIC but it needs to be an Gigabit CT card for an X-RAY maxine, all this is software requirements from the app i need to run in the VM. :/.
The app needs an amd graphics card RX560 or better (for 3d rendering of medical 3d scans) an intel E1000(will emulate) and Gigabit CT for direct comunication with the X-RAY machine, and i need the USB for the UPS to shutdown it when needed.

Thank you for all the help.