[SOLVED] PCI specific device passthrough

[ukro]

Greetings,
So i have this devices in my home server and i want to pass one specific device/pci port. It is in the same IOMMU group as network adapter.But i don't want to pass the adapter. Only one PCIe slot. Can i ? How? Pls help
Device i need to pass is
/sys/kernel/iommu_groups/6/devices/0000:00:1c.4

```
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:9b43] (rev 05)
IOMMU Group 1 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:9bc5] (rev 05)
IOMMU Group 2 00:14.0 USB controller [0c03]: Intel Corporation Device [8086:a3af]
IOMMU Group 3 00:16.0 Communication controller [0780]: Intel Corporation Device [8086:a3ba]
IOMMU Group 4 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a382]
IOMMU Group 5 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:a3e9] (rev f0)
IOMMU Group 5 00:1b.4 PCI bridge [0604]: Intel Corporation Device [8086:a3eb] (rev f0)
IOMMU Group 5 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
IOMMU Group 6 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a392] (rev f0)
IOMMU Group 6 00:1c.3 PCI bridge [0604]: Intel Corporation Device [8086:a393] (rev f0)
IOMMU Group 6 00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:a394] (rev f0)
IOMMU Group 6 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 6 05:00.0 System peripheral [0880]: Device [1ac1:089a]
IOMMU Group 7 00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:a398] (rev f0)
IOMMU Group 8 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a3c8]
IOMMU Group 8 00:1f.2 Memory controller [0580]: Intel Corporation Device [8086:a3a1]
IOMMU Group 8 00:1f.3 Audio device [0403]: Intel Corporation Device [8086:a3f0]
IOMMU Group 8 00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:a3a3]

```
In GUI i see that i can pass only the full 6 Group
I put the x1 card to x16 slot, it showed up but my ethernet stopped working oO

 

[leesteken]

It is a bit strange to pass a PCI bridge (00:1c.4), as those are done automatically with an actual device. Are you sure? The only device besides the network controller in group 6 is a "System peripheral" (05:00.0), but I don't really know what that is.
Currently, I see a USB controller, a "Communications" controller and an NVME drive that appear to be good candidates for PCI passthrough, and also an integrated GPU.
In general, try moving the device to another PCI slot to see if it becomes an isolated IOMMU group (PCI bridges in a group are not a problem).

 

[ukro]

It is a bit strange to pass a PCI bridge (00:1c.4), as those are done automatically with an actual device. Are you sure? The only device besides the network controller in group 6 is a "System peripheral" (05:00.0), but I don't really know what that is.
Currently, I see a USB controller, a "Communications" controller and an NVME drive that appear to be good candidates for PCI passthrough, and also an integrated GPU.
In general, try moving the device to another PCI slot to see if it becomes an isolated IOMMU group (PCI bridges in a group are not a problem).

I need to passthrpugh pci google coral. When i put to pciex1 it is bound with immou group with ethernet. If i put it to pciex16 it is bound to immou group with nvme...

So if there is no way to seperate device from immou group or to pass one specific device from immou group. I need to buy an adapter and move my nvme to pcix1 with 250MB/s speed cap ;( and then the vnme slots will be empty and i pass the whole immou group with the coral card.....

 

[leesteken]

Sometimes a BIOS update or a different motherboard can change the IOMMU groups.
You can also add pcie_acs_overrride=downstream to the kernel parameters in GRUB or systemd-boot to forcefully break your IOMMU groups. This kernel patch is already built-in to the Proxmox kernels. There is no guarantee that the device will work with PCI passthrough and/or not interfere with other devices in the same group! If you don't run unknown code by untrusted users on your VMs, the risk is probably small and it might be worth a try to see if you can get it to work that way.

 

[ukro]

Sometimes a BIOS update or a different motherboard can change the IOMMU groups.
You can also add pcie_acs_overrride=downstream to the kernel parameters in GRUB or systemd-boot to forcefully break your IOMMU groups. This kernel patch is already built-in to the Proxmox kernels. There is no guarantee that the device will work with PCI passthrough and/or not interfere with other devices in the same group! If you don't run unknown code by untrusted users on your VMs, the risk is probably small and it might be worth a try to see if you can get it to work that way.

Exactly what i was searching for ! thank you for poiting that out !!! god bless ! Will report when i will be able to do it :> take care

 

[ukro]

Exactly what i was searching for ! thank you for poiting that out !!! god bless ! Will report when i will be able to do it :> take care

The numbers did changed a bit, but it stayed bound to the ethernet. I am now trying go deep in rabbit hole with the acs :>

 

[ukro]

So nothing worked. I have one more shot to try then i am jsut using the pcix1 adapter..... went through like 30tutorials....

 

[Itay1778]

The numbers did changed a bit, but it stayed bound to the ethernet. I am now trying go deep in rabbit hole with the acs :>

I'm sorry to let you know but from what it looks like you'll never be able to separate them (hope you prove me wrong)
They all seem to be connected to the same PCI bridge, so even though their IDs are different they are basically in the same group forever. This is how this motherboard was designed. I too was in a similar problem like this and just bought a PCIe network card.

 

[ukro]

I'm sorry to let you know but from what it looks like you'll never be able to separate them (hope you prove me wrong)
They all seem to be connected to the same PCI bridge, so even though their IDs are different they are basically in the same group forever. This is how this motherboard was designed. I too was in a similar problem like this and just bought a PCIe network card.

Yeah....One way is to use container as there you can pass it without issue. And in my case i cant use PCI network card as i have One group for all PCIx1 + network adapter and another group PCIx16 and NVME storage. So my only option is to put my NVME to the PCIx1 adapter slot ;( and then put the PCI coral x1 card to PCIx16 slot ;(

 

[leesteken]

The numbers did changed a bit, but it stayed bound to the ethernet. I am now trying go deep in rabbit hole with the acs :>

This is strange to me. The override option should always break the groups. Even though passthrough might still not work, it should show all devices separately. Maybe try pcie_acs_override=downstream,multifunction? You can also add it once by pressing 'e' in the Proxmox boot menu. Can you show the resulting groups (with the coral device) with for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done?

 

[ukro]

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

root@matrix:~# ./listiommu.sh
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:9b43] (rev 05)
IOMMU Group 1 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:9bc5] (rev 05)
IOMMU Group 2 00:14.0 USB controller [0c03]: Intel Corporation Device [8086:a3af]
IOMMU Group 3 00:16.0 Communication controller [0780]: Intel Corporation Device [8086:a3ba]
IOMMU Group 4 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a382]
IOMMU Group 5 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:a3e9] (rev f0)
IOMMU Group 5 00:1b.4 PCI bridge [0604]: Intel Corporation Device [8086:a3eb] (rev f0)
IOMMU Group 5 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
IOMMU Group 6 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a392] (rev f0)
IOMMU Group 6 00:1c.3 PCI bridge [0604]: Intel Corporation Device [8086:a393] (rev f0)
IOMMU Group 6 00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:a394] (rev f0)
IOMMU Group 6 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 6 05:00.0 System peripheral [0880]: Device [1ac1:089a]
IOMMU Group 7 00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:a398] (rev f0)
IOMMU Group 8 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a3c8]
IOMMU Group 8 00:1f.2 Memory controller [0580]: Intel Corporation Device [8086:a3a1]
IOMMU Group 8 00:1f.3 Audio device [0403]: Intel Corporation Device [8086:a3f0]
IOMMU Group 8 00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:a3a3]

With pcie_acs_override=downstream,multifunction

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:9b43] (rev 05)
IOMMU Group 1 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:9bc5] (rev 05)
IOMMU Group 2 00:14.0 USB controller [0c03]: Intel Corporation Device [8086:a3af]
IOMMU Group 3 00:16.0 Communication controller [0780]: Intel Corporation Device [8086:a3ba]
IOMMU Group 4 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a382]
IOMMU Group 5 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:a3e9] (rev f0)
IOMMU Group 6 00:1b.4 PCI bridge [0604]: Intel Corporation Device [8086:a3eb] (rev f0)
IOMMU Group 6 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a392] (rev f0)
IOMMU Group 8 00:1c.3 PCI bridge [0604]: Intel Corporation Device [8086:a393] (rev f0)
IOMMU Group 8 00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:a394] (rev f0)
IOMMU Group 8 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 8 05:00.0 System peripheral [0880]: Device [1ac1:089a]
IOMMU Group 9 00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:a398] (rev f0)
IOMMU Group 10 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a3c8]
IOMMU Group 10 00:1f.2 Memory controller [0580]: Intel Corporation Device [8086:a3a1]
IOMMU Group 10 00:1f.3 Audio device [0403]: Intel Corporation Device [8086:a3f0]
IOMMU Group 10 00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:a3a3]

The device i need to seperate is
05:00.0 System peripheral [0880]: Device [1ac1:089a]

 

[leesteken]

Thank you for showing me that it is indeed possible that the pcie_acs_overrride does not always break up IOMMU groups. I'm still very surprised.
As you are about to buy additional hardware; have you read of people being successful with passing through this (particular) device? Having it in a separate group is required but no guarantee for success.

 

[ukro]

It is a not common product to use in proxmox, but i have found 1 tutorial where everything is working. Will let you know in 2 days it will arrive and i will put my host os nvme drive to PCIx1 group with ethernet and leave the pcix16/nvme group to the coral pci card. Fingers crossed :>
https://www.reddit.com/r/Proxmox/comments/n34f8q/proxmox_vm_ubuntu_2004_frigate_2x_google_coral_tpu/

 

[ukro]

Thank you for showing me that it is indeed possible that the pcie_acs_overrride does not always break up IOMMU groups. I'm still very surprised.
As you are about to buy additional hardware; have you read of people being successful with passing through this (particular) device? Having it in a separate group is required but no guarantee for success.

It is working. I see the device in VM. but the command in grub was neccessary as without it it just bound with some "random" pci device. Now its fine . I used the nvme->PCI with the ethernet group and put the coral pci to GPU/NVME group