[Solved] Asus TUF B550 Plus and IOMMU Groups

[notarobot]

Hello everyone,
I bought a new Asus Tuf Gaming B550 Plus and I'm trying to make it work with Proxmox.
For some reasons I need that one controller plugged on one of the PCI slots is pass through to a VM.
Thing is IOMMU grouping on this motherboard is rubbish, so after research I tried to activate ACS override but no luck, IOMMU groups doesn't move.

Here are the IOMMU groups (see that nasty group 15 ?):

IOMMU Group 0
    00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU Group 1
    00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU Group 2
    00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU Group 3
    00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU Group 4
    00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU Group 5
    00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU Group 6
    00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU Group 7
    00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU Group 8
    00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU Group 9
    00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU Group 10
    00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU Group 11
    00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU Group 12
    00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
    00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU Group 13
    00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
    00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
    00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
    00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
    00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
    00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
    00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
    00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
IOMMU Group 14
    01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
IOMMU Group 15
    02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43ee
    02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43eb
    02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43e9
    03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
    03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
    03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
    03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
    03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
    03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
    03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
    04:00.0 VGA compatible controller: NVIDIA Corporation GM206GL [Quadro M2000] (rev a1)
    04:00.1 Audio device: NVIDIA Corporation GM206 High Definition Audio Controller (rev a1)
    06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
    07:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
    08:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
    0a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8125 (rev 04)
IOMMU Group 16
    0b:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
IOMMU Group 17
    0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU Group 18
    0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU Group 19
    0d:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
IOMMU Group 20
    0d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
IOMMU Group 21
    0d:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

Here's the content of my /etc/kernel/cmdline:

root=ZFS=rpool/ROOT/pve-1 boot=zfs
quiet amd_iommu=on
pcie_acs_override=downstream,multifunction
nofb nomodeset

So as always I'm asking for help: am I doing this wrong ? What are my options ?

Thanks in advance for any help.

 

[leesteken]

pcie_acs_override=downstream,multifunction should split everything into separate groups, which appears not to be the case for you. Maybe put everything on a single line instead of four? Make sure to run update-initramfs -u before rebooting. Are you sure you are using systemd-boot and not GRUB (/etc/default/grub)?
Maybe try another BIOS version? pcie_acs_override is not a magic fix, your motherboard+BIOS might just not work with for this particular passthrough.

 

[notarobot]

pcie_acs_override=downstream,multifunction should split everything into separate groups, which appears not to be the case for you. Maybe put everything on a single line instead of four? Make sure to run update-initramfs -u before rebooting. Are you sure you are using systemd-boot and not GRUB (/etc/default/grub)?
Maybe try another BIOS version? pcie_acs_override is not a magic fix, your motherboard+BIOS might just not work with for this particular passthrough.

I'm sure I'm not using grub and to be 100% positive I modified /etc/default/grub as well.
I also upgraded to the last version of the BIOS

 

[leesteken]

I'm sure I'm not using grub and to be 100% positive I modified /etc/default/grub as well.
I also upgraded to the last version of the BIOS

Sometimes an older BIOS works better for AM4 motherboards. Sometimes there just isn't a BIOS available from the manufacturer that works well for PCI passthrough for a specific motherboard.
Your IOMMU groups look like the pcie_acs_override is not applied or not working. I thought that the Proxmox developers always included the kernel patch that is necessary. Which kernel version are you running?
If the kernel supports it, then somehow your pcie_acs_override is not applied because it will always (without checking whether it is secure, safe, or working) split the IOMMU groups. This suggest double checking your changes. What does cat /proc/cmdline give you? Is there a line in dmesg or journalctl -b 0 that mentions the override?

 

[notarobot]

Thanks for the answer !
My kernel is PVE 5.4.101-1
But there seems to be a problem as you can see in my /proc/cmdline:
initrd=\EFI\proxmox\5.4.101-1-pve\initrd.img-5.4.101-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
It does not seems to reflect what I have in /etc/kernel/cmdline ?

 

[leesteken]

Since you are booting with UEFI and systemd-boot, maybe rerun pve-efiboot-tool refresh and reboot?
Can you temporarily mount (one of) your ESP partitions (the command above shows you the paths) and check the loader/entries/*,conf files?

 

[notarobot]

It's working ! Thanks for pointing out that /etc/kernel/cmdline wasn't all on a single line, that was it !
Now because I think this is not really secure I'd like to passthrough only one pci slot.
Is it possible to have a IOMMU group for only one PCI slot instead of all ? I don't know if I'm being clear.

 

[leesteken]

Is it possible to have a IOMMU group for only one PCI slot instead of all ? I don't know if I'm being clear.

You cannot change the IOMMU groups. It is determined by the motherboard and the BIOS. You can put it in different slots and see if that helps. You can try different BIOS versions, or buy a X570 motherboard. And you can use the pcie_acs_override if you don't care about the device isolation, which means that the device passed through to the VM can still talk to the host and/or devices in other VMs.
I would expect the two x16 PCIe slots to be in separate groups each (like X470) or at least the one closest to the CPU (like B450), because they connect directly to the CPU. All x1 PCIe slots are together with other devices that are connected to the B550 chipset. What device in which slot are you trying to passthrough?

 

[notarobot]

I thought the pcie_acs_override=id:nnnn:nnnn command line would give me that.
I'm interested in the 07:00.0 card

 

[leesteken]

I thought the pcie_acs_override=id:nnnn:nnnn command line would give me that.
I'm interested in the 07:00.0 card

The SATA controller? Would it be a work-around to just passthrough the drives instead?
In which PCIe slot (numbered from the CPU) is it? Have you tried putting it in the first PCIe x16 slot? Or the second PCIe x16 slot (but try the first first)?

 

[notarobot]

Yes I tried but it's a pcie 1x port and they are all in the same group.
Passing the drives through qemu could be an option but not in this case as it's an OpenMediavault migration from physical to virtual and I need the disks untouched. If i attached them has virtual SCSI qemu drives the RAID is no more recognized by OpenMediavault.

 

[leesteken]

Someone has had more luck with a related motherboards: the first PCIe (x16) slot was separated. Maybe an earlier BIOS version might help? Some BIOS/AGESA versions are know to break PCI passthrough and/or change IOMMU groups, but I don't know which for your motherboard. Or just accept that you have to override the isolation (which should be fine if you trust the VM), or buy a different motherboard. With this motherboard, it just isn't going to be perfect, but it might be good enough.
EDIT: pcie_acs_override=downstream is probably enough to separate the SATA controller at 07:00.0. No need to split xx:yy.0 from xx:yy.1 etc. which is what multifunction does.

 

[notarobot]

I think that will stay that way until I can buy a better MB. My bad for not checking this before buying.
Do you know if there could be an impact on performances and stability though ?

 

[leesteken]

Do you know if there could be an impact on performances and stability though ?

Probably not. The VM might be able to read/write data of the host via DMA because it is not securely isolated. Or maybe it is securely isolated and your BIOS is just wrong, or the manufacturer could not be bothered to do the grouping correctly. Just don't run untrusted programs and don't allow other users to access the VM, would be my advise.