[SOLVED] pci passthrough / IOMMU unable to set up

[iri]

Hello guys,
I am trying to get two HDDs to passthrough via my HBA raid controller (in IT mode) to a FreeNAS VM in Proxmox.

Mainboard: ASRock B550M-ITX ac
CPU: AMD Ryzen 3 3200G

UEFI settings:
SVM Mode -> set enabled
IOMMU -> set enabled.

"lspci -k"
01:00.0 RAID bus controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) Subsystem: Dell 6Gbps SAS HBA Adapter Kernel driver in use: mpt3sas Kernel modules: mpt3sas

I followed the pci_passthrough guide but after checking with "dmesg | grep -e DMAR -e IOMMU" it shows:
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.969858] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter. [ 0.971474] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 13.138593] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

I also added the parameter iommu=pt and pcie_acs_override=downstream to GRUB_CMDLINE_LINUX_DEFAULT but it does not help.

nano /etc/default/grub
GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Proxmox Virtual Environment" GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream" GRUB_CMDLINE_LINUX=""

nano /etc/pve/qemu-server/100.conf
bootdisk: ide0 cores: 2 ide0: local-lvm:vm-100-disk-0,size=8G ide2: none,media=cdrom memory: 8192 name: FreeNAS net0: e1000=D6:DB:C0:91:B5:72,bridge=vmbr0,firewall=1 numa: 0 ostype: other scsihw: virtio-scsi-pci smbios1: uuid=c0bab1d7-5c97-4882-9ad7-9f10ffe9b8d4 sockets: 1 vmgenid: fc15d134-f2a1-4b5c-98cb-2184412d2485 machine: q35 hostpci0: 01:00.0,pcie=1


Maybe any ideas how to solve this?

Thank you very much in advance for your input!

 

[leesteken]

Your output from dmesg | grep -e DMAR -e IOMMU looks file. I don't see any problems or error messages in your post. Why did you need to use pcie_acs_override?

You can use cat /proc/cmdline to verify that your kernel options are passed correctly.
Check you IOMMU groups using for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done; to make sure that the HBA raid controller is isolated in an IOMMU group (and this also checks whether IOMMU is enabled).
You might need to prevent mpt3sas from loading by blacklisting it and/or adding options vfio-pci ids=... to /etc/modprobe.d/, which also requires loading the VFIO-modules by adding them to /etc/modules. Maybe you have done all this already, but it is not clear to me from your post.

 

[iri]

It seems to work now, I can see the disks listed in my FreeNAS VM.
I guess the mpt3sas blacklisting did the job as the other stuff was already applied from my side (added VFIO modules, isolated IOMMU group)
Thank you very much!