[SOLVED] pci passthrough / IOMMU unable to set up

iri

New Member
Aug 17, 2020
2
0
1
Hello guys,
I am trying to get two HDDs to passthrough via my HBA raid controller (in IT mode) to a FreeNAS VM in Proxmox.

Mainboard: ASRock B550M-ITX ac
CPU: AMD Ryzen 3 3200G

UEFI settings:
SVM Mode -> set enabled
IOMMU -> set enabled.

"lspci -k"
01:00.0 RAID bus controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) Subsystem: Dell 6Gbps SAS HBA Adapter Kernel driver in use: mpt3sas Kernel modules: mpt3sas

I followed the pci_passthrough guide but after checking with "dmesg | grep -e DMAR -e IOMMU" it shows:
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.969858] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter. [ 0.971474] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 13.138593] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

I also added the parameter iommu=pt and pcie_acs_override=downstream to GRUB_CMDLINE_LINUX_DEFAULT but it does not help.

nano /etc/default/grub
GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Proxmox Virtual Environment" GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream" GRUB_CMDLINE_LINUX=""

nano /etc/pve/qemu-server/100.conf
bootdisk: ide0 cores: 2 ide0: local-lvm:vm-100-disk-0,size=8G ide2: none,media=cdrom memory: 8192 name: FreeNAS net0: e1000=D6:DB:C0:91:B5:72,bridge=vmbr0,firewall=1 numa: 0 ostype: other scsihw: virtio-scsi-pci smbios1: uuid=c0bab1d7-5c97-4882-9ad7-9f10ffe9b8d4 sockets: 1 vmgenid: fc15d134-f2a1-4b5c-98cb-2184412d2485 machine: q35 hostpci0: 01:00.0,pcie=1


Maybe any ideas how to solve this?

Thank you very much in advance for your input!
 

avw

Active Member
May 31, 2020
429
68
28
the Netherlands
Your output from dmesg | grep -e DMAR -e IOMMU looks file. I don't see any problems or error messages in your post. Why did you need to use pcie_acs_override?

You can use cat /proc/cmdline to verify that your kernel options are passed correctly.
Check you IOMMU groups using for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done; to make sure that the HBA raid controller is isolated in an IOMMU group (and this also checks whether IOMMU is enabled).
You might need to prevent mpt3sas from loading by blacklisting it and/or adding options vfio-pci ids=... to /etc/modprobe.d/, which also requires loading the VFIO-modules by adding them to /etc/modules. Maybe you have done all this already, but it is not clear to me from your post.
 
  • Like
Reactions: iri

iri

New Member
Aug 17, 2020
2
0
1
It seems to work now, I can see the disks listed in my FreeNAS VM.
I guess the mpt3sas blacklisting did the job as the other stuff was already applied from my side (added VFIO modules, isolated IOMMU group)
Thank you very much!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!