Pass-through: SAS2208 HBA works, SAS3408 doesn't

[brunokc]

Hi all,

New to proxmox and new to the forums.

I've been finally able to setup a Proxmox on my main server at home and I'm off and running with a few VMs and now playing with CTs for the first time.

I have some storage that I'd like to pass to a TrueNAS Scale VM I'm setting up. I have 3 HBAs on this machine: 2 SAS2208 HBAs and a SAS3408. This is a temporary situation until I can migrate some data. One of the SAS2208 is for the host itself. The second one is attached to a bunch of disks from an older system (2 large arrays NTFS formatted). The SAS3408 was brough in to handle new disks and to facilitate the migration from the old disks to the new ones. Ultimately the intention is to permanently pass the SAS3408 to TrueNAS but I keep getting an error when I try to start the VM:

[209039.655840] vfio-pci 0000:82:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.

The VM doesn't start.

I've been researching the subject, but other than a kernel patch to allow me to pass "relax_rmrr" as a parameter to the intel_iommu module, I couldn't find anything definitive about it. On a wimp, I decided to try to pass the second SAS2208 and to my surprise that worked just fine.

This is on Promox 7.4 with the latest updates as of today and running with kernel 5.15.107.

The VM config:

agent: 1
balloon: 8192
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 8
cpu: host
efidisk0: vmstorage:vm-500-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:82:00,pcie=1,rombar=0
ide2: local:iso/TrueNAS-SCALE-22.12.2.iso,media=cdrom,size=1699538K
machine: q35
memory: 16384
meta: creation-qemu=7.2.0,ctime=1682885372
name: Jupiter
net0: virtio=<mac>,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
scsi0: vmstorage:vm-500-disk-0,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=eb851d72-95e7-4763-af76-bc9b9f0e5466
sockets: 1
vmgenid: 6960e40b-3a86-4701-b3cc-cd4fef9b7925

Host config:

root@pvemain:~# lspci -nnk -s 82:00
82:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID Tri-Mode SAS3408 [1000:0017] (rev 01)
Subsystem: Broadcom / LSI MegaRAID Tri-Mode SAS3408 [1000:9441]
Kernel driver in use: vfio-pci
Kernel modules: megaraid_sas

root@pvemain:~# lsmod | grep vfio
vfio_pci 16384 0
vfio_pci_core 73728 1 vfio_pci
vfio_virqfd 16384 1 vfio_pci_core
vfio_iommu_type1 45056 0
vfio 45056 2 vfio_pci_core,vfio_iommu_type1
irqbypass 16384 114 vfio_pci_core,kvm

root@pvemain:~# cat /etc/default/grub | grep iommu
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

root@pvemain:~# find /sys/kernel/iommu_groups/ -type l | grep :82
/sys/kernel/iommu_groups/81/devices/0000:82:00.0
root@pvemain:~# find /sys/kernel/iommu_groups/ -type l | grep /81
/sys/kernel/iommu_groups/81/devices/0000:82:00.0

root@pvemain:~# lspci -nnk -s 81:00
81:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID SAS 2208 [Thunderbolt] [1000:005b] (rev 01)
Subsystem: Broadcom / LSI MegaRAID SAS 9265-8i [1000:9265]
Kernel driver in use: megaraid_sas
Kernel modules: megaraid_sas

root@pvemain:~# find /sys/kernel/iommu_groups/ -type l | grep :81
/sys/kernel/iommu_groups/80/devices/0000:81:00.0
root@pvemain:~# find /sys/kernel/iommu_groups/ -type l | grep /80
/sys/kernel/iommu_groups/80/devices/0000:81:00.0

Has anyone here encountered a similar problem?

Thanks!