SAS2008 passthrough works but fails in VM

[fesarlis]

Hello, I am new to Proxmox.

I am just posting this to make sure I am not wasting my time to do something that can't be done. In particular, to passthrough an SAS2008 (IT-flashed Dell Perc H200 with P20 firmware). Tried to use the HBA in bare metal and it worked like a charm.

I have searched and incorporated almost every suggestion found in other threads.

All this happen on an HP Proliant ML310e Gen8 V2.

1. I have 4 SAS disks attached to the HBA.
2. GRUB cmdline: quiet intel_iommu=on,relax_rmrr iommu=pt pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt intremap=no_x2apic_optout mpt3sas.max_queue_depth=10000 reserve=0x80000000,0xfffffff

The result of all this is that the HBA is passed through properly (lspci inside the VM shows the HBA). HOWEVER there are NO DISKS attached to it and a kernel error occurs (I attach a screenshot).

PVE dmesg logs this:

[ 171.293303] DMAR: DRHD: handling fault status reg 3
[ 171.293330] DMAR: [DMA Read NO_PASID] Request device [07:00.0] fault addr 0x10026b000 [fault reason 0x0c] non-zero reserved fields in PTE
[ 186.482699] DMAR: DRHD: handling fault status reg 3
[ 186.482761] DMAR: [DMA Read NO_PASID] Request device [07:00.0] fault addr 0x10026b000 [fault reason 0x0c] non-zero reserved fields in PTE

Here is the PVE dmesg for mpt. You will notice this is a PVE 7.4 with patched kernel (relaxablermrr). Some people mentioned that going back to kernel 5.15 solved the issue. No luck at all in my case. Tried PVE 7.4, 8, 9.

# dmesg|grep mpt
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.158-2-pve-relaxablermrr root=/dev/mapper/pve-root ro quiet intel_iommu=on,relax_rmrr iommu=pt pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt intremap=no_x2apic_optout mpt3sas.max_queue_depth=10000 reserve=0x80000000,0xfffffff
[ 0.005460] Device empty
[ 0.066510] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.158-2-pve-relaxablermrr root=/dev/mapper/pve-root ro quiet intel_iommu=on,relax_rmrr iommu=pt pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt intremap=no_x2apic_optout mpt3sas.max_queue_depth=10000 reserve=0x80000000,0xfffffff
[ 1.122306] mpt3sas version 40.100.00.00 loaded
[ 1.122804] mpt3sas 0000:07:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 1.124425] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32811852 kB)
[ 1.177651] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 1.177678] mpt2sas_cm0: MSI-X vectors supported: 1
[ 1.177683] mpt2sas_cm0: 0 1 1
[ 1.177853] mpt2sas_cm0: High IOPs queues : disabled
[ 1.177856] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 35
[ 1.177858] mpt2sas_cm0: iomem(0x00000000f7ff0000), mapped(0x00000000248df764), size(65536)
[ 1.177862] mpt2sas_cm0: ioport(0x0000000000004000), size(256)
[ 1.230459] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 1.230463] mpt2sas_cm0: sending message unit reset !!
[ 1.232004] mpt2sas_cm0: message unit reset: SUCCESS
[ 1.259748] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 1.259823] mpt2sas_cm0: request pool(0x00000000749d47bd) - dma(0x112a00000): depth(3492), frame_size(128), pool_size(436 kB)
[ 1.273995] mpt2sas_cm0: sense pool(0x00000000e4963513) - dma(0x113180000): depth(3367), element_size(96), pool_size (315 kB)
[ 1.274078] mpt2sas_cm0: reply pool(0x00000000b2eff303) - dma(0x113200000): depth(3556), frame_size(128), pool_size(444 kB)
[ 1.274083] mpt2sas_cm0: config page(0x000000004ed4655d) - dma(0x113153000): size(512)
[ 1.274085] mpt2sas_cm0: Allocated physical memory: size(7579 kB)
[ 1.274086] mpt2sas_cm0: Current Controller Queue Depth(3364),Max Controller Queue Depth(3432)
[ 1.274087] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[ 1.319490] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x02), BiosVersion(07.39.00.00)
[ 1.319503] mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[ 1.321890] mpt2sas_cm0: sending port enable !!
[ 1.322791] mpt2sas_cm0: hba_port entry: 000000004b1b6e2f, port: 255 is added to hba_port list
[ 1.325073] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x5b8ca3a0e79ac000), phys(8)
[ 1.325890] mpt2sas_cm0: handle(0x9) sas_address(0x5000c500577ff6fd) port_type(0x1)
[ 1.326136] mpt2sas_cm0: handle(0xa) sas_address(0x5000c5006261dc8d) port_type(0x1)
[ 1.326380] mpt2sas_cm0: handle(0xb) sas_address(0x5000c5005770129d) port_type(0x1)
[ 1.326625] mpt2sas_cm0: handle(0xc) sas_address(0x5000c500559b956d) port_type(0x1)
[ 1.340994] mpt2sas_cm0: port enable: SUCCESS
[ 157.461651] mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500559b956d)
[ 157.461668] mpt2sas_cm0: removing handle(0x000c), sas_addr(0x5000c500559b956d)
[ 157.461676] mpt2sas_cm0: enclosure logical id(0x5b8ca3a0e79ac000), slot(4)
[ 157.461685] mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500577ff6fd)
[ 157.461691] mpt2sas_cm0: removing handle(0x0009), sas_addr(0x5000c500577ff6fd)
[ 157.461697] mpt2sas_cm0: enclosure logical id(0x5b8ca3a0e79ac000), slot(7)
[ 157.461705] mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c5006261dc8d)
[ 157.461711] mpt2sas_cm0: removing handle(0x000a), sas_addr(0x5000c5006261dc8d)
[ 157.461716] mpt2sas_cm0: enclosure logical id(0x5b8ca3a0e79ac000), slot(6)
[ 157.461723] mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c5005770129d)
[ 157.461728] mpt2sas_cm0: removing handle(0x000b), sas_addr(0x5000c5005770129d)
[ 157.461734] mpt2sas_cm0: enclosure logical id(0x5b8ca3a0e79ac000), slot(5)
[ 157.461830] mpt2sas_cm0: sending message unit reset !!
[ 157.463394] mpt2sas_cm0: message unit reset: SUCCESS


I would appreciate some guidance. Thank you.