Issues passing through HBA card to TrueNAS

[Keele]

Hi all,

I am having issues passing through a SAS2008 LSI card to a TrueNAS VM in Proxmox.

The installer works no problem running TrueNAS on bare-metal but when I run the TrueNAS installer in a VM with the HBA card passed through, the machine hangs on the following screen:


Obviously it doesn't hang on baremetal so I think I am missing something somewhere?

Hardware:
HP Proliant ML310e Gen8 V2
Xeon CPU E3-1245 v3
32GB DDR3 ECC Memory

This is my GRUB commandline:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on,relax_rmrr allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction"

My VM config:


IOMMU groups are separated OK:


Any help is appreciated

 

[homelabenthusiast]

@Keele try to switch the processor type to host and give it more ram and cpu.

 

[Keele]

@Keele try to switch the processor type to host and give it more ram and cpu.

Not sure why it was set so low.

Anyway I've upped it to 16GB RAM, and 4 cores with host type, but it is still getting stuck on the same part.

 

[Keele]

For clarity, I have tried the following:

Booting TrueNAS Core via USB on baremetal allows the installer to load and I see all the drives populated as normal.


Booting TrueNAS Core inside of Proxmox results in the aforementioned hanging behaviour. Booting TrueNAS Scale OR UnRAID in Proxmox is OK but no drives show up - so clearly the passing through isn't working properly, even if the card is sometimes seen.

 

[cheiss]

Hi,

This is my GRUB commandline:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on,relax_rmrr allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction"

First of the relax_rmrr - are you using https://github.com/kiler129/relax-intel-rmrr or similar?
Can you please post the output of pveversion -v, uname -a and cat /proc/cmdline?

Second, are allow_unsafe_interrupts=1 and pcie_acs_override=downstream,multifunction both actually needed?
Especially he latter can often cause more problems than it solve, if applied without knowing what it does.

Seeing as this is enterprise hardware, they are normally, in most cases, pretty well behaved in regards to e.g. PCI passthrough and don't need any special workarounds.

 

[Keele]

Hi,


First of the relax_rmrr - are you using https://github.com/kiler129/relax-intel-rmrr or similar?
Can you please post the output of pveversion -v, uname -a and cat /proc/cmdline?

Second, are allow_unsafe_interrupts=1 and pcie_acs_override=downstream,multifunction both actually needed?
Especially he latter can often cause more problems than it solve, if applied without knowing what it does.

Seeing as this is enterprise hardware, they are normally, in most cases, pretty well behaved in regards to e.g. PCI passthrough and don't need any special workarounds.

Thanks for your reply.

relax_rmrr is needed on some HP Proliant servers due to their RMRR restrictions. On newer servers this can be disabled but not for older servers. This patch was included in the Proxmox kernal as of version 6.4 iirc. That's needed for IOMMU to work or else an error such as Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. is thrown.

Here is the output of pveversion -v, uname -a and cat /proc/cmdline.

Spoiler: pveversion -v

proxmox-ve: 8.1.0 (running kernel: 6.5.13-1-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5: 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph-fuse: 17.2.7-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.1
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.5-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve2

uname -a : Linux giantprox 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1 (2024-02-05T13:50Z) x86_64 GNU/Linux
cat /proc/cmdline : BOOT_IMAGE=/boot/vmlinuz-6.5.13-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on,relax_rmrr allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction

allow_unsafe_interrupts=1 was added for troubleshooting stems but made no difference so I am ought to remove it.
pcie_acs_override=downstream,multifunction is required to separate my SAS card into its own IOMMU group, which I believe I need for passthrough?

 

[cheiss]

relax_rmrr is needed on some HP Proliant servers due to their RMRR restrictions.

Though they had pretty good firmware normally, but that's alright then.

allow_unsafe_interrupts=1 was added for troubleshooting stems but made no difference so I am ought to remove it.

Yeah, if it does not solve anything, removing it is good - since it can cause stability issue and such.

pcie_acs_override=downstream,multifunction is required to separate my SAS card into its own IOMMU group, which I believe I need for passthrough?

Normally, this parameter should not be needed for passthrough. It forces the PCI subsystem to separate devices into separate groups, but this may still not work depending on the firmware and chipset. So there is no guarantee.

Have you tried putting the SAS card into another (physical) PCIe-slot, if you have others available? Maybe swap cards around if necessary. Again, depending on the firmware, it can put devices into different groups depending on the used PCIe slot. That's the general advice before resorting to above parameter.

And to get another datapoint: Have you tried booting any Linux distribution in the VM with the card passthrough'd? Would be interesting if/what Linux reports there.

 

[Keele]

Yeah, if it does not solve anything, removing it is good - since it can cause stability issue and such.

I've removed it.

Have you tried putting the SAS card into another (physical) PCIe-slot, if you have others available? Maybe swap cards around if necessary. Again, depending on the firmware, it can put devices into different groups depending on the used PCIe slot. That's the general advice before resorting to above parameter

I've tried my x16 slot as well as both of my x8 slots but I haven't been able to separate it into its own IOMMU group.

And to get another datapoint: Have you tried booting any Linux distribution in the VM with the card passthrough'd? Would be interesting if/what Linux reports there.

I tried Ubuntu server 22.04 and I noticed this error whilst booting the installer. It did load to the install screen but none of my drives showed up when it asked me where I wanted to install to.


Looks like some sort of driver issue - maybe it needs blacklisting from the host? But as previously mentioned my SAS card works on baremetal so I don't think the card is at fault here. I will work on it and report back.

 

[Keele]

I borrowed a SAS3008 generation card to try in my server but the TrueNAS installer hangs on the same part as before.

I think the server is just not compatible for PCIe passthrough, since I have determined that the HBA card is not the issue here.

 

[homelabenthusiast]

@Keele just out of curiousity... Have you tried Truenas Scale already to see how it goes?

 

[Keele]

@Keele just out of curiousity... Have you tried Truenas Scale already to see how it goes?

Yes. It boots but none of my drives attached to the HBA card show up. I haven’t tried it with the loan card yet but I imagine it will be the same.

 

[Keele]

Just as an update... I did NOT get this resolved - PCI passthrough does not work on a HP ML310e Gen8.

What I ended up doing instead is using the CPU and RAM from my machine and sticking it in a Supermicro X10SLM+-LN4F mobo I got off eBay for just £15. That board supports PCI passthrough no problems and also has very good IOMMU grouping.