HBA passthough

[Jimmy Brown]

Hello all,

I have a spare PC which I am planning to use for Proxmox server backups, SMB transfer, image library, plex server and other stuff. I am currently having an issue with the HBA not working.
When I add the PCI mapping of the HBA to the Truenas, I get an error.



From this point on, I can't do anything. it stays on this screen. I order for me to get pass this, I need to re install proxmox.

The pc spec:

1. MSI B550 Tomahawk
2. 5950X
3. 1080TI - PCIE slot 1
4. HBA - LSI 9200-8i(IT Mode) - plugged into pcie slot 3
5. x4 HDD drives (x2 4TB, x2 10TB)
6. Samsung 2.5 SSD 256GB
7. M.2 512GB slot 1
8. USB adaptor which has an external m.2 128GB

I wanted to utilise m.2 slot 2 but couldn't as it interferes with the bus lane for HBA.
My next option down the line is to get 2 SSD for redundancy and plug it into the motherboard Sata.

Anyhow, I can see proxmox has detected the HDD/SDD



Proxmox can see the HBA by running this command in shell

lspci | grep -iE "sas|raid|fibre"

root@ryzen:~# lspci | grep -iE "sas|raid|fibre"
02:00.1 RAID bus controller: Advanced Micro Devices, Inc. [AMD] Device 43bd
04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
root@ryzen:~#



I am using the community script to install Truenas- https://community-scripts.org/scripts?q=truenas

I am using the advanced setup process and did not choose any disk as it would be backward approach adding it one by one.


I have created the HBA under Datacenter > Resource mappings



When I create a container for Truenas and try and add the PCIE HBA into the hardware section I see the 1st picture error and can't do anything after that.



When I add the drivers individually using this command, I can see the drives within truenas.

qm set 100 -scsi1 /dev/disk/by-id/ata-ST10000VN0008-2PJ103_ZS512X
qm set 100 -scsi2 /dev/disk/by-id/ata-ST10000VN0008-2PJ103_ZS51FR
qm set 100 -scsi3 /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1PRZP
qm set 100 -scsi3 /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E4YTFZ


Can someone point me into the right direction please

 

[leesteken]

Put the HBA in the first PCIe slot, otherwise the device is not in a separate IOMMU group: https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation
I have posted about the B550 (and AM4 in general) IOMMU groups many times before: https://forum.proxmox.com/search/11...s]=1&c[nodes][0]=16&c[users]=leesteken&o=date

EDIT: Temporarily disable IOMMU in the motherboard BIOS (or the kernel command line in the boot menu) if your Proxmox becomes unresponsive because a VM with PCI(e) passthrough starts automatically. You can then remove the passthrough or disable the auto start.

 

[gfngfn256]

Just for clarity:

You have been clearly warned (in your above posted image) that the HBA controller is not in an isolated IOMMU group (15). This means that it shares the IOMMU group (15) with other devices, check the above in the GUI from your posted image to see all those that carry the same IOMMU group (15).

When you passthrough a device, it becomes unavailable to the host. So if your device does not have correct IOMMU group segregation, you will loose that whole "group" (15) to the host. Hence the havoc you have experienced.

 

[Jimmy Brown]

Put the HBA in the first PCIe slot, otherwise the device is not in a separate IOMMU group: https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation
I have posted about the B550 (and AM4 in general) IOMMU groups many times before: https://forum.proxmox.com/search/11...s]=1&c[nodes][0]=16&c[users]=leesteken&o=date

EDIT: Temporarily disable IOMMU in the motherboard BIOS (or the kernel command line in the boot menu) if your Proxmox becomes unresponsive because a VM with PCI(e) passthrough starts automatically. You can then remove the passthrough or disable the auto start.

Thank you Lee.
I thought I did my research thoroughly. Thank you veery much for pointing me to the article and post.

I actually manually enabled IOMMU in the motherboard. I'll try setting it to 'disable' and moving the HBA to slot pcie 1 and see how that goes.

 

[Jimmy Brown]

Just for clarity:

You have been clearly warned (in your above posted image) that the HBA controller is not in an isolated IOMMU group (15). This means that it shares the IOMMU group (15) with other devices, check the above in the GUI from your posted image to see all those that carry the same IOMMU group (15).

When you passthrough a device, it becomes unavailable to the host. So if your device does not have correct IOMMU group segregation, you will loose that whole "group" (15) to the host. Hence the havoc you have experienced.

Yep, I thought I did all my research and was scratching my head why it kept failing. Thank you for pointing out my mistake!

 

[gfngfn256]

I'll try setting it to 'disable'

IOMMU needs to be enabled to allow correct passthrough of a PCI device.
Leesteken's above advice of disabling the IOMMU in the BIOS was a temporary measure to allow you to regain control of the host - to adjust the current config.

You can read up on the whole passthrough procedure in the Proxmox wiki/doc here.