[SOLVED] VM Timeout with PCI HBA card

[newpain01]

I have a Dell Precision T3610 with a 10 core Xeon and 96GB ECC RAM running Proxmox. Just installed TrueNAS Scale and it was booting fine until I configured PCI passthrough. I enabled IOMMU and I can see all hard drives from the HBA in Proxmox, so the card is working fine.

I am allocating 4 cores and 16GB RAM to the VM.

The HBA card is seen as:

03:00.0 RAID bus controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
        Subsystem: Fujitsu Technology Solutions HBA Ctrl SAS 6G 0/1 [D2607]
        Flags: bus master, fast devsel, latency 0, IRQ 32, IOMMU group 17
        I/O ports at d000 [size=256]
        Memory at fb600000 (64-bit, non-prefetchable) [size=16K]
        Memory at fb580000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at fb100000 [disabled] [size=512K]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [d0] Vital Product Data
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

root@pve:~# cat /etc/pve/qemu-server/100.conf
boot: order=scsi0
cores: 4
hostpci0: 0000:03:00.0,pcie=1,rombar=0
machine: q35
memory: 16536
meta: creation-qemu=6.2.0,ctime=1675873673
name: TrueNAS
net0: virtio=E6:0F:2C:A6:D2:42,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: VMpool:vm-100-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=6f2bc71b-4eb6-4101-8cae-d405e5328aa0
sockets: 1
vmgenid: d8e98f8f-411a-4619-8b6b-7b87e93f1e58

I found this thread and used his settings from comment #3, but it doesn't work for me.

As soon as I click Start on the VM (or run `qm start 100` the Proxmox server becomes unresponsive and I have to physically turn it off and start it again.
I am not able to tail logs from `/var/log/syslog` during startup.

Any ideas what might cause this and how to troubleshoot it?

 

[Neobin]

Please provide the full output from the PVE-host in code-tags each of: cat /proc/cmdline and:

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

Also, to be sure: Your PVE-system-disk(s) and/or any guest-storage is not connected to that HBA?!

 

[newpain01]

Please provide the full output from the PVE-host in code-tags each of: cat /proc/cmdline and:

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

Also, to be sure: Your PVE-system-disk(s) and/or any guest-storage is not connected to that HBA?!

My PVE SSDs are connected to the same HBA, on a different channel, is that the problem?
I have 2 SSDs for Proxmox in Raid 1 and 2 SSDs for the VMs also in Raid 1, all 4 connected to one channel on the HBA and then 4x14TB HDDs connected to the other channel on the HBA.

 

[Neobin]

My PVE SSDs are connected to the same HBA, on a different channel, is that the problem?

With PCIe-passthrough the whole device, including all its functions and also all that is connected to it, is passed through to the VM and the host "loses" it; so yes, this is a problem.

 

[newpain01]

With PCIe-passthrough the whole device, including all its functions and also all that is connected to it, is passed through to the VM and the host "loses" it; so yes, this is a problem.

Oh wow, I wasn't aware that the whole device is passed through to the VM, I thought it would still be available in Proxmox.
Thank you so much, I spent days troubleshooting this and couldn't figure it out.