Proxmox (v7.1-10) passthrough problem (LSI3008 SAS controller to TrueNAS Core)

[thimplicity]

Hi everyone,
I am new to proxmox. I got it installed and installed TrueNAS as well as a few containers for Docker. I have an on-board LSI 3008 SAS controller on my Asrock Rack EPC612D4U-8R board that I flashed into IT mode. I connected two SATA drives to it to test. Now, the "only" thing missing is the PCI(e) passthrough from proxmox to TrueNAS Core. I followed the steps in the overall proxmox instructions, but unfortunately it does not work. As soon as I add the controller to the TrueNAS VM, it will not boot and is stuck in the "booting" although I boot from an NVME and not from one of the drives connected to the controller. The drive is shown in the BIOS of the controller.

Thanks for some guidance and let me know which additional information I would need to provide.

 

[jpeppard]

Can you access shell on the proxmox node and issue command "pveversion"?

The latest kernel (5.13.19-4-pve) seems to be breaking PCI passthrough. If you are running 5.13.19-4-pve, try booting to an older kernel from the GRUB boot menu. 5.13.19-3-pve is working fine for me.

 

[thimplicity]

Can you access shell on the proxmox node and issue command "pveversion"?

The latest kernel (5.13.19-4-pve) seems to be breaking PCI passthrough. If you are running 5.13.19-4-pve, try booting to an older kernel from the GRUB boot menu. 5.13.19-3-pve is working fine for me.

pve-manager/7.1-10/6ddebafe (running kernel: 5.13.19-3-pve)

This is what I get, so it seems it is not the kernel version

 

[jpeppard]

How long are you waiting for it to boot? My LSI 9207 would take upwards of 2 to 3 full minutes to initialize while booting my TrueNAS VM. Once I disabled the boot function on the LSI card BIOS it takes about 30 seconds.

 

[thimplicity]

How long are you waiting for it to boot? My LSI 9207 would take upwards of 2 to 3 full minutes to initialize while booting my TrueNAS VM. Once I disabled the boot function on the LSI card BIOS it takes about 30 seconds.

What do you mean with "it"? The TrueNAS VM or the controller?
How would I disable the boot? Do you mean the setting that I would want to boot from the controller in the motherboard BIOS?

 

[jpeppard]

What do you mean with "it"? The TrueNAS VM or the controller?
How would I disable the boot? Do you mean the setting that I would want to boot from the controller in the motherboard BIOS?

I mean how long are you waiting for the TrueNAS vm to boot? I assume that is what is getting "stuck"

Do you have a screenshot of what you see on the vm console when it is stuck?

On my LSI card I hit "Ctrl C" when prompted during the VM boot to access the card bios.

Also, now that I'm thinking about it, has the onboard LSI been flashed to IT mode?

 

[neiion]

having similar issues after upgrading to lastest Kernel. getting the following error:
TASK ERROR: Cannot bind 0000:43:00.0 to vfio

this pci device is an LSI SAS2008 pci-express Fusion-MPT SAS-2 [Falcon] [IT Mode Flashed] passed through to a VM (truenas core)

had to revert to Linux 5.13.19-3-pve at the proxmox boot options for vm to boot correctly .

 

[thimplicity]

I mean how long are you waiting for the TrueNAS vm to boot? I assume that is what is getting "stuck"

Do you have a screenshot of what you see on the vm console when it is stuck?

On my LSI card I hit "Ctrl C" when prompted during the VM boot to access the card bios.

Also, now that I'm thinking about it, has the onboard LSI been flashed to IT mode?

View attachment 33916

I did not even know you can influence the starting time (except by doing it manually of course ). Let me check that one. I just set it to auto-boot.

And yes, I flashed it into IT mode earlier today

 

[jpeppard]

I did not even know you can influence the starting time (except by doing it manually of course ). Let me check that one. I just set it to auto-boot.

And yes, I flashed it into IT mode earlier today

Definitely give it a try. As long as you see that ctrl C option for the HBA BIOS when you boot the TrueNAS VM the passthrough itself should be working. If you don't get that far it may be something else.

 

[thimplicity]

OK, I figured it out. One big thing and three smaller things:

Big: Change from PCI to PCIe
Small:
- Deactivate boot from LSI controller in motherboard BIOS
- Delay Truenas VM start by 2 mins
- Deactivate Rom-Bar in the VM settings

Thanks @jpeppard for helping!

 

[jpeppard]

OK, I figured it out. One big thing and three smaller things:

Big: Change from PCI to PCIe
Small:
- Deactivate boot from LSI controller in motherboard BIOS
- Delay Truenas VM start by 2 mins
- Deactivate Rom-Bar in the VM settings

Thanks @jpeppard for helping!

Awesome, glad to hear it's working!