PCI Passthrough

Hm... what does "no longer have the rom" mean? Where can i find the right rom and how do i flash it (probably there is a manual for that)?
 
I have the LSI SAS2308... hopefully there is a solution for this card? *pray*

Do you mean IT-Mode flash?

Thank you!
 
Last edited:
Snag the firmware and try to flash it without the rom file, its very straight forward. You will need the card installed into a machine and a bootable dos USB flash drive with your firmware.
 
It is a onboard Controller... The result of "Flash without rom" is IT-Mode?

There is IT firmware which you might want to try out also, but I am honestly not to sure. I think you loose raid features with IT mode, unless you are looking for a simple HBA. Should still be able to flash the onboard, really just need the bootable dos USB drive. If you really get stuck, I can dig up the steps I used to flash the cards to get you moving in the right direction.
 
Thanks for your help

I need it for the first step already :( (i'm to afraid to make something wrong)

First I don't need any RAID-Features or something.....

What i have to do in the first step?

Where can i get the right firmware/rom?

THANK YOU!!

Edit:

I ran a Update for the LSI Controller to V16... But i made a mistake i think... I don't have the SAS Address (so i didn't entered it at the end) but the Controller is flashed now... What's about the SAS Address??????

peeehhhhhwwww... found the address on the motherboard :D

Edit2:

It seems that the Controller is now recognized correctly in the VM... No errors at bootup
 
Last edited:
I have the LSI SAS2308... hopefully there is a solution for this card? *pray*Do you mean IT-Mode flash?Thank you!
If you dig through this thread you will find a number of links pointing you in the correct direction. If you are worried about bricking the card then I would be a bit leary as your unit can't be replaced without pulling the MB. Tough call, you can also try digging into pointing to the rom file but I am unsure if that can be done. PCI passthrough is a tough one.
 
Hi,

i flashed to IR-Mode today (Version P16) and now it seems to work. There are some issues with spin-down i have to fix but so far its working. Thanks for your help anyway!
 
Hi,

i flashed to IR-Mode today (Version P16) and now it seems to work. There are some issues with spin-down i have to fix but so far its working. Thanks for your help anyway!

Good to hear, im glad more people are using pci passthrough, so far it has been great for us.
 
I know I'm commenting on an old thread, but this thread came up in my Google searches and hopefully this will help someone out in the future...

I had the same mpt bios error as the OP trying to passthough two LSI 9211i (m1015) and one LSI 1068 (br10i) IT mode JBOD controllers. What worked for me was switching to q35, vfio and using the rombar option.

add to /etc/pve/qemu-server/<vmid>.conf:
Code:
machine: q35
hostpci0: xx:xx.0,driver=vfio,pcie=1,rombar=off

This stops the error from showing up at VM start, and the card gets reinitialized when the guest starts. I can see all 24 drives attached to the cards, and ZFS (installed in the guest OS) runs beautifully.

Daniel
 
  • Like
Reactions: yo78e and janssensm
I know I'm commenting on an old thread, but this thread came up in my Google searches and hopefully this will help someone out in the future...

I had the same mpt bios error as the OP trying to passthough two LSI 9211i (m1015) and one LSI 1068 (br10i) IT mode JBOD controllers. What worked for me was switching to q35, vfio and using the rombar option.

add to /etc/pve/qemu-server/<vmid>.conf:
Code:
machine: q35
hostpci0: xx:xx.0,driver=vfio,pcie=1,rombar=off

This stops the error from showing up at VM start, and the card gets reinitialized when the guest starts. I can see all 24 drives attached to the cards, and ZFS (installed in the guest OS) runs beautifully.

Daniel

This is a great info! I did get them working but it was a hassle. Im actually looking to do this again so I am hoping your suggestions will make it easier. Greatly appreciated!
 
I have for days tried to run Freenas under Proxmox and using HBA passthrough without total success.

My approach is two brand new storage servers Huawei 5288 V5 (manufactured early 2019) with onboard LSI HBA (LSI SAS3 MPT Controller sas3008 (TI-Mode)) as well as an onboard LSI based Raidcontroller. (Avago Megaraid sas3004-iMR)

intel_mmio and blacklisting of the HBA works well and the vm can see the controller and the disks but the start of the VM is not clean.. It takes a while and the SeaBios reports "Unable to load Avago Technologies MPT BIOS" MPT BIOS Fault 02h encountered at adapter PCI(01h,00h,00h)


Firmware Fault Code: 2667h

After that error the VM starts *if I give the vm ony 32Gig RAM
If I give it more it will not start but give a time out error.

The server has 64Gig ram and no other VMs defined.

This seems to be a problem that have been reported various times in the past but never addressed.. People with general LSI cards have cleaned their flash and only flashed the ROM part and omitted the BIOS part to get the vm start clean regardless of amout of ram.

Some indicate that SeaBIOS is picking up the SAS HBA bios but i a wrong way and that is the cause of this problem (?)

This does not seem to be an issue on other non linux based hypervisors running bare metal

Is this something that can be fixed i SeaBios or telling the vm not to load the bios part from the HBA ?
The HBA in my servers have only one configurable option. "Enable/Disable Legacy BIOS"

I am aware that Freenas under Proxmox is not encouraged but there shouldn't be issues like this when passing thru PCIe devices to VMs if it works on other platforms. IE not a harware issue.

Regards, Anders
 
Last edited:
Greetings Helle:

Were you ever able to achieve HBA passthrough to FreeNAS?
I finally had to give Up trying.
If this was a home brew system with regular HBAs I could have tried reflash them without the ROM part but my two systems where brand new Huawei servers intended for production and since there was limited interest in this issue I had to ditch using Proxmoxen as hypervisor.
I guess ESX would have worked but went freenas on bare metal.

If/when this problem gets sorted out I might try converting my freenas boxes to proxmox
 
Greetings Helle:

I have been working 12 hour days on an extended 4th of July shutdown schedule at my day job, so have had little spare time to work on my PCI passthrough issue at home. However, I am happy to report that I have just figured it out during a break. I now have access to all of my SAS drives via my Dell H710 HBA flashed with IT firmware from within FreeNAS running on a VM under PVE - YES!!! I am super excited as now I can move forward with my project. I know exactly what stumped me, but want to go back and document the steps for those attempting the same. I plan to complete a fresh install of PVE, fresh install of FreeNAS, and will post my CLI steps that got me to this point soon...
 
Greetings Helle:

I have been working 12 hour days on an extended 4th of July shutdown schedule at my day job, so have had little spare time to work on my PCI passthrough issue at home. However, I am happy to report that I have just figured it out during a break. I now have access to all of my SAS drives via my Dell H710 HBA flashed with IT firmware from within FreeNAS running on a VM under PVE - YES!!! I am super excited as now I can move forward with my project. I know exactly what stumped me, but want to go back and document the steps for those attempting the same. I plan to complete a fresh install of PVE, fresh install of FreeNAS, and will post my CLI steps that got me to this point soon...
That is awesome. Unfortunately I cannot do this unsupported reflashing of the embedded sas controllers on my fairly expensive Huawei servers.. I would rather solve the issue the software way and that doesn't include switching to esx or other supposedly working hypervisors.

Thanks for reporting your success. I wish I could soon report something similar.

/Helle
 
Last edited:
I know I'm commenting on an old thread, but this thread came up in my Google searches and hopefully this will help someone out in the future...

I had the same mpt bios error as the OP trying to passthough two LSI 9211i (m1015) and one LSI 1068 (br10i) IT mode JBOD controllers. What worked for me was switching to q35, vfio and using the rombar option.

add to /etc/pve/qemu-server/<vmid>.conf:
Code:
machine: q35
hostpci0: xx:xx.0,driver=vfio,pcie=1,rombar=off

This stops the error from showing up at VM start, and the card gets reinitialized when the guest starts. I can see all 24 drives attached to the cards, and ZFS (installed in the guest OS) runs beautifully.

Daniel

I also got two M1015 flashed to IT mode which I want to pass through, I tried everything, but my virtual machine doesn't want to recognize the controllers..

lspci -nnk:

Code:
01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
        Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (9210-8i) [1000:3040]
        Kernel driver in use: vfio-pci
        Kernel modules: mpt3sas
02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
        Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (9210-8i) [1000:3040]
        Kernel driver in use: vfio-pci
        Kernel modules: mpt3sas

vm-100.conf
Code:
cpu: qemu64
hostpci0: 01:00,pcie=1,romfile=mptsas2.rom
hostpci1: 02:00,pcie=1,romfile=mptsas2.rom
machine: q35
memory: 8192

If I don't pass the romfile or don't set rombar to off, then the VM will take forever to boot because it's trying to load the MPT BIOS which also fails with the fault code 2667h.

iommu groups are separate:
Code:
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/12/devices/0000:02:00.0

/etc/default/grub
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"

/etc/modprobe.d/*

Code:
options kvm ignore_msrs=1
blacklist mpt3sas
options vfio-pci ids=1000:0072
options vfio_iommu_type1 allow_unsafe_interrupts=1

But when I execute lspci in FreeNAS, the controllers won't appear, neither does sas2flash/sas3flash recognize any LSI adapters.

It's a completely fresh install of Proxmox VE 6.1 without any additional kernel patches.

Hardware:
Intel Xeon E3-1230v3
ASRock Rack E3C222D4U

For the life of me I don't wanna use ESXi anymore, but if I can't even pass through some SAS controllers I can't really trust in the stability of the system overall...
 
I also got two M1015 flashed to IT mode which I want to pass through, I tried everything, but my virtual machine doesn't want to recognize the controllers..

lspci -nnk:

Code:
01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
        Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (9210-8i) [1000:3040]
        Kernel driver in use: vfio-pci
        Kernel modules: mpt3sas
02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
        Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (9210-8i) [1000:3040]
        Kernel driver in use: vfio-pci
        Kernel modules: mpt3sas

vm-100.conf
Code:
cpu: qemu64
hostpci0: 01:00,pcie=1,romfile=mptsas2.rom
hostpci1: 02:00,pcie=1,romfile=mptsas2.rom
machine: q35
memory: 8192

If I don't pass the romfile or don't set rombar to off, then the VM will take forever to boot because it's trying to load the MPT BIOS which also fails with the fault code 2667h.

iommu groups are separate:
Code:
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/12/devices/0000:02:00.0

/etc/default/grub
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"

/etc/modprobe.d/*

Code:
options kvm ignore_msrs=1
blacklist mpt3sas
options vfio-pci ids=1000:0072
options vfio_iommu_type1 allow_unsafe_interrupts=1

But when I execute lspci in FreeNAS, the controllers won't appear, neither does sas2flash/sas3flash recognize any LSI adapters.

It's a completely fresh install of Proxmox VE 6.1 without any additional kernel patches.

Hardware:
Intel Xeon E3-1230v3
ASRock Rack E3C222D4U

For the life of me I don't wanna use ESXi anymore, but if I can't even pass through some SAS controllers I can't really trust in the stability of the system overall...

https://forum.proxmox.com/threads/vm-w-pcie-passthrough-not-working-after-upgrading-to-6-0.56021/
From post
pcie is only available in q35, but you can passthrough pcie devices as pci devices using i440fx, and since the pci hardware is all virtual there should be no noticable speed impact

I was having the same problem, this was the fix I found and freenas is running again with cards detected.

make sure you uncheck pci-e or remove pcie=1 before switching to i440fx.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!