PCI Passthrough
- Thread starter adamb
- Start date
Snag the firmware and try to flash it without the rom file, its very straight forward. You will need the card installed into a machine and a bootable dos USB flash drive with your firmware.
There is IT firmware which you might want to try out also, but I am honestly not to sure. I think you loose raid features with IT mode, unless you are looking for a simple HBA. Should still be able to flash the onboard, really just need the bootable dos USB drive. If you really get stuck, I can dig up the steps I used to flash the cards to get you moving in the right direction.
Thanks for your help
I need it for the first step already
(i'm to afraid to make something wrong)
First I don't need any RAID-Features or something.....
What i have to do in the first step?
Where can i get the right firmware/rom?
THANK YOU!!
Edit:
I ran a Update for the LSI Controller to V16... But i made a mistake i think... I don't have the SAS Address (so i didn't entered it at the end) but the Controller is flashed now... What's about the SAS Address??????
peeehhhhhwwww... found the address on the motherboard
Edit2:
It seems that the Controller is now recognized correctly in the VM... No errors at bootup
I need it for the first step already
(i'm to afraid to make something wrong)First I don't need any RAID-Features or something.....
What i have to do in the first step?
Where can i get the right firmware/rom?
THANK YOU!!
Edit:
I ran a Update for the LSI Controller to V16... But i made a mistake i think... I don't have the SAS Address (so i didn't entered it at the end) but the Controller is flashed now... What's about the SAS Address??????
peeehhhhhwwww... found the address on the motherboard
Edit2:
It seems that the Controller is now recognized correctly in the VM... No errors at bootup
Last edited:
If you dig through this thread you will find a number of links pointing you in the correct direction. If you are worried about bricking the card then I would be a bit leary as your unit can't be replaced without pulling the MB. Tough call, you can also try digging into pointing to the rom file but I am unsure if that can be done. PCI passthrough is a tough one.
Good to hear, im glad more people are using pci passthrough, so far it has been great for us.
I know I'm commenting on an old thread, but this thread came up in my Google searches and hopefully this will help someone out in the future...
I had the same mpt bios error as the OP trying to passthough two LSI 9211i (m1015) and one LSI 1068 (br10i) IT mode JBOD controllers. What worked for me was switching to q35, vfio and using the rombar option.
add to /etc/pve/qemu-server/<vmid>.conf:
This stops the error from showing up at VM start, and the card gets reinitialized when the guest starts. I can see all 24 drives attached to the cards, and ZFS (installed in the guest OS) runs beautifully.
Daniel
I had the same mpt bios error as the OP trying to passthough two LSI 9211i (m1015) and one LSI 1068 (br10i) IT mode JBOD controllers. What worked for me was switching to q35, vfio and using the rombar option.
add to /etc/pve/qemu-server/<vmid>.conf:
Code:
machine: q35
hostpci0: xx:xx.0,driver=vfio,pcie=1,rombar=offThis stops the error from showing up at VM start, and the card gets reinitialized when the guest starts. I can see all 24 drives attached to the cards, and ZFS (installed in the guest OS) runs beautifully.
Daniel
This is a great info! I did get them working but it was a hassle. Im actually looking to do this again so I am hoping your suggestions will make it easier. Greatly appreciated!
I have for days tried to run Freenas under Proxmox and using HBA passthrough without total success.
My approach is two brand new storage servers Huawei 5288 V5 (manufactured early 2019) with onboard LSI HBA (LSI SAS3 MPT Controller sas3008 (TI-Mode)) as well as an onboard LSI based Raidcontroller. (Avago Megaraid sas3004-iMR)
intel_mmio and blacklisting of the HBA works well and the vm can see the controller and the disks but the start of the VM is not clean.. It takes a while and the SeaBios reports "Unable to load Avago Technologies MPT BIOS" MPT BIOS Fault 02h encountered at adapter PCI(01h,00h,00h)
Firmware Fault Code: 2667h
After that error the VM starts *if I give the vm ony 32Gig RAM
If I give it more it will not start but give a time out error.
The server has 64Gig ram and no other VMs defined.
This seems to be a problem that have been reported various times in the past but never addressed.. People with general LSI cards have cleaned their flash and only flashed the ROM part and omitted the BIOS part to get the vm start clean regardless of amout of ram.
Some indicate that SeaBIOS is picking up the SAS HBA bios but i a wrong way and that is the cause of this problem (?)
This does not seem to be an issue on other non linux based hypervisors running bare metal
Is this something that can be fixed i SeaBios or telling the vm not to load the bios part from the HBA ?
The HBA in my servers have only one configurable option. "Enable/Disable Legacy BIOS"
I am aware that Freenas under Proxmox is not encouraged but there shouldn't be issues like this when passing thru PCIe devices to VMs if it works on other platforms. IE not a harware issue.
Regards, Anders
My approach is two brand new storage servers Huawei 5288 V5 (manufactured early 2019) with onboard LSI HBA (LSI SAS3 MPT Controller sas3008 (TI-Mode)) as well as an onboard LSI based Raidcontroller. (Avago Megaraid sas3004-iMR)
intel_mmio and blacklisting of the HBA works well and the vm can see the controller and the disks but the start of the VM is not clean.. It takes a while and the SeaBios reports "Unable to load Avago Technologies MPT BIOS" MPT BIOS Fault 02h encountered at adapter PCI(01h,00h,00h)
Firmware Fault Code: 2667h
After that error the VM starts *if I give the vm ony 32Gig RAM
If I give it more it will not start but give a time out error.
The server has 64Gig ram and no other VMs defined.
This seems to be a problem that have been reported various times in the past but never addressed.. People with general LSI cards have cleaned their flash and only flashed the ROM part and omitted the BIOS part to get the vm start clean regardless of amout of ram.
Some indicate that SeaBIOS is picking up the SAS HBA bios but i a wrong way and that is the cause of this problem (?)
This does not seem to be an issue on other non linux based hypervisors running bare metal
Is this something that can be fixed i SeaBios or telling the vm not to load the bios part from the HBA ?
The HBA in my servers have only one configurable option. "Enable/Disable Legacy BIOS"
I am aware that Freenas under Proxmox is not encouraged but there shouldn't be issues like this when passing thru PCIe devices to VMs if it works on other platforms. IE not a harware issue.
Regards, Anders
Last edited:
I finally had to give Up trying.
If this was a home brew system with regular HBAs I could have tried reflash them without the ROM part but my two systems where brand new Huawei servers intended for production and since there was limited interest in this issue I had to ditch using Proxmoxen as hypervisor.
I guess ESX would have worked but went freenas on bare metal.
If/when this problem gets sorted out I might try converting my freenas boxes to proxmox
Greetings Helle:
I have been working 12 hour days on an extended 4th of July shutdown schedule at my day job, so have had little spare time to work on my PCI passthrough issue at home. However, I am happy to report that I have just figured it out during a break. I now have access to all of my SAS drives via my Dell H710 HBA flashed with IT firmware from within FreeNAS running on a VM under PVE - YES!!! I am super excited as now I can move forward with my project. I know exactly what stumped me, but want to go back and document the steps for those attempting the same. I plan to complete a fresh install of PVE, fresh install of FreeNAS, and will post my CLI steps that got me to this point soon...
I have been working 12 hour days on an extended 4th of July shutdown schedule at my day job, so have had little spare time to work on my PCI passthrough issue at home. However, I am happy to report that I have just figured it out during a break. I now have access to all of my SAS drives via my Dell H710 HBA flashed with IT firmware from within FreeNAS running on a VM under PVE - YES!!! I am super excited as now I can move forward with my project. I know exactly what stumped me, but want to go back and document the steps for those attempting the same. I plan to complete a fresh install of PVE, fresh install of FreeNAS, and will post my CLI steps that got me to this point soon...
That is awesome. Unfortunately I cannot do this unsupported reflashing of the embedded sas controllers on my fairly expensive Huawei servers.. I would rather solve the issue the software way and that doesn't include switching to esx or other supposedly working hypervisors.
Thanks for reporting your success. I wish I could soon report something similar.
/Helle
Last edited:
I also got two M1015 flashed to IT mode which I want to pass through, I tried everything, but my virtual machine doesn't want to recognize the controllers..
lspci -nnk:
Code:
01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (9210-8i) [1000:3040]
Kernel driver in use: vfio-pci
Kernel modules: mpt3sas
02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (9210-8i) [1000:3040]
Kernel driver in use: vfio-pci
Kernel modules: mpt3sasvm-100.conf
Code:
cpu: qemu64
hostpci0: 01:00,pcie=1,romfile=mptsas2.rom
hostpci1: 02:00,pcie=1,romfile=mptsas2.rom
machine: q35
memory: 8192If I don't pass the romfile or don't set rombar to off, then the VM will take forever to boot because it's trying to load the MPT BIOS which also fails with the fault code 2667h.
iommu groups are separate:
Code:
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/12/devices/0000:02:00.0/etc/default/grub
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"/etc/modprobe.d/*
Code:
options kvm ignore_msrs=1
blacklist mpt3sas
options vfio-pci ids=1000:0072
options vfio_iommu_type1 allow_unsafe_interrupts=1But when I execute lspci in FreeNAS, the controllers won't appear, neither does sas2flash/sas3flash recognize any LSI adapters.
It's a completely fresh install of Proxmox VE 6.1 without any additional kernel patches.
Hardware:
Intel Xeon E3-1230v3
ASRock Rack E3C222D4U
For the life of me I don't wanna use ESXi anymore, but if I can't even pass through some SAS controllers I can't really trust in the stability of the system overall...
https://forum.proxmox.com/threads/vm-w-pcie-passthrough-not-working-after-upgrading-to-6-0.56021/
From post
I was having the same problem, this was the fix I found and freenas is running again with cards detected.
make sure you uncheck pci-e or remove pcie=1 before switching to i440fx.