OMV - pci passthrough

[mkyb14]

Looking to see if anyone else has setup OMV with pci passthrough successfully. Currently I've followed the Wiki and a few other googles to see if I've done anything outside the norm.

So far I've followed the wiki, running an intel processor e3 1230 v5, enabled everything, located the PCI card as 01:00.0 etc.

currently this is a supermicro sc846, with an m1015 IT mode card no boot option enabled.

lspci shows the card in the console
find /sys/kernal/immou_groups/ shows it under group 1
edited the vm.conf to add machine q35 and hostpci0: 01:00.0,pcie=1

both vm and host have been rebooted. Far as I can tell, nothing is showing up in OMV.



https://pve.proxmox.com/wiki/Pci_passthrough#Required_modules


at a loss for the moment on what to test next?

 

[dcsapak]

can you post the lspci output from the host and the vm ?

 

[mkyb14]

https://drive.google.com/file/d/0B15p4ppbxdipWi1PZ2J6YVFSTjQ/view?usp=sharing

https://drive.google.com/file/d/0B15p4ppbxdipS0ZiREF1NHNVX0U/view?usp=sharing

 

[mkyb14]

what's funny now, is it locks up when I start the VM for OVM. I'm going to remove the new hostpci0 from the .conf file and see if that unlocks the host. right now, I have to hard reboot, getting nothing but input/output errors.

 

[mkyb14]

here is the output from dmesg DMAR

https://drive.google.com/file/d/0B15p4ppbxdipS0ZiREF1NHNVX0U/view?usp=sharing

dmesg | grep cap
root@pve:~# dmesg | grep ecap

[ 0.043179] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da

-a at the end states is compatible...

 

[mkyb14]

for the sake of help, i've recorded the commands used in the wiki to show the output... I'm missing something or not interpreting the output correctly to fix the issue.

https://drive.google.com/file/d/0B15p4ppbxdipLUtMNWtEVTVVZ0E/view?usp=sharing
google drive plays in 360p, you'll have to download it for HD.

 

[dcsapak]

can you post the output of

find /sys/kernel/iommu_groups -type l | sort -t '/' -n -k 5

also did you blacklist the driver on the host, so that the host does not try to use the device?
also if you modify /etc/modprobe.d/ files you have to run

update-initramfs -u

afterwards

 

[mkyb14]

root@pve:~# find /sys/kernel/iommu_groups -type l | sort -t '/' -n -k 5

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:14.0
/sys/kernel/iommu_groups/2/devices/0000:00:14.2
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.1
/sys/kernel/iommu_groups/4/devices/0000:00:17.0
/sys/kernel/iommu_groups/5/devices/0000:00:1c.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.1
/sys/kernel/iommu_groups/7/devices/0000:00:1c.2
/sys/kernel/iommu_groups/8/devices/0000:00:1c.3
/sys/kernel/iommu_groups/9/devices/0000:00:1c.6
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.2
/sys/kernel/iommu_groups/10/devices/0000:00:1f.4
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/12/devices/0000:04:00.0
/sys/kernel/iommu_groups/13/devices/0000:05:00.0
/sys/kernel/iommu_groups/14/devices/0000:06:00.0
/sys/kernel/iommu_groups/15/devices/0000:07:00.0
/sys/kernel/iommu_groups/15/devices/0000:08:00.0

As for blacklisting, i'll have to google it unless you have a link to a guide. Thanks for the direction.

 

[mkyb14]

ok, so every time I try and start the VM, with:
hostpci0: 01:00.0,pcie=1
machine: q35

the whole system locks up. cannot force quite anything. it would seem since the Raid card and the SAS2008 JBOD card on in the same group it's locking up the system trying to passthrough to the VM. What other commands could I run to try and troubleshoot this?

I've edited the blacklist.conf to include:

# This file contains a list of modules which are not supported by Proxmox VE
# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist mpt2sas

 

[mkyb14]

Any thoughts from the community?

 

[SwampRabbit]

Any thoughts from the community?

@mkyb14

I've gotten two Supermicro HBAs to passthrough to OMV, I didn't have a lot of luck with the IBM (M1015 and M1115) or HighPoint HBAs I have.
After a lot of fussing with the VM config settings, simply swapping HBAs did the trick for me. FYI, this is running on a older Supermicro motherboard though so that may have been part of the problem.

There were a few "gotchas" during my testing, also the wiki entries for this aren't the greatest either (at least last I looked).

I made a few quick notes but they are on a external drive, which I can try and pull up and share with you tomorrow, pending my workload.

 

[mkyb14]

@SwampRabbit , I'd appreciate it if you have time.

 

[SwampRabbit]

@mkyb14 so it seems my notes weren't that good, probably because I was doing it as a test while I waited for PVE 5.0 to be officially released.

The system (old FreeNAS server) that will run my OMV VM is:
CPU - Xeon 1230v2
Motherboard - Supermicro X9SCM-F-O
Memory - 32GB Crucial 1600mhz ECC
2 x SUPERMICRO AOC-SAS2LP-MV8

I tried both IBM M1015 and M1115 in IT mode but when it came to passing the controllers over to OMV things got messed up.
I did blacklist mpt2sas and mpt3sas in PVE but it seem to work when I was testing the IBM controllers, but I could have given up too soon.
I think as far as I could get was passing through the individual disks to OMV, not the IBM controllers themselves.

To get the Supermicro controllers to work I have:

blacklist mvsas
update-initramfs -u

dcsapak already shared the update-initramfs command, this isn't in the wiki from what I remember.
I always blacklist, update-initramfs, then reboot so I found it odd it wasn't in the wiki.

My OMV VM config file has the following for the two Supermicro controllers:

hostpci0: 01:00.0,pcie=1
hostpci1: 02:00.0,pcie=1

From my notes that worked for me, but I might have had all three blacklisted because I didn't need them anyway.

None of that may help you because it seems you already are working those items.
I can say that I think part of the problem is that the Supermicro boards can come with multiple SATA/soft RAID controllers which try to activate modules and drivers at boot, blacklisting just one may not fix the issue.

You could try and blacklist mpt2sas, mpt3sas, and mvsas, update-initramfs, then reboot.
Then check controller info with

lspci -k

to make sure PVE is leaving the controllers alone.

I do plan on starting to bring this setup online with PVE 5 now that its been out awhile, so I can attempt the IBM controllers again.

 

[mkyb14]

@SwampRabbit thank you for the notes. I've added mvsas to the blacklist and rebooted.

I'm not sure how you tell if PVE is leaving it alone. Below is the output from the lspci -k
I've for the 01:00.0 being the IBM1015 IT, and a regular raid 9270 below. I'm looking to only pass the IBM to the OMV VM.



root@pve:~# lspci -k

00:00.0 Host bridge: Intel Corporation Skylake Host Bridge/DRAM Registers (rev 07)

Subsystem: Super Micro Computer Inc Skylake Host Bridge/DRAM Registers

Kernel driver in use: ie31200_edac

Kernel modules: ie31200_edac

00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 07)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:01.1 PCI bridge: Intel Corporation Skylake PCIe Controller (x8) (rev 07)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)

Subsystem: Super Micro Computer Inc Sunrise Point-H USB 3.0 xHCI Controller

Kernel driver in use: xhci_hcd

00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H Thermal subsystem (rev 31)

Subsystem: Super Micro Computer Inc Sunrise Point-H Thermal subsystem

Kernel driver in use: intel_pch_thermal

Kernel modules: intel_pch_thermal

00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)

Subsystem: Super Micro Computer Inc Sunrise Point-H CSME HECI

Kernel modules: mei_me

00:16.1 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #2 (rev 31)

Subsystem: Super Micro Computer Inc Sunrise Point-H CSME HECI

Kernel modules: mei_me

00:17.0 RAID bus controller: Intel Corporation C600/X79 series chipset SATA RAID Controller (rev 31)

Subsystem: Intel Corporation C600/X79 series chipset SATA RAID Controller

Kernel driver in use: ahci

Kernel modules: ahci

00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #1 (rev f1)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:1c.1 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #2 (rev f1)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:1c.2 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #3 (rev f1)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:1c.3 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #4 (rev f1)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:1c.6 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #7 (rev f1)

Kernel driver in use: pcieport

Kernel modules: shpchp

00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)

Subsystem: Super Micro Computer Inc Sunrise Point-H LPC Controller

00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)

Subsystem: Super Micro Computer Inc Sunrise Point-H PMC

00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)

Subsystem: Super Micro Computer Inc Sunrise Point-H SMBus

Kernel driver in use: i801_smbus

Kernel modules: i2c_i801

01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]

Kernel modules: mpt3sas

02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] (rev 05)

Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9270-8i

Kernel driver in use: megaraid_sas

Kernel modules: megaraid_sas

03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection

Kernel driver in use: igb

Kernel modules: igb

04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection

Kernel driver in use: igb

Kernel modules: igb

05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection

Kernel driver in use: igb

Kernel modules: igb

06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection

Kernel driver in use: igb

Kernel modules: igb

07:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)

Kernel modules: shpchp

08:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)

Subsystem: Super Micro Computer Inc ASPEED Graphics Family

Kernel driver in use: ast

Kernel modules: ast

 

[mkyb14]

If it is passing through correctly (which i'm unsure of). Within OMV, scanning disks shows nothing. Not sure if I'm supposed to be doing something to make them show up etc.

I've included three images of the PVE setup for OMV incase I setup the VM somehow incorrectly for SFF passthrough to the host.

https://drive.google.com/file/d/0B15p4ppbxdipLUtMNWtEVTVVZ0E/view?usp=sharing

https://drive.google.com/file/d/0B15p4ppbxdipbGJSZ1NJN0tRT0U/view?usp=sharing

https://drive.google.com/file/d/0B15p4ppbxdipWmRsYUUyTWlscUk/view?usp=sharing

 

[SwampRabbit]

The mvsas is for Marvel controllers, my Supermicro ones have Marvel controllers. Sorry if that was confusing, posted that very late.

From what I understand from the lspci-k output is that your M1015 is using the mpt3sas driver, so that is the one you should blacklist. But I would blacklist mpt2sas as well, just in case it tries to fall back to that.

Blacklisting these shouldn't affect your 9270 since that is using mrsas / "megaraid sas".

I can spin up my server sometime next week and drop one of the M1015 or M1115s in and test with passing them through while also passing through the Supermicro controller.

One thing you can try is pulling out the 9270 and boot from one of the onboard sata connections, blacklisting the mpt2sas and mpt3sas, then see if the M1015 passes through.

Are you using the same OMV VM over and over while testing? I would try doing a fresh OMV VM install after your blacklist.

 

[mkyb14]

ok, progress. I kept wondering why I could never do the "find" command and get the group types to list... eventually moved the m1015 to a lower x4 PCI slot and it worked.

hostpci0: 09:00.0,pcie=1
machine: q35

double checked blacklist, mpt2 and mpt3 and mvsas etc.

Deleted and reinstalled OMV, went to hard disks and there they were...

Now, I thought I was in the clear and went to create a raid array, error. even trying to simply wipe the drive error.

mpt2sas_cm0: fault_state(0x2622) !

no idea what that means. the m1015 uses the mpt3sas module, so I want back and unblacklisted the mpt2sas etc.. nothing.

Added driver=vfio,rombar etc... this actually stops the drives from showing, so I removed them.

So right now, I can passthrough the disks, but if I try and wipe or do anything with them, I get the error above and eventually in the GUI and communication failure error....

**also, I changed the pci 3.0 x4 slot on the MB to Gen2 per another article using super micro boards.
Different motherboard, worth a shot, didn't do anything .
https://forums.freenas.org/index.ph...card-and-disk-detection-problems.21118/page-2


At a loss again.

 

[SwampRabbit]

I am pretty sure the M1015s need a x8 slot, never tried them in a x4 slot. Interesting that you got some response from it.
I also don't think that they are UEFI compatible if that is worth anything related to this.

It is odd that PVE uses the mpt3sas module but OMV shows it using the mpt2sas instead. That could also be part of the issue, although I don't know what to get from those errors.

You can try and pass through the controller but only pass through the disks to the VMs. I think the wiki has info on that. Only downfall is that you don't get smart data if I remember correctly.

 

[gaz]

Most PCIe cards can run with fewer lanes, as long as the card physically fits in the slot.
One thing to note - you cannot pass through a PCIe card connected to the processor lanes on an E3 Xeon. That's why it wouldn't work when it was at address 01:00. Changing the board into a chipset PCIe slot solves that problem (in the 09:00 position).

Not sure why you're seeing errors, but it's at least passing through something.

 

[martini dry]

hostpci0: 09:00.0,pcie=1
machine: q35

mkyb14, try without "pcie=1". It worked for me. I am running OMV 3 @ Proxmox 5 on Supermicro X10SRH-CLN4F and I passed through the on-board LSI SAS 3008. However, I am not sure how big impact it might have on performance to omit "pcie=1" option.