PCI Passthrough Qotom Mini PC

athilopi · Mar 4, 2021

Hi,

I am trying to make PCI Passthrough in Proxmox to OpnSense. Unfortunately I can't get it working. I already checked and posted in the OpnSense forums, but the settings in OpnSense should be fine. So I think the PCI Passthrough is causing the problems, or I want to exclude pci passthrough from the possible errors. I hope you can help me.

I followed the manual from https://www.thomas-krenn.com/en/wiki/Enable_Proxmox_PCIe_Passthrough and I had no problems and I think it (should) work(s).

In the following I add some shell output, I hope this helps.

The only thing what is confusing for me ist the output of qm monitor info pci. There's the ethernet controller with hostpci0 with the device number 16. When I execute find /sys/kernel/iommu_groups/ -type l the "mounted" pci device is in group 20 and has the id 0000:06:00.0. But I don't know whether that's a problem or not.

lsmod | grep vfio


vfio_pci               53248  1
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  10 vfio_pci,kvm
vfio_iommu_type1       32768  1
vfio                   32768  5 vfio_iommu_type1,vfio_pci

find /sys/kernel/iommu_groups/ -type l


/sys/kernel/iommu_groups/17/devices/0000:03:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/15/devices/0000:01:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/13/devices/0000:00:1e.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.2
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:00:1c.4
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/18/devices/0000:04:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.1
/sys/kernel/iommu_groups/16/devices/0000:02:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/14/devices/0000:00:1f.2
/sys/kernel/iommu_groups/14/devices/0000:00:1f.0
/sys/kernel/iommu_groups/14/devices/0000:00:1f.3
/sys/kernel/iommu_groups/14/devices/0000:00:1f.4
/sys/kernel/iommu_groups/4/devices/0000:00:15.1
/sys/kernel/iommu_groups/4/devices/0000:00:15.0
/sys/kernel/iommu_groups/12/devices/0000:00:1c.5
/sys/kernel/iommu_groups/2/devices/0000:00:08.0
[B]/sys/kernel/iommu_groups/20/devices/0000:06:00.0 mounted device in vm[/B]
/sys/kernel/iommu_groups/10/devices/0000:00:1c.3
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:05:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.2

lspci -nnk


00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:5904] (rev 02)
        Subsystem: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:2015]
        Kernel driver in use: skl_uncore
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 620 [8086:5916] (rev 02)
        Subsystem: Intel Corporation HD Graphics 620 [8086:2015]
        Kernel driver in use: i915
        Kernel modules: i915
00:08.0 System peripheral [0880]: Intel Corporation Skylake Gaussian Mixture Model [8086:1911]
        Subsystem: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model [8086:2015]
00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller [8086:9d2f] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller [8086:7270]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Thermal subsystem [8086:9d31] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP Thermal subsystem [8086:7270]
        Kernel driver in use: intel_pch_thermal
        Kernel modules: intel_pch_thermal
00:15.0 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #0 [8086:9d60] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP Serial IO I2C Controller [8086:7270]
        Kernel driver in use: intel-lpss
        Kernel modules: intel_lpss_pci
00:15.1 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #1 [8086:9d61] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP Serial IO I2C Controller [8086:7270]
        Kernel driver in use: intel-lpss
        Kernel modules: intel_lpss_pci
00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-LP CSME HECI #1 [8086:9d3a] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP CSME HECI [8086:1999]
        Kernel driver in use: mei_me
        Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] [8086:9d03] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] [8086:7270]
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port [8086:9d10] (rev f1)
        Kernel driver in use: pcieport
00:1c.1 PCI bridge [0604]: Intel Corporation Device [8086:9d11] (rev f1)
        Kernel driver in use: pcieport
00:1c.2 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port [8086:9d12] (rev f1)
        Kernel driver in use: pcieport
00:1c.3 PCI bridge [0604]: Intel Corporation Device [8086:9d13] (rev f1)
        Kernel driver in use: pcieport
00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #5 [8086:9d14] (rev f1)
        Kernel driver in use: pcieport
00:1c.5 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 [8086:9d15] (rev f1)
        Kernel driver in use: pcieport
00:1e.0 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Serial IO UART Controller #0 [8086:9d27] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP Serial IO UART Controller [8086:7270]
        Kernel driver in use: intel-lpss
        Kernel modules: intel_lpss_pci
00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-LP LPC Controller [8086:9d58] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP LPC Controller [8086:7270]
00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-LP PMC [8086:9d21] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP PMC [8086:7270]
00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-LP HD Audio [8086:9d71] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP HD Audio [8086:7270]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-LP SMBus [8086:9d23] (rev 21)
        Subsystem: Intel Corporation Sunrise Point-LP SMBus [8086:7270]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
01:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
        Subsystem: Intel Corporation I211 Gigabit Network Connection [8086:0000]
        Kernel driver in use: igb
        Kernel modules: igb
02:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
        Subsystem: Intel Corporation I211 Gigabit Network Connection [8086:0000]
        Kernel driver in use: igb
        Kernel modules: igb
03:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
        Subsystem: Intel Corporation I211 Gigabit Network Connection [8086:0000]
        Kernel driver in use: igb
        Kernel modules: igb
04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
        Subsystem: Intel Corporation I211 Gigabit Network Connection [8086:0000]
        Kernel driver in use: igb
        Kernel modules: igb
05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
        Subsystem: Intel Corporation I211 Gigabit Network Connection [8086:0000]
        Kernel driver in use: igb
        Kernel modules: igb
[B]06:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
        Subsystem: Intel Corporation I211 Gigabit Network Connection [8086:0000]
        Kernel driver in use: vfio-pci
        Kernel modules: igb[/B]

qm monitor info pci


Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      PCI subsystem 1af4:1100
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      PCI subsystem 1af4:1100
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      PCI subsystem 1af4:1100
      BAR4: I/O at 0xe0e0 [0xe0ef].
      id ""
  Bus  0, device   1, function 2:
    USB controller: PCI device 8086:7020
      PCI subsystem 1af4:1100
      IRQ 11, pin D
      BAR4: I/O at 0xe080 [0xe09f].
      id "uhci"
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      PCI subsystem 1af4:1100
      IRQ 9, pin A
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1234:1111
      PCI subsystem 1af4:1100
      BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
      BAR2: 32 bit memory at 0xfea74000 [0xfea74fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id "vga"
  Bus  0, device   3, function 0:
    Class 0255: PCI device 1af4:1002
      PCI subsystem 1af4:0005
      IRQ 11, pin A
      BAR0: I/O at 0xe000 [0xe03f].
      BAR4: 64 bit prefetchable memory at 0xfe400000 [0xfe403fff].
      id "balloon0"
  Bus  0, device   5, function 0:
    SCSI controller: PCI device 1af4:1004
      PCI subsystem 1af4:0008
      IRQ 10, pin A
      BAR0: I/O at 0xe040 [0xe07f].
      BAR1: 32 bit memory at 0xfea75000 [0xfea75fff].
      BAR4: 64 bit prefetchable memory at 0xfe404000 [0xfe407fff].
      id "scsihw0"
  Bus  0, device  16, function 0:
    Ethernet controller: PCI device 8086:1539
      PCI subsystem 8086:0000
      IRQ 11, pin A
      BAR0: 32 bit memory at 0xfea40000 [0xfea5ffff].
      BAR2: I/O at 0xe0a0 [0xe0bf].
      BAR3: 32 bit memory at 0xfea70000 [0xfea73fff].
      id "hostpci0"
  Bus  0, device  18, function 0:
    Ethernet controller: PCI device 1af4:1000
      PCI subsystem 1af4:0001
      IRQ 10, pin A
      BAR0: I/O at 0xe0c0 [0xe0df].
      BAR1: 32 bit memory at 0xfea76000 [0xfea76fff].
      BAR4: 64 bit prefetchable memory at 0xfe408000 [0xfe40bfff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net0"
  Bus  0, device  30, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 10, pin A
      BUS 0.
      secondary bus 1.
      subordinate bus 1.
      IO range [0xd000, 0xdfff]
      memory range [0xfe800000, 0xfe9fffff]
      prefetchable memory range [0xfe200000, 0xfe3fffff]
      BAR0: 64 bit memory at 0xfea77000 [0xfea770ff].
      id "pci.1"
  Bus  0, device  31, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 11, pin A
      BUS 0.
      secondary bus 2.
      subordinate bus 2.
      IO range [0xc000, 0xcfff]
      memory range [0xfe600000, 0xfe7fffff]
      prefetchable memory range [0xfe000000, 0xfe1fffff]
      BAR0: 64 bit memory at 0xfea78000 [0xfea780ff].
      id "pci.2"

Ramalama · Mar 4, 2021

Don't passthrough with pcie, untick this in the passthrough settings in the gui.

Cheers

Ramalama · Mar 4, 2021

Ah and add this inside opnsense:

In /boot/loader.conf.local
hw.pci.honor_msi_blacklist=0

The file doesn't exist, but you can create it with vi.

And reboot xD
Cheers

athilopi · Mar 4, 2021

Hey,

thanks for your fast help. If you mean the pci express checkbox. I can't even check it, so it was unchecked.

But I am stuck with creating the file and writing the line. I think that's because of my german keyboard layout. I want to write :wq! or something like this, but I am not able to do it. When I type in q immediately it writes out q is not a vi command.

Ramalama · Mar 5, 2021

Have you set q35 instead of i440(default) as system?

And common... That's vi... You can install nano or find out how vi works xD
Nano is available for opnsense/pfsense through other package repos too xD

Ramalama · Mar 5, 2021

A - edit the line
Esc -> :wq!

Type in the shell simply till you find the right keys xD

athilopi · Mar 5, 2021

Ramalama said:
Have you set q35 instead of i440(default) as system?

And common... That's vi... You can install nano or find out how vi works xD
Nano is available for opnsense/pfsense through other package repos too xD

Where can I set q35 as default? I am a proxmox newbie, so I don't know such trivial things. I posted the screenshot above where I add pcihost via gui. That's the only thing which I knew where to change setting for pcihost.

I created and added the file, after that I rebooted. But it's still not working.

Ramalama · Mar 5, 2021

Send your vm config pls. The file inside etc/pve/qm/...

athilopi · Mar 5, 2021

GNU nano 3.2                  /etc/pve/qemu-server/100.conf                            

boot: order=scsi0;ide2;net0
cores: 2
hostpci0: 06:00.0
ide2: local:iso/OPNsense-20.7-OpenSSL-dvd-amd64.iso,media=cdrom
memory: 2048
name: opnsense
net0: virtio=12:FF:F2:20:6F:02,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
parent: PPPOE
scsi0: local-lvm:vm-100-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=252a4b32-48e6-46f7-80a9-d109d754fc8b
sockets: 1
vmgenid: 7b05701c-95f2-4b4b-9345-45fbbbd51bcf

Ramalama · Mar 5, 2021

okay, make this look like this:

Memory: min 2gb
Processors: min 1 core -> use "host" for better performance
BIOS: OVMF
Display: You use Default VGA, after installation you can use serial0 if you really want
Machine: Q35 (IMPORTANT for Passthrough!)
SCSI Controller: VirtIO (Better Performance)
Harddisk: discard + writeback = better performance
EFI Disk, you need this for OVMF Bios!
hostpci0: 06:00.0
Serial port: (Only needed if you use Display: Serial terminal) You dont need!

Boot Order: you probably need to reinstall opnsense, so don't forget to tick here ide for your CD Image
Use tablet for pointer: has no sense or any function if guest doesn't have a gui
Hotplug: Makes no sense, because it doesn't work anyway
ACPI Support: yes!
QEMU Guest Agent: freebsd doesn't hve guest tools anyway available

Everything else is anyway default, use the same.

You may need probably to reinstall Opnsense. Do it...

Ramalama · Mar 5, 2021

Please dont forget, after you reinstalled, to add his again:
In /boot/loader.conf.local
hw.pci.honor_msi_blacklist=0

If it still doesn't work, then you need to check iommu, bios settings etc..

athilopi · Mar 6, 2021

Hey,

so I did everything as you wrote and "playing" with settings for hours...
Still not working. I've already posted the IOMMU settings and they should be right. As mentioned I followed Thomas Krenns manual. I've been in the OPNsense VM bios, but I don't know what to check there. There are not many options.

I don't know what to do now..

Ramalama · Mar 6, 2021

Well, i know why it doesn't work, but i totally forgot it

You don't have the driver module blacklisted.
And you cannot blacklist it, because you are using the intel nics for your host either.

But there is a way to throw the nics that you want to passthrough out.

One moment, need to find the command xD

Ramalama · Mar 6, 2021

Shut down opnsense, run this command on proxmox. Then proxmox should pull out the nic.

/usr/bin/virsh nodedev-dettach pci_0000_06_00_0

Then start opnsense again.

Cheers

athilopi · Mar 6, 2021

/usr/bin/virsh: No such file or directory

should I install virsh? If yes, how?

Ramalama · Mar 6, 2021

No wait, there is another option to bind vfio to your intel nic.

Just dunno how to do it.
It's easy to bind vfio to vendor:devid, but this will lead you to no network interfaces at all.

Need to google how to bind it to 06:00.0 bus.
There must be a way.

Ramalama · Mar 6, 2021

echo "vfio-pci" > /sys/bus/pci/devices/0000\:06\:00.0/driver_override
modprobe -i vfio-pci

Im not 100% sure if that's right, but try this. In theory your nic should be not visible anymore in the proxmox hist after this.

You can check the output with:
lspci -vvs 0000:06:00.0 | grep driver

There should be vfio pci xD

Ramalama · Mar 6, 2021

Forget my last post, it doesnt work!

But i tryed just myself right now, this works!

1. shutdown opnsense
2.

Bash:

wget https://raw.githubusercontent.com/andre-richter/vfio-pci-bind/master/vfio-pci-bind.sh
chmod +x vfio-pci-bind.sh
./vfio-pci-bind.sh 0000:06:00.0

3. check yourself if it worked:
lspci -vvs 0000:06:00.0 | grep driver -> Kernel driver in use: vfio-pci
4. Start opnsense!

Cheers xD

athilopi · Mar 7, 2021

Ramalama said:
Forget my last post, it doesnt work!

But i tryed just myself right now, this works!

1. shutdown opnsense
2.

Bash:

wget https://raw.githubusercontent.com/andre-richter/vfio-pci-bind/master/vfio-pci-bind.sh chmod +x vfio-pci-bind.sh ./vfio-pci-bind.sh 0000:06:00.0

3. check yourself if it worked:
lspci -vvs 0000:06:00.0 | grep driver -> Kernel driver in use: vfio-pci
4. Start opnsense!

Cheers xD

Without executing the wget script, the output ist already Kernel driver in use: vfio-pci.

Ramalama · Mar 7, 2021

athilopi said:
Without executing the wget script, the output ist already Kernel driver in use: vfio-pci.

Then it should work!

PCI Passthrough Qotom Mini PC

Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Well-Known Member

Well-Known Member

Member

Well-Known Member