PCIe Passthrough not working

[brbaker]

Hi,
I am trying to pass through a network device to a VM running pfSense and I cannot get it working. When I start pfSense the network card is not visible at all. I think I have set everything up properly, can anyone suggest what is wrong?

Installed PCI devices are:
# lspci -nn
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
01:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
02:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
03:00.0 Network controller [0280]: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] [8086:24fb] (rev 10)

The device I want to pass through is 01:00.0

Listing the hardware returns:
# Ishw -class network
*-network
description: Ethernet controller
product: 82574L Gigabit Network Connection
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:01:00.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: pm msi pciexpress msix cap_list rom
configuration: driver=vfio-pci latency=0
resources: irq:16 memory:a13c0000-a13dffff memory:a1300000-a137ffff ioport:4000(size=32) memory:a13e0000-a13e3fff memory:a1380000-a13bffff

The VMID is 100, and /etc/pve/qemu-server/100.conf contains:
bootdisk: virtio0
cores: 2
hostpci1: 01:00,pcie=1,rombar=0
ide2: NAS1:iso/pfSense-CE-2.4.4-RELEASE-p3-amd64.iso,media=cdrom
machine: q35
memory: 2048
name: pfSense
net0: virtio=16:B9:C8:87E:54,bridge=vmbr1,firewall=1
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=a18cecad-2bf6-4e81-ad32-aa2b2d7d0142
sockets: 1
usb0: host=148f:2573,usb3=1
virtio0: local-lvm:vm-100-disk-0,size=32G
vmgenid: 412dae1f-41c4-49eb-9a58-398bd33c350b

/etc/default/grub has the following:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

/etc/modules contains:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Validating it is all enabled (this is a bit different to the doco)
# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.012056] ACPI: DMAR 0x000000007E41DBC0 0000A8 (v01 INTEL EDK2 00000002 01000013)
[ 0.182375] DMAR: IOMMU enabled
[ 0.349186] DMAR: Host address width 39
[ 0.349187] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.349192] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.349193] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.349196] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.349197] DMAR: RMRR base: 0x0000007e704000 end: 0x0000007e94dfff
[ 0.349198] DMAR: RMRR base: 0x0000007f800000 end: 0x0000008fffffff
[ 0.349200] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.349201] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.349201] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.351514] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.315529] DMAR: No ATSR found
[ 1.315585] DMAR: dmar0: Using Queued invalidation
[ 1.315588] DMAR: dmar1: Using Queued invalidation
[ 1.345917] DMAR: Intel(R) Virtualization Technology for Directed I/O

Checking they are all in different groups returns:
# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/3/devices/0000:00:12.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.6
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/10/devices/0000:02:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.5
/sys/kernel/iommu_groups/9/devices/0000:00:1f.3
/sys/kernel/iommu_groups/9/devices/0000:00:1f.6
/sys/kernel/iommu_groups/9/devices/0000:00:1f.4

Finally, checking it is all set up properly:
# lspci -nnk
01:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
Subsystem: Intel Corporation Gigabit CT Desktop Adapter [8086:a01f]
Kernel driver in use: vfio-pci
Kernel modules: e1000e

And according to the doco this last step says it should all be good to go.... So I am not sure what is going on.

 

[brbaker]

I think the issue may be related to a FreeBSD kernel bug.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922

I built a Fedora Workstation v31 and was able to pass through the Wifi NIC from the motherboard by adding the following to ID.conf
machine: q35
hostpci0: 03:00,pcie=1,rombar=0

pfSense needs to pick up the new freeBSD kernel patch when it is available.

 

[voodooutt]

This still a bug? I am trying to do the exact same thing. I have a Ethernet PCIe card I am wanting to pass through to my pfSense VM. Can't see the card in pfSense.

 

[brbaker]

This still a bug? I am trying to do the exact same thing. I have a Ethernet PCIe card I am wanting to pass through to my pfSense VM. Can't see the card in pfSense.

View attachment 17157
View attachment 17156

As I said - you cannot pass through PCIe to pfSense as there is a bug in BSD Linux.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922


You apparently can map it through as PCI through if you don't click the PCIe checkbox in the passthrough options in the pve UI.

 

[voodooutt]

Thanks, got it to work by not clicking the PCIe checkbox.