[SOLVED] Passing NIC to VM kills networking and usb input (PCI passthrough)

[PacDDrake]

I am trying to set up pfSense in a VM based on Techno Tim's tutorial video (except for UEFI).

I also recently acquired a Dell 5719 quad port PCI-e which I want to use for both WAN and LAN connectivity.

However, when attempting to start the VM, my network connection dies and I am unable to use the keyboard.
Not being able to send inputs to Proxmox makes troubleshooting this issue quite hard.

Has anyone encountered similar issues or does anyone have any suggestions on how to troubleshoot the issue?

 

[Dunuin]

Verify that the NIC got its own IOMMU group. It might help to try another PCIe port that is directly connected to the CPU and not just to the chipset. If connected to the chipset the PCIe device is often in a group together with other onboard stuff like USB controllers, onboard NIC and so on. So passing through the NIC to the VM might also passthrough other devices of the same IOMMU group und things like USB not working anymore might happen.

 

[PacDDrake]

I have done the following:

• edited /etc/default/grub and changed GRUB_CMDLINE_LINUX_DEFAULT="quiet" to GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset"
• applied changes with "update-grub"
• added vfio modules to /etc/modules: vfio, vfio_pci, vfio_virqfd
• applied changes with "update-initramfs -u"
• rebooted

After that I confirmed that each port on the NIC has its own IOMMU group.

When trying to start the VM my network and usb stay up, however I get the following error:

"kvm: -device vfio-pci,host=0000:04:00.0,id=hostpci1.0,bus=ich9-pcie-port-2,addr=0x0.0,multifunction=on: vfio 0000:04:00.0: device is already attached"

I have also noticed that, once I try to start the VM, the network interfaces from my NIC don't show up in proxmox anymore with "ip a".

I assume that one of the flags I set for grub causes this, however, I am not sure which one or if at all.
My second guess is that vfio is messing something up there.

Any advice on how to proceed?

 

[PacDDrake]

Update: I managed to resolve the issue.

The problem was in the configuration file of the VM: The ID of the PCI devices for my NIC were missing the decimal portion.
Proxmox tried to attach the device with the PCI-ID 0000:04:00, however, "lspci" shows that each port of my NIC has the ID 0000:04:00.0 through 0000:04:00.3

What I did was modify the VMs configuration file: /etc/pve/qemu-server/<vm-id>.conf:

hostpci0: 0000:04:00,pcie=1 changed to: hostpci0: 0000:04:00.0,pcie=1
hostpci1: 0000:04:00,pcie=1 changed to: hostpci1: 0000:04:00.1,pcie=1

When adding PCI devices from the Web UI the decimal points were not included, so Proxmox tried to attach a non-existent PCI-ID twice.

TL;DR:

• make sure that IOMMU is supported by your motherboard and enabled in the BIOS
• make sure your machine does not group the ports on your device into one IOMMU-group (unless you want to attach the whole device)

-> the feature to break up IOMMU groups is called "acs override" I believe and can be configured in grub

• make sure that your VMs config file (/etc/pve/qemu-server/<vm_id>.conf) has the correct PCI IDs for the devices you wish to pass through including decimal points

-> adding PCI devices from the Web UI does not add the decimal points for PCI device IDs as of PVE 7.0

Addendum:

Since I am trying to pass a NIC to a VM it seems that it is only possibly to pass the whole device to the VM.
in this case do not edit the config file since this will cause the VM to fail to bind the network ports to network devices.
Pass the whole device instead by choosing one port and enabeling "all functions". This will pass all the LAN ports to the VM.

 

[Aubs]

I found this thread as one of the pages at the beginning of my journey to try and overcome the whole host networking going down when configuring passthrough using the same video as the OP as a basis, but with OPNsense instead of pfSense (THIS article explains some of the differences between the two, one is pfSense has more plugins, but the other is that OPNsense base is more secure with cleaner code. pfSense is built on FreeBSD, while OPNsense is built on HardenedBSD, a fork of FreeBSD focused on security edit thanks @pengu1n because link).

I've documented everything I had to do for the NIC passthrough and splitting the IOMMU groups in the following SuperUser post, hope it is useful for anyone else who, like me, might come across this page looking for the answer but not finding it:

https://superuser.com/a/1790195/714988

 

[pengu1n]

OPN used to be based on HardenedBSD, not anymore. Doesn't matter though.