Connection lost when using pci-passthrough

Keyinator

Member
Jan 29, 2022
26
0
6
22
Hello all,

I currently have a single nic on my proxmox host and am using two bridges as wan and lan for my pfsense vm where all traffic goes through.
Now I wanted to passthrough the nic into the vm in order to allow for better performance.

I followed the info at https://pve.proxmox.com/wiki/PCI_Passthrough

However, when I add the nic to pci-passthrough for my vm, I lose all connections. The kvm gets no info, also soft shutdown (hetzner calls it "pressing the power button" does not work anymore and I have to do a hard shutdown.

I have attached some log messages which might provide more insight. On my pfsense /var/log/system I could not find any entries at the corresponding times.
The "device not found" error from bash is not out of the ordinary since a script was trying to access the passthroughed nic's interface.
 

Attachments

  • 1680222746626.png
    1680222746626.png
    6.4 KB · Views: 11
  • 1680223781569.png
    1680223781569.png
    39 KB · Views: 10
  • 1680224008331.png
    1680224008331.png
    26.7 KB · Views: 10
Check your IOMMU groups: cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done. You cannot share devices from the same group between VMs or between a VM and the Proxmox host. PCI(e) devices in the same group can communicate with each other bypassing the IOMMU protection.
 
  • Like
Reactions: Keyinator
Check your IOMMU groups: cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done. You cannot share devices from the same group between VMs or between a VM and the Proxmox host. PCI(e) devices in the same group can communicate with each other bypassing the IOMMU protection.
Thank you very much! The nic was sharing an iommo group.
 
Check your IOMMU groups: cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done. You cannot share devices from the same group between VMs or between a VM and the Proxmox host. PCI(e) devices in the same group can communicate with each other bypassing the IOMMU protection.
I have tried to find an acs option in the bios but only found sr-iov support which I enabled.
(My hardware is of hetzner's ax51-nvme which uses an amd3700x processor)

I additionally enabled the following boot config: GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pcie_acs_override=downstream,multifunction" and applied it using sudo update-grub

However my output still looks like this (shared IOMMU group):
1680303460493.png

Any idea what else I could try?
 
Last edited:
Just use a separate virtual bridge with only that network port to the VM instead of PCIe passthrough. That will even be more secure than breaking the VM memory isolation with pcie_acs_override.
 
Just use a separate virtual bridge with only that network port to the VM instead of PCIe passthrough. That will even be more secure than breaking the VM memory isolation with pcie_acs_override.
That's what I am doing right now.
However, I have occasional ddos attacks and it puts load on both the vm and host (I have multiqueue enabled).

In the end it would save a good amount of resources for this case.
 
In the end it would save a good amount of resources for this case.
Only motherboards with an X570/X570S chipset can passthrough almost everything. I don't know about Threadripper and EPYC motherboards but every other Ryzen motherboard has the limitation that only PCIe lanes from the CPU can be passed through. Sometimes a BIOS update can help, since the groups are determined by the physical motherboard and BIOS (but sometimes BIOS updates have made things worse). Maybe you can add an additional network controller in the PCIe slot coming from the CPU? Maybe Hetzner can help you switch?
 
  • Like
Reactions: Keyinator
I just talked to a technician and the network card is onboard (should've known that).
Seems like it won't be possible then.

I am currently debating on getting a 10g connection for better (self-made) ddos protection.
Maybe it will work with the 10g one.
 
I just talked to a technician and the network card is onboard (should've known that).
Seems like it won't be possible then.

I am currently debating on getting a 10g connection for better (self-made) ddos protection.
Maybe it will work with the 10g one.
I didn't see any other network device in your screenshot. Indeed, that on-board one is in the same group as the ASPEED display and probably is also used for IPMI.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!