[SOLVED] VM do not start when passthrough more then one nic port of a quad card

[BurningSquirrel-de]

Hello,

I am sorry, I need help with an IOMMU problem.
I have a fresh installed Proxmox system on Server Hardware (Asus Z11PR-D16 Board, 2 socket Xeon Silver and i350t quad nic).

I installed the PCI passthrough like described in the Proxmox manual. What is noticeable that almost every device has his own iommu group, even every port on the intel quad nic, has his own iommu group.

Now I create a VM and passthrough one of the four ports as pci-e device to the VM. The VM starts without any problem.

When I add a second (or more) port to the vm, the start of the vm fails with this error:

kvm: -device vfio-pci,host=0000:af:00.1,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0: vfio 0000:af:00.1: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Transport endpoint is not connected

'find /sys/kernel/iommu_groups/ -type l' shows this output (excerpt):

/sys/kernel/iommu_groups/129/devices/0000:af:00.0 /sys/kernel/iommu_groups/130/devices/0000:af:00.1 /sys/kernel/iommu_groups/131/devices/0000:af:00.2 /sys/kernel/iommu_groups/132/devices/0000:af:00.3

There is no collision with other iommu groups.

Has someone an idea how to solve this problem?

Thank and regards,

Fabian

 

[leesteken]

Please show cat /proc/cmdline to make sure the system is not lying about the IOMMU group because you used pcie_acs_override.
There was a similar issue in this thread were two audio devices worked but not three. It looks like Proxmox or VFIO has problems with two devices that are "the same"?
An alternative (with none of the drawbacks of PCIe passthrough) would be to use multiple VFIO network devices on separate virtual bridges.

 

[BurningSquirrel-de]

Please show cat /proc/cmdline to make sure the system is not lying about the IOMMU group because you used pcie_acs_override.
There was a similar issue in this thread were two audio devices worked but not three. It looks like Proxmox or VFIO has problems with two devices that are "the same"?
An alternative (with none of the drawbacks of PCIe passthrough) would be to use multiple VFIO network devices on separate virtual bridges.

Hello,

cmdline is okay.
initrd=\EFI\proxmox\5.15.60-2-pve\initrd.img-5.15.60-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

My intention was to use the passthrough, because the vm should be a firewall.
There some "rumors" about security and performance issues, when using proxmox bridges. I am not at 100% in this topic. Do you think bridges are an alternative to passthrough?

Regards,

Fabian

 

[leesteken]

My intention was to use the passthrough, because the vm should be a firewall.

Have you tried passthrough of the whole device with all ports? Passthrough only the first device (af:00.0) and enable All Functions. (It will show as af:00 in the VM configuration file).

There some "rumors" about security and performance issues, when using proxmox bridges. I am not at 100% in this topic. Do you think bridges are an alternative to passthrough?

I'm not sure. I guess it depends on what kind of attack you want to defend against.

 

[BurningSquirrel-de]

Have you tried passthrough of the whole device with all ports? Passthrough only the first device (af:00.0) and enable All Functions. (It will show as af:00 in the VM configuration file).

This did not work A pity, it sounded like a solution in my first impression.

Meanwhile i also tried to make the host not use the card, by a file in modprobe.d, but it always throws the same error.

 

[Dunuin]

i350-T4 v1 works fine here. Make sure you not got a fake one. Big portion of all those i350-T4 out there are chinese fakes. And bridges+virtio NICs are fast enough for Gbit ethernet. Passthrough is more important if you want to use 10Gbit+ NICs.

When only passthrough of one port is working you could use that single passthroughed port for WAN and the other 3 using bridges and virtio. Then the 3 virtual NICs are on the safer LAN side behind the firewall so security wouldn't be that important. Also got another benefit. Guests on different networks but on the same PVE node could communicate with much more than 1Gbit (could for example be between 4-20Gbit). So internal communication would be much faster and throughput just drops to 1 Gbit when packets need to leave the PVE host.

 

[BurningSquirrel-de]

i350-T4 v1 works fine here. Make sure you not got a fake one. Big portion of all those i350-T4 out there are chinese fakes. And bridges+virtio NICs are fast enough for Gbit ethernet. Passthrough is more important if you want to use 10Gbit+ NICs.

F***, it seems to me that the card I bought is more on the fake side, than on the genuine side.

When only passthrough of one port is working you could use that single passthroughed port for WAN and the other 3 using bridges and virtio. Then the 3 virtual NICs are on the safer LAN side behind the firewall so security wouldn't be that important. Also got another benefit. Guests on different networks but on the same PVE node could communicate with much more than 1Gbit (could for example be between 4-20Gbit). So internal communication would be much faster and throughput just drops to 1 Gbit when packets need to leave the PVE host.

Okay, thanks for this. I save this in my mind as an option. Good idea!

 

[BurningSquirrel-de]

Hello,

I have solved my problem.

Unfortunately I have not a satisfactory solution, I swapped the card from a 8x PCIe slot to one of the 16x ones (PCI6 attached to the first CPU).

I have no clue why this worked.

Does anyone know if there is a difference between the x8 slots and the 16x slots? (apart from the obvious speed and fitting features)
Is there a difference between the initialization of the 16x and 8x slots?

Regards,

Fabian