Device assigned more than once? It's weird error.

[QCodec]

I'm building a new proxmox machine with some used and new part. (AMD Ryzen 7 pro 2700 etc well it isn't important.)
I installed proxmox 8.2.2 without a glitch. Now I'm trying to install pfsense with intel i350 4 port lan card.
I know I can't assign each port more than once. But it doesn't show correct value for each port? This seems not right.
All show same IOMMU group number. (unlike my other i225 NIC)

0000:03:00.0
0000:03:00.1
0000:03:00.2
0000:03:00.3

And I can't assign them. After few times, it caused EXT4-fs error. I can't reboot. Detected aborted journal then remounting filesystem read-only.
Faulty NIC? Wrong iommu setting? What could be the problem?

 

[Moayad]

Hi,

Can you check if the IOMMU is enabled in BIOS and that Proxmox VE boot option include `intel_iommu=on` or `amd_iommu=on`?

 

[QCodec]

Hi,

Can you check if the IOMMU is enabled in BIOS and that Proxmox VE boot option include `intel_iommu=on` or `amd_iommu=on`?

Yeap, it's on.

root@pve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.000000] AMD-Vi: Using global IVHD EFR:0xf77ef22294ada, EFR2:0x0
[ 0.330724] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.335134] AMD-Vi: Extended features (0xf77ef22294ada, 0x0): PPR NX GT IA GA PC GA_vAPIC
[ 0.335150] AMD-Vi: Interrupt remapping enabled
[ 0.335285] AMD-Vi: Virtual APIC enabled
[ 0.335382] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

But I will try "amd_iommu=on". (10 min later) It's same. It doesn't work.
One of the posts from March shows the same issue on an Intel i350 NIC. It should be a coincidence.

https://forum.proxmox.com/threads/multi-port-nic-passthrough-one-nic-per-vm.119637/post-642918

 

[leesteken]

amd_iommu=on is nonsense as it is enabled by default (and on is therefore not a valid value: https://www.kernel.org/doc/html/v6.8/admin-guide/kernel-parameters.html )
Furthermore, dmesg | grep -e DMAR -e IOMMU -e AMD-Vi is not a very good test to check if IOMMU is on for AMD systems. Simply look if you have multiple IOMMU groups in the Proxmox web GUI or dmesg/journalctl or using the command line: https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation . You can then also immediately check whether your device is alone in a group.

 

[QCodec]

They're in same group no.12.

0000:04:00.0
0000:04:00.1
0000:04:00.2
0000:04:00.3

"pcie_acs_override=downstream" doesn't work.
Now I'm trying "amd_iommu=forced_isolation". And it doesn't work either.

 

[leesteken]

They're in same group no.12.

There are also other devices in group 12, which will be removed from the host when you pass any device from that group to a VM.

0000:04:00.0
0000:04:00.1
0000:04:00.2
0000:04:00.3

"pcie_acs_override=downstream" doesn't work.

Those are four functions of one PCI(e) device. Try pcie_acs_override=downstream,multifunction instead. Please be aware of the security risk since you are breaking IOMMU isolation.

What motherboard/chipset are you using? Most AMD Ryzen chipsets put everything in one big group exception for one or two PCIe slots and one M.2 slot (all connected directly to the CPU) .

 

[QCodec]

This is Lenovo m725s with latest firmware 2023. I will try "pcie_acs_override=downstream,multifunction" now.

And it doesn't show any change.

 

[leesteken]

It's Lenovo m725s.

I can't easily find which chipset is used, but it's AM4 and unless it uses X570(S) it's very limited in (secure) PCI(e) passthrough. The better work-around would be to the right PCIe slot or use a M.2 to PCIe x4 converter.

I will try "pcie_acs_override=downstream,multifunction" now.

I recommend against it and would suggest using multiple virtual bridges instead, otherwise the VM can read all of your Proxmox host (and other VM's) memory. But it's up to you of course.

 

[QCodec]

I can't easily find which chipset is used, but it's AM4 and unless it uses X570(S) it's very limited in (secure) PCI(e) passthrough. The better work-around would be to the right PCIe slot or use a M.2 to PCIe x4 converter.

I recommend against it and would suggest using multiple virtual bridges instead, otherwise the VM can read all of your Proxmox host (and other VM's) memory. But it's up to you of course.

It's B450 chipset with AMD Ryzen 7 pro 2700. Two of x1 pcie 2.0 and one pci 3 slots etc etc. Tried other slot too but no go.
As I said early "pcie_acs_override=downstream,multifunction" doesn't work either. M.2 to PCIe x4 converter would be the last option, but I give up.

Now I'm hungry and sleepy.

 

[leesteken]

It's B450 chipset with AMD Ryzen 7 pro 2700. Two of x1 pcie 2.0 and one pci 3 slots etc etc. Tried other slot too but no go.

Strange, B450 ought to put the x16 slot into a separate group, but every motherboard might be different. If you are using a CPU with integrated graphics, things might also be different.

As I said early "pcie_acs_override=downstream,multifunction" doesn't work either.

I did not see that in this thread, sorry. Does it do anything? It should at least have some effect (on the big chipset group). Check with cat /proc/cmdline whether its present.

EDIT: Sometimes the motherboard puts the M.2 behind a PCIe multiplexer and blocks passthrough, so that might also not work. Is there a PCIe tree layout/schematic known for your motherboard?