Proxmox7 IOMMU

[ruffpl]

Hi. I was using and learning Proxmox 6 for some time. Now I had reformat my server and made a fresh install of Proxmox 7. Before upgrade system was on 2 drives- RAIDZ1 and I had no problems after I turn on IOMMU (grub, added vfio, etc). I had one VM especially for sas controller and quad 1GbE card to it. Now in Proxmox 7 I put system on BTRFS RAID1 array. I had turn on IOMMU fallowed this tutorial (everything until gpu blacklist)

https://www.youtube.com/watch?v=5ce-CcYjqe8&t=1s

Now some groups are changed compared to Proxmox 6. When I put command:

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

In PVE7 it looks like this:

IOMMU group 18 03:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 19 03:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU group 20 04:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 21 04:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 22 05:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)

And in PVE6 it was:

IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU group 1 00:01.1 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x8) [8086:1905] (rev 07)
IOMMU group 1 00:01.2 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x4) [8086:1909] (rev 07)
IOMMU group 1 01:00.0 PCI bridge [0604]: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch [111d:8018] (rev 0e)
IOMMU group 1 02:02.0 PCI bridge [0604]: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch [111d:8018] (rev 0e)
IOMMU group 1 02:04.0 PCI bridge [0604]: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch [111d:8018] (rev 0e)
IOMMU group 1 03:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 1 03:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 1 04:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 1 04:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 1 05:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)


Is there some way to arrange it so devices 18, 19, 20, 21 and 22 was as one group like it was in PVE6 group 1?

 

[leesteken]

Why would you want to do that? More IOMMU groups (without using pcie_acs_override) is better and gives more flexibility, as one cannot share devices from the same group between VMs and/or the Proxmox host. Having more groups should not cause problems, and there are lots of posts on this forum with troubles with too few groups (with too many devices in them). Maybe the newer kernel of Proxmox 7 just has better support for your motherboard.
Just pass the SAS controller and Quad network controller to the VM; you pass them by their PCI ID (03:00, 04:00 and 05:00, with All Functions enabled). It should be no problem that they are in multiple groups. Or does the person in that video use the pcie_acs_override? Then just remove that to get the actual IOMMU groups.

 

[ruffpl]

Maybe the newer kernel of Proxmox 7 just has better support for your motherboard.

Actually I had both pcie devices in appointed slots (and one group) because in PVE6 I had problems with them. Whenever quad GbE card was sellected as main internet connection in Proxmox adding sas card to VM was causing problems (loosing connection. some hdds not visible, etc) Finally I ended with onboard GbE slot as main Proxmox connection and both cards added into on VM ( then there was no conflicts).
Now actually when it is finally possible I can use quad GbE card as main bond connection for Proxmox but I dont know exactly how to do it so I will ask in new post.