Pass through pcie card disconnects Proxmox

[ruffpl]

I have a problem that I can not fix by myself.

I had created VM (type- q35, network interface- e1000,bridge=vmbr0). I had activated intel_iommu/ vfio, vfio_iommu_type1, vfio_pci, vfio_virqfd, updated grub etc. so there was no problems with devices.
Everything works ok with VM but when I want to add pcie sas controller to it I am loosing connection to proxmox. I am using sas controller in IT mode in one slot and pcie DELL INTEL PRO 1000VT quad port NIC in second slot (internal ethernet card in mobo is disabled). For now it is connected only by one cable from dell nic to my swith but I need to do two things. First- I would like to have it working with sas controller. Second- I would like to have Proxmox menagement interface managed by all four ports connected together to switch (as link aggregation) to redoubled the speed (thats why I bought swith and quad card). Should I use this option like in this video ? Can I use all 4 ports for that or only 3 (or maybe use internal mobo eth card to install proxmox, after that set the whole controller to it as main interface and leave internal port as spare with different ip adress/ if it is even possible)? Can I get help how to do it correct and how to my make my VM working with pcie controller without loosing connection?

 

[leesteken]

Can you please show us your IOMMU groups (with for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done), so we can see if there are more devices in the same group as your SAS controller? Devices in the same group cannot be split between (different) VMs and the Proxmox host.
Maybe you can find the information about bonding in this part of the Proxmox forum?

 

[ruffpl]

I want to assign the sas controller with all disks to only one VM. My IOMMu groups look like this:

root@PROXMOX:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec2] (rev 07)
IOMMU group 10 00:1d.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port [8086:a298] (rev f0)
IOMMU group 11 00:1f.0 ISA bridge [0601]: Intel Corporation Z370 Chipset LPC/eSPI Controller [8086:a2c9]
IOMMU group 11 00:1f.2 Memory controller [0580]: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller [8086:a2a1]
IOMMU group 11 00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
IOMMU group 11 00:1f.4 SMBus [0c05]: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller [8086:a2a3]
IOMMU group 12 06:00.0 Non-Volatile memory controller [0108]: Device [1e0f:0001]
IOMMU group 13 08:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8822BE 802.11a/b/g/n/ac WiFi adapter [10ec:b822]
IOMMU group 14 09:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
IOMMU group 15 0a:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
IOMMU group 16 0b:00.0 Non-Volatile memory controller [0108]: Device [1e0f:0001]
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU group 1 00:01.1 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x8) [8086:1905] (rev 07)
IOMMU group 1 01:00.0 PCI bridge [0604]: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch [111d:8018] (rev 0e)
IOMMU group 1 02:02.0 PCI bridge [0604]: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch [111d:8018] (rev 0e)
IOMMU group 1 02:04.0 PCI bridge [0604]: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch [111d:8018] (rev 0e)
IOMMU group 1 03:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 1 03:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 1 04:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 1 04:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10e8] (rev 01)
IOMMU group 1 05:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation UHD Graphics 630 (Desktop) [8086:3e92]
IOMMU group 3 00:14.0 USB controller [0c03]: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller [8086:a2af]
IOMMU group 4 00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI [8086:a2ba]
IOMMU group 5 00:1b.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port [8086:a2e7] (rev f0)
IOMMU group 6 00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port [8086:a290] (rev f0)
IOMMU group 7 00:1c.2 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port [8086:a292] (rev f0)
IOMMU group 8 00:1c.4 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port [8086:a294] (rev f0)
IOMMU group 9 00:1c.6 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port [8086:a296] (rev f0)

 

[leesteken]

The SAS controller (05:00.0) is in group 1 together with your 4 Ethernet controllers (the PCI bridges don't matter). This explains why you lose the connection to Proxmox: they cannot be used by the host when the SAS controller is passed to the VM. (They can be used by that same VM, if you would want to.)
Try to move the SAS controller and/or the quad port NIC to other PCIe slots and see if that puts the SAS in a group alone. Or consider updating the BIOS version of your motherboard: sometime this changes the IOMMU groups, and sometimes it makes it worse,

 

[ruffpl]

The SAS controller (05:00.0) is in group 1 together with your 4 Ethernet controllers (the PCI bridges don't matter). This explains why you lose the connection to Proxmox: they cannot be used by the host when the SAS controller is passed to the VM. (They can be used by that same VM, if you would want to.)
Try to move the SAS controller and/or the quad port NIC to other PCIe slots and see if that puts the SAS in a group alone. Or consider updating the BIOS version of your motherboard: sometime this changes the IOMMU groups, and sometimes it makes it worse,

So (some other option)- is it possible to add sas controller and quad ethernet controller (both of them) into one VM and use the internal (mobo) as proxmox menagement interface?

 

[leesteken]

So (some other option)- is it possible to add sas controller and quad ethernet controller (both of them) into one VM and use the internal (mobo) as proxmox menagement interface?

Yes, if both devices support passthrough then that should work fine. First configure the Proxmox Network (or /etc/network/interface) to use the other single NIC and make sure it works. Then just try passing both devices to the same VM. It will probably work (unless it doesn't or does not support a restart of the VM, which happens sometimes).