Partial ethernet passthrough

[Lucas Rey]

Hello community, I have an Intel X710-T4 card used mainly with OPNSense. I setup the OPNSense machine with VirtIO Paravirtualized network devices, so that the VM has 3 interfaces: GREEN, BLU and RED. The other port (PCI 01:00.0) is used for Proxmox management.

01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710/X557-AT 10GBASE-T (rev 02)
        Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-T4
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710/X557-AT 10GBASE-T (rev 02)
        Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-T
01:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710/X557-AT 10GBASE-T (rev 02)
        Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-T
01:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710/X557-AT 10GBASE-T (rev 02)
        Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-T

What I would like to do is passthrough only specific PCI address, e.g. RED and GREEN to try to increase performance.
I tried starting with RED interface (PCI 01:00.1) but as soon I started the VM I completely lost every connectivity, even the management.



Do you know please if is possible to passthrough only a specific PCI address? Or it will be passed the whole card?
Thank you
Lucas

EDIT: Obviously IOMMU is correctly configured on Proxmox as I already used it to passthrough GPU to other VM

 

[leesteken]

Did you check (or can you show) the IOMMU groups of your system: https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation ?

 

[Phips]

Can you check on the command line of your Opnsense VM which network adapters are available to the virtual system?

Best regards,
Phips

 

[Lucas Rey]

Did you check (or can you show) the IOMMU groups of your system: https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation ?

Yes, edited the main post, IOMMU is correctly configured on Proxmox as I already used it to passthrough GPU to other VM

Can you check on the command line of your Opnsense VM which network adapters are available to the virtual system?

The problem is that I cannot reach anything anymore, and I have to reboot Proxmox to restore the connectivity (luckily I can access console so that I manually removed the PCI 01:00.0 from OPNSense config). I don't think anyway the problem is OPNSense cause I lost every connectivity not only the RED one. I'm planning to move the managment to internal network card so that at least I'll be able to reach Proxmox GUI.

But the main question is always the same, is it possibile to passthrough only 2 eth ports from the same card leaving the other to the host?
I'm searching but so far, I cannot find anything related.

 

[leesteken]

The problem is that I cannot reach anything anymore, and I have to reboot Proxmox to restore the connectivity (luckily I can access console so that I manually removed the PCI 01:00.0 from OPNSense config).

Still sound like the four functions are not in separate IOMMU groups, or you are using pcie_acs_override for break the groups but the hardware does not actually support passthrough of separate functions. Please show the IOMMU groups (see the Wiki link I posted) and the output of cat /proc/cmdline.

 

[Lucas Rey]

Still sound like the four functions are not in separate IOMMU groups, or you are using pcie_acs_override for break the groups but the hardware does not actually support passthrough of separate functions. Please show the IOMMU groups (see the Wiki link I posted) and the output of cat /proc/cmdline.

My bad I didn't understood correctly your previous post:

This is the cmdline output:

# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-4-pve root=/dev/mapper/pve-root ro net.ifnames=0 biosdevnames=0 quiet intel_iommu=on

And this is the complete list of IOMMU groups:

┌──────────┬────────┬──────────────┬────────────┬────────┬──────────────────────────────────────────────────┬──────┬──────────────────┬─────────────────────────────────────────
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                      │ mdev │ subsystem_device │ subsystem_device_name                   
╞══════════╪════════╪══════════════╪════════════╪════════╪══════════════════════════════════════════════════╪══════╪══════════════════╪═════════════════════════════════════════
│ 0x010601 │ 0x7ae2 │ 0000:00:17.0 │          7 │ 0x8086 │ Alder Lake-S PCH SATA Controller [AHCI Mode]     │      │ 0x7ae2           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x010802 │ 0xa80c │ 0000:02:00.0 │         16 │ 0x144d │                                                  │      │ 0xa801           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x010802 │ 0x5415 │ 0000:03:00.0 │         17 │ 0xc0a9 │                                                  │      │ 0x2b00           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x010802 │ 0x540a │ 0000:05:00.0 │         19 │ 0xc0a9 │ P2 NVMe PCIe SSD                                 │      │ 0x5021           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x020000 │ 0x1a1d │ 0000:00:1f.6 │         11 │ 0x8086 │ Ethernet Connection (17) I219-V                  │      │ 0x1a1d           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x020000 │ 0x1589 │ 0000:01:00.0 │         12 │ 0x8086 │ Ethernet Controller X710/X557-AT 10GBASE-T       │      │ 0x0001           │ Ethernet Converged Network Adapter X710-
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x020000 │ 0x1589 │ 0000:01:00.1 │         13 │ 0x8086 │ Ethernet Controller X710/X557-AT 10GBASE-T       │      │ 0x0000           │ Ethernet Converged Network Adapter X710-
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x020000 │ 0x1589 │ 0000:01:00.2 │         14 │ 0x8086 │ Ethernet Controller X710/X557-AT 10GBASE-T       │      │ 0x0000           │ Ethernet Converged Network Adapter X710-
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x020000 │ 0x1589 │ 0000:01:00.3 │         15 │ 0x8086 │ Ethernet Controller X710/X557-AT 10GBASE-T       │      │ 0x0000           │ Ethernet Converged Network Adapter X710-
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x020000 │ 0x8125 │ 0000:04:00.0 │         18 │ 0x10ec │ RTL8125 2.5GbE Controller                        │      │ 0x8125           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x030000 │ 0xa780 │ 0000:00:02.0 │          0 │ 0x8086 │ Raptor Lake-S GT1 [UHD Graphics 770]             │      │ 0xa780           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x040300 │ 0x7ad0 │ 0000:00:1f.3 │         11 │ 0x8086 │ Alder Lake-S HD Audio Controller                 │      │ 0x222a           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x050000 │ 0x7aa7 │ 0000:00:14.2 │          4 │ 0x8086 │ Alder Lake-S PCH Shared SRAM                     │      │ 0x0000           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x060000 │ 0xa700 │ 0000:00:00.0 │          1 │ 0x8086 │                                                  │      │ 0xa700           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x060100 │ 0x7a84 │ 0000:00:1f.0 │         11 │ 0x8086 │ Z690 Chipset LPC/eSPI Controller                 │      │ 0x7a84           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x060400 │ 0xa70d │ 0000:00:01.0 │          2 │ 0x8086 │                                                  │      │ 0xa70d           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x060400 │ 0xa74d │ 0000:00:06.0 │          3 │ 0x8086 │                                                  │      │ 0x0000           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x060400 │ 0x7ac8 │ 0000:00:1a.0 │          8 │ 0x8086 │ Alder Lake-S PCH PCI Express Root Port #25       │      │ 0x7ac8           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x060400 │ 0x7ab9 │ 0000:00:1c.0 │          9 │ 0x8086 │ Alder Lake-S PCH PCI Express Root Port #2        │      │ 0x7ab9           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x060400 │ 0x7abc │ 0000:00:1c.4 │         10 │ 0x8086 │ Alder Lake-S PCH PCI Express Root Port #5        │      │ 0x7abc           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x078000 │ 0x7ae8 │ 0000:00:16.0 │          6 │ 0x8086 │ Alder Lake-S PCH HECI Controller #1              │      │ 0x7ae8           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x0c0330 │ 0x7ae0 │ 0000:00:14.0 │          4 │ 0x8086 │ Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller │      │ 0x7ae0           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x0c0500 │ 0x7aa3 │ 0000:00:1f.4 │         11 │ 0x8086 │ Alder Lake-S PCH SMBus Controller                │      │ 0x7aa3           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x0c8000 │ 0x7acc │ 0000:00:15.0 │          5 │ 0x8086 │ Alder Lake-S PCH Serial IO I2C Controller #0     │      │ 0x7acc           │                                         
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────┼──────┼──────────────────┼─────────────────────────────────────────
│ 0x0c8000 │ 0x7aa4 │ 0000:00:1f.5 │         11 │ 0x8086 │ Alder Lake-S PCH SPI Controller                  │      │ 0x7aa4           │                                         
└──────────┴────────┴──────────────┴────────────┴────────┴──────────────────────────────────────────────────┴──────┴──────────────────┴─────────────────────────────────────────