[SOLVED] PCI (NIC) Passthrough

[Schar]

Hi, all who read this post
since two/three days I'm struggling with PCI Passthrough,
my Server has 4 Onboard NICs (Ethernet Cards) and I want to Pass two of them to my VM.

IOMMU is alredy enabled

dmesg | grep -e DMAR -e IOMMU
[    0.014209] ACPI: DMAR 0x000000007B7BC000 0000B8 (v01 LENOVO SYSTEM_X 00000001 MSFT 20091021)
[    0.640674] DMAR: IOMMU enabled
[    1.240298] DMAR: Host address width 46
[    1.240299] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.240306] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.240307] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.240311] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.240313] DMAR: RMRR base: 0x0000006895d000 end: 0x0000006895ffff
[    1.240314] DMAR: ATSR flags: 0x0
[    1.240318] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.240319] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.240320] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.240321] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.240322] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.241118] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    2.561840] DMAR: dmar1: Using Queued invalidation
[    2.609796] DMAR: Intel(R) Virtualization Technology for Directed I/O

and PCIs are in seperated groups, ( find /sys/kernel/iommu_groups/ -type l output attached)

lspci -nn | grep Ethernet Output

16:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
16:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
16:00.2 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
16:00.3 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)

i also added "[14e4:1657" to /etc/modprobe.d/kvm-intel.conf
which doesn't make a lot of sense since all the NICs have same Vendor-: Device ID, but wanted to test it (didn't work)

and at the end

lspci -nnk | grep Ethernet
16:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
        Subsystem: Lenovo NetXtreme BCM5719 Gigabit Ethernet PCIe [17aa:400e]
16:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
        Subsystem: Lenovo NetXtreme BCM5719 Gigabit Ethernet PCIe [17aa:400e]
16:00.2 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
        Subsystem: Lenovo NetXtreme BCM5719 Gigabit Ethernet PCIe [17aa:400e]
16:00.3 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
        Subsystem: Lenovo NetXtreme BCM5719 Gigabit Ethernet PCIe [17aa:400e]

any idea what I'm doing wrong?

I'm fairly new with Linux, so please give more hint when you need any command output or any action from my side,

Thank you all

Regards

Schar

 

[Schar]

just find out that the Ethernets are all in the same group,

find /sys/kernel/iommu_groups/ -type l | grep 16:00

/sys/kernel/iommu_groups/19/devices/0000:16:00.3
/sys/kernel/iommu_groups/19/devices/0000:16:00.1
/sys/kernel/iommu_groups/19/devices/0000:16:00.2
/sys/kernel/iommu_groups/19/devices/0000:16:00.0

is there anyway i could regroup them in seprate groups?

 

[janssensm]

Hi, I do use passthrough myself but do not have much experience with it, so bare with me.
As said here [1] things can be tricky when not separated completely:

You have onboard nics, so you cannot switch them to a different slot, so that limits options within the onboard part.
And I

A possible solution.
As stated in the wiki [2], you could try to use Unsafe interrupts. Passthrough remains a bit risky, this setting could solve the separation, but can be worse for stability. So if that solves the separation, you could try and run the system for a while, put some heavy load on the nics and vm's and host and see if it keeps running fine.

Another possible solution:
Buy 2 extra nics and put them in free pci slots, and make sure they are in separate groups.
Then you can pass through those 2 extra nics.
Or you can passthrough the complete set of onboard nics as 0000:16:00 (then you cannot use any of them on the host, but all in the vm)

Can you describe what error you get or what part exactly doesn't work with you current setup?
Does the vm start? Doesn't the vm detect the cards?

[1] https://forum.proxmox.com/threads/pcie-passthrough-atheros-wifi.73204/post-327360
[2] https://pve.proxmox.com/wiki/PCI(e)_Passthrough

 

[Schar]

Hi, I do use passthrough myself but do not have much experience with it, so bare with me.
As said here [1] things can be tricky when not separated completely:

You have onboard nics, so you cannot switch them to a different slot, so that limits options within the onboard part.
And I

A possible solution.
As stated in the wiki [2], you could try to use Unsafe interrupts. Passthrough remains a bit risky, this setting could solve the separation, but can be worse for stability. So if that solves the separation, you could try and run the system for a while, put some heavy load on the nics and vm's and host and see if it keeps running fine.

Another possible solution:
Buy 2 extra nics and put them in free pci slots, and make sure they are in separate groups.
Then you can pass through those 2 extra nics.
Or you can passthrough the complete set of onboard nics as 0000:16:00 (then you cannot use any of them on the host, but all in the vm)

Can you describe what error you get or what part exactly doesn't work with you current setup?
Does the vm start? Doesn't the vm detect the cards?

[1] https://forum.proxmox.com/threads/pcie-passthrough-atheros-wifi.73204/post-327360
[2] https://pve.proxmox.com/wiki/PCI(e)_Passthrough

with the current Setup, whenever I start the VM( click on the start button) the whole server crash, with no sign, I see no changes on the proxmox Prompt, but the VM is not pingable anymore and Web-UI get unreachable, but when I connect a monitor the server I could see the Proxmox logging prompt (unfortunatelyI got now keyboard to do anything on the physical server and SSH not possible anymore,
The only option is to restart the whole server,

My question regarding your suggestion with passing all Ethernets (4NICs) to the VM is,

Then how could I reach my proxmox when there is no Interface for Proxmox anymore, and how could proxmox reach to network/internet?


Update: just added the Unsafe interrupts, same reauslt, unfortunatly the unsafe interrupts didnt works for me,

 

[janssensm]

Then how could I reach my proxmox when there is no Interface for Proxmox anymore, and how could did proxmox reach to network/internet?

That's only possible if you had extra nics installed already, your host will indeed be crippled if you would do that with your current setup.
But this configuration crashes your server, which is bad and probably has all to do with passing through a device that isn't completely isolated into it's own group.
With current hardware you could still try with the unsafe setting, bet that's a gamble.

So my best advice I can give, also for stability is to buy extra nics and put them in pci slots.
Still no guarantee that they will be isolated, but probably they are.

 

[Schar]

That's only possible if you had extra nics installed already, your host will indeed be crippled if you would do that with your current setup.
But this configuration crashes your server, which is bad and probably has all to do with passing through a device that isn't completely isolated into it's own group.
With current hardware you could still try with the unsafe setting, bet that's a gamble.

So my best advice I can give, also for stability is to buy extra nics and put them in pci slots.
Still no guarantee that they will be isolated, but probably they are.

I do have a spare Card which has tow Ports, will give it a shot, hope that solves my problem,

Thank you for your time and ipnut

ill mark this threat as solved, since apperently there is no way to regroup the onboard cards in seprated iommu_groups

 

[Sebastian Köllmayr]

Hei

Just passtrough the the PCI-e that you want to use, feks: 16:00.0 and 16:00.1.
That how i do it on my server with 4 integrated NICs tha are passed trough to different VM's.
Be aware that some OS's need the machine type set to "machine: pc-q35-3.1" to work ->that you have to do manuelly.

Regards
Sebastian

 

[Schar]

Hei

Just passtrough the the PCI-e that you want to use, feks: 16:00.0 and 16:00.1.
That how i do it on my server with 4 integrated NICs tha are passed trough to different VM's.
Be aware that some OS's need the machine type set to "machine: pc-q35-3.1" to work ->that you have to do manuelly.

Regards
Sebastian

Hi Sebastian,

thanks for your reply.

i tried that, unfortunately as soon as I started the VM the proxmox lost ip connectivity.
till now I tied every time just passing through the 16:00.3, and Proxmox use 16:00.0 for his IP connectivity.

have you face such problem, or do you believe that the trick is “machine: pc-q35-3.1"

thanks
Schar

 

[Sebastian Köllmayr]

Hi

I could be different things.
I need the machine type pc-q35-3.1 for typeBSD to passthrough the nics.

I can although remember that i needed to find out what MAC-address is what NIC.
Thats because the naming of the NICs and howthere where listed in lspci didnt match.

Do you have the 4 NICs in different iommu_groups when you look at the output of
# find /sys/kernel/iommu_groups/ -type l

Regards
Sebastian

 

[Schar]

Hi

I could be different things.
I need the machine type pc-q35-3.1 for typeBSD to passthrough the nics.

I can although remember that i needed to find out what MAC-address is what NIC.
Thats because the naming of the NICs and howthere where listed in lspci didnt match.

Do you have the 4 NICs in different iommu_groups when you look at the output of
# find /sys/kernel/iommu_groups/ -type l

Regards
Sebastian

No, they are all in the same IOMMU-Groups (16:00.x)

 

[Sebastian Köllmayr]

You could although try
adding "pcie_acs_override=downstream" to kernel boot commandline
https://pve.proxmox.com/wiki/Pci_passthrough

 

[Sebastian Köllmayr]

Hi

i remembered more.
It can be how the 4 NICs are connected ->directly to the processor PCIE lanes or through the PCH.

maybe you could try SR-IOV
https://pve.proxmox.com/wiki/PCI(e)_Passthrough