Can you pass onboard NICs through to a VM?

V

victorhooi

Well-Known Member
Apr 3, 2018
250
20
58
38
Hi,

I am trying to setup ntopng Edge on Proxmox 5.2, on a Dell RX720XD (with four onboard Gigabit NICs).

The developers of ntopng Edge have said it is better to do PCI passthrough, rather than setup a bridge:

the problem is that when using virtual adapters you in essence use a switch and thus we cannot implement the bump in the wire as the collision domain will prevent it from working, So yes, you need PCIe passthrough
Click to expand...

Does anybody know if it's possible to pass onboard NICs like this through to a KVM-based VM?

If so, what is the process?

(I saw https://pve.proxmox.com/wiki/Pci_passthrough, but it seems very focused on GPUs, and I'm not sure if there's caveats or differences when using onboard NICs like this).

Thanks,
Victor
 
Ok, so I've followed the instructions at https://pve.proxmox.com/wiki/Pci_passthrough - specifically editing /etc/default/grub, and /etc/modules.

After rebooting, I see:

Code: 
root@gcc-proxmox:~# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x00000000BD3346F4 000158 (v01 DELLOE DELLOSE  00000001 DELL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.000000] DMAR: Host address width 46
[    0.000000] DMAR: DRHD base: 0x000000d5000000 flags: 0x0
[    0.000000] DMAR: dmar0: reg_base_addr d5000000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.000000] DMAR: DRHD base: 0x000000df900000 flags: 0x1
[    0.000000] DMAR: dmar1: reg_base_addr df900000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.000000] DMAR: RMRR base: 0x000000bf458000 end: 0x000000bf46ffff
[    0.000000] DMAR: RMRR base: 0x000000bf450000 end: 0x000000bf450fff
[    0.000000] DMAR: RMRR base: 0x000000bf452000 end: 0x000000bf452fff
[    0.000000] DMAR: ATSR flags: 0x0
[    0.000000] DMAR-IR: IOAPIC id 2 under DRHD base  0xd5000000 IOMMU 0
[    0.000000] DMAR-IR: IOAPIC id 0 under DRHD base  0xdf900000 IOMMU 1
[    0.000000] DMAR-IR: IOAPIC id 1 under DRHD base  0xdf900000 IOMMU 1
[    0.000000] DMAR-IR: HPET id 0 under DRHD base 0xdf900000
[    0.000000] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.000000] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.913262] DMAR: dmar0: Using Queued invalidation
[    0.913271] DMAR: dmar1: Using Queued invalidation
[    0.913289] DMAR: Setting RMRR:
[    0.913808] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf452000 - 0xbf452fff]
[    0.914449] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf450000 - 0xbf450fff]
[    0.914466] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf458000 - 0xbf46ffff]
[    0.914468] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf458000 - 0xbf46ffff]
[    0.914471] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.915105] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.915121] DMAR: Intel(R) Virtualization Technology for Directed I/O

and:

Code: 
root@gcc-proxmox:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:03:00.0
/sys/kernel/iommu_groups/45/devices/0000:7f:0f.6
/sys/kernel/iommu_groups/45/devices/0000:7f:0f.4
/sys/kernel/iommu_groups/45/devices/0000:7f:0f.2
/sys/kernel/iommu_groups/45/devices/0000:7f:0f.0
/sys/kernel/iommu_groups/45/devices/0000:7f:0f.5
/sys/kernel/iommu_groups/45/devices/0000:7f:0f.3
/sys/kernel/iommu_groups/45/devices/0000:7f:0f.1
/sys/kernel/iommu_groups/35/devices/0000:40:03.2
/sys/kernel/iommu_groups/7/devices/0000:00:11.0
/sys/kernel/iommu_groups/25/devices/0000:3f:0c.2
/sys/kernel/iommu_groups/25/devices/0000:3f:0c.0
/sys/kernel/iommu_groups/25/devices/0000:3f:0c.7
/sys/kernel/iommu_groups/25/devices/0000:3f:0c.1
/sys/kernel/iommu_groups/25/devices/0000:3f:0c.6
/sys/kernel/iommu_groups/15/devices/0000:01:00.0
/sys/kernel/iommu_groups/15/devices/0000:01:00.1
/sys/kernel/iommu_groups/43/devices/0000:7f:0d.0
/sys/kernel/iommu_groups/43/devices/0000:7f:0d.1
/sys/kernel/iommu_groups/43/devices/0000:7f:0d.6
/sys/kernel/iommu_groups/43/devices/0000:7f:0d.2
/sys/kernel/iommu_groups/33/devices/0000:40:02.0
/sys/kernel/iommu_groups/5/devices/0000:00:03.0
/sys/kernel/iommu_groups/23/devices/0000:3f:0a.3
/sys/kernel/iommu_groups/23/devices/0000:3f:0a.1
/sys/kernel/iommu_groups/23/devices/0000:3f:0a.2
/sys/kernel/iommu_groups/23/devices/0000:3f:0a.0
/sys/kernel/iommu_groups/13/devices/0000:00:1e.0
/sys/kernel/iommu_groups/41/devices/0000:7f:0b.3
/sys/kernel/iommu_groups/41/devices/0000:7f:0b.0
/sys/kernel/iommu_groups/31/devices/0000:3f:13.0
/sys/kernel/iommu_groups/31/devices/0000:3f:13.5
/sys/kernel/iommu_groups/31/devices/0000:3f:13.1
/sys/kernel/iommu_groups/31/devices/0000:3f:13.6
/sys/kernel/iommu_groups/31/devices/0000:3f:13.4
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:3f:08.0
/sys/kernel/iommu_groups/11/devices/0000:00:1c.7
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/48/devices/0000:7f:13.4
/sys/kernel/iommu_groups/48/devices/0000:7f:13.0
/sys/kernel/iommu_groups/48/devices/0000:7f:13.5
/sys/kernel/iommu_groups/48/devices/0000:7f:13.1
/sys/kernel/iommu_groups/48/devices/0000:7f:13.6
/sys/kernel/iommu_groups/38/devices/0000:7f:08.0
/sys/kernel/iommu_groups/28/devices/0000:3f:0f.6
/sys/kernel/iommu_groups/28/devices/0000:3f:0f.4
/sys/kernel/iommu_groups/28/devices/0000:3f:0f.2
/sys/kernel/iommu_groups/28/devices/0000:3f:0f.0
/sys/kernel/iommu_groups/28/devices/0000:3f:0f.5
/sys/kernel/iommu_groups/28/devices/0000:3f:0f.3
/sys/kernel/iommu_groups/28/devices/0000:3f:0f.1
/sys/kernel/iommu_groups/18/devices/0000:08:00.0
/sys/kernel/iommu_groups/46/devices/0000:7f:10.2
/sys/kernel/iommu_groups/46/devices/0000:7f:10.0
/sys/kernel/iommu_groups/46/devices/0000:7f:10.7
/sys/kernel/iommu_groups/46/devices/0000:7f:10.5
/sys/kernel/iommu_groups/46/devices/0000:7f:10.3
/sys/kernel/iommu_groups/46/devices/0000:7f:10.1
/sys/kernel/iommu_groups/46/devices/0000:7f:10.6
/sys/kernel/iommu_groups/46/devices/0000:7f:10.4
/sys/kernel/iommu_groups/36/devices/0000:40:05.0
/sys/kernel/iommu_groups/36/devices/0000:40:05.2
/sys/kernel/iommu_groups/8/devices/0000:00:16.0
/sys/kernel/iommu_groups/8/devices/0000:00:16.1
/sys/kernel/iommu_groups/26/devices/0000:3f:0d.1
/sys/kernel/iommu_groups/26/devices/0000:3f:0d.6
/sys/kernel/iommu_groups/26/devices/0000:3f:0d.2
/sys/kernel/iommu_groups/26/devices/0000:3f:0d.0
/sys/kernel/iommu_groups/16/devices/0000:02:00.0
/sys/kernel/iommu_groups/16/devices/0000:02:00.1
/sys/kernel/iommu_groups/44/devices/0000:7f:0e.1
/sys/kernel/iommu_groups/44/devices/0000:7f:0e.0
/sys/kernel/iommu_groups/34/devices/0000:40:03.0
/sys/kernel/iommu_groups/6/devices/0000:00:05.2
/sys/kernel/iommu_groups/6/devices/0000:00:05.0
/sys/kernel/iommu_groups/24/devices/0000:3f:0b.0
/sys/kernel/iommu_groups/24/devices/0000:3f:0b.3
/sys/kernel/iommu_groups/14/devices/0000:00:1f.0
/sys/kernel/iommu_groups/42/devices/0000:7f:0c.6
/sys/kernel/iommu_groups/42/devices/0000:7f:0c.2
/sys/kernel/iommu_groups/42/devices/0000:7f:0c.0
/sys/kernel/iommu_groups/42/devices/0000:7f:0c.7
/sys/kernel/iommu_groups/42/devices/0000:7f:0c.1
/sys/kernel/iommu_groups/32/devices/0000:40:01.0
/sys/kernel/iommu_groups/4/devices/0000:00:02.2
/sys/kernel/iommu_groups/22/devices/0000:3f:09.0
/sys/kernel/iommu_groups/12/devices/0000:00:1d.0
/sys/kernel/iommu_groups/40/devices/0000:7f:0a.0
/sys/kernel/iommu_groups/40/devices/0000:7f:0a.3
/sys/kernel/iommu_groups/40/devices/0000:7f:0a.1
/sys/kernel/iommu_groups/40/devices/0000:7f:0a.2
/sys/kernel/iommu_groups/30/devices/0000:3f:11.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/20/devices/0000:09:01.0
/sys/kernel/iommu_groups/10/devices/0000:00:1c.0
/sys/kernel/iommu_groups/39/devices/0000:7f:09.0
/sys/kernel/iommu_groups/29/devices/0000:3f:10.0
/sys/kernel/iommu_groups/29/devices/0000:3f:10.7
/sys/kernel/iommu_groups/29/devices/0000:3f:10.5
/sys/kernel/iommu_groups/29/devices/0000:3f:10.3
/sys/kernel/iommu_groups/29/devices/0000:3f:10.1
/sys/kernel/iommu_groups/29/devices/0000:3f:10.6
/sys/kernel/iommu_groups/29/devices/0000:3f:10.4
/sys/kernel/iommu_groups/29/devices/0000:3f:10.2
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:09:00.0
/sys/kernel/iommu_groups/19/devices/0000:0b:00.0
/sys/kernel/iommu_groups/19/devices/0000:0a:00.0
/sys/kernel/iommu_groups/47/devices/0000:7f:11.0
/sys/kernel/iommu_groups/37/devices/0000:41:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1a.0
/sys/kernel/iommu_groups/27/devices/0000:3f:0e.1
/sys/kernel/iommu_groups/27/devices/0000:3f:0e.0

So it seems like my motherboard (Dell R720XD) does support PCI passthrough.

If I check output of lscpi

Code: 
root@gcc-proxmox:~# lspci
...
01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
...
41:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

It should be simply enough to pass those directly through to the VM by hand-editing "node.conf", right?

I suppose there's no friendly command, that also sanitises/checks my input? (Like there is when passing through disks).
 
/sys/kernel/iommu_groups/15/devices/0000:01:00.0
/sys/kernel/iommu_groups/15/devices/0000:01:00.1

/sys/kernel/iommu_groups/16/devices/0000:02:00.0
/sys/kernel/iommu_groups/16/devices/0000:02:00.1

you can't just pass a single port to your vm but you need to pass 2 port due to they are in the same iommu group
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back