[TUTORIAL] SR-IOV in AMD motherboard for two intel NICs: 1Gbps (E1G42ET - Intel 82576) 10 Gbps (X520-DA2 - Intel 82599ES) in Proxmox 6.3-4

daisychain · Feb 28, 2021

Hi all,
Below you have the steps to get SR-IOV working in a Gigabyte GA-AB350M-Gaming 3 (Rev. 1.x) motherboard. This is very much based on the tutorial by "Enabling SR-IOV for Intel NIC (X550-T2) on Proxmox 6" by @Sandbo Thanks.

Motherboard Gigabyte GA-AB350M-Gaming 3 (Rev. 1.x). BIOS version F50d
CPU AMD Ryzen 5 1500X
Three PCI expansion slots in the motherboard:
- 1 x slot PCI Express x16 running x16 (PCIEX16). Populated with a NVIDIA GTX1060 graphics card. This follows PCI Express 3.0 standard.
- 1 x slot PCI Express x1 running x1 (PCIEX1). Populated with an intel NIC. Dual 1Gbps (E1G42ET - Intel 82576). This follows PCI Express 2.0 standard.
- 1 x slot PCI Express x16 running x4 (PCIEX4). Populated with an intel NIC. Dual 10Gbps (X520-DA2 - Intel 82599ES). This follows PCI Express 2.0 standard.
Proxmox 6.3-4 in cluster (the other node an HP micro gen8 but that should not have any effect on this)

1. BIOS settings
As explained in other tutorials it is must to have support form the motherboard in IOMMU.

GA-AB350M BIOS F50d

Enable IOMMU

A side note, please bear in mind that proxmox needs SVM mode enabled for KVM functionality (not related to this, but good to remember)

2. Proxmox Configuration
As explained in other tutorials as of version 6.3-4 of Proxmox there is the need to add /etc/modprobe.d module configurations for the NIC drivers in order to set the maximum number of virtual functions for each NIC, add /etc/modules vfio modules and update the /etc/default/grub bootloader with command lines to enable IOMMU,

Add configuration files in /etc/modprobe.d (create the files as probably they do not exist). igb.conf provide 4 virtual functions for each of the two 1Gbps ports of E1G42ET. ixgbe.conf provide 8 virtual functions for each of the two ports 10Gbps of X520-DA2
- file /etc/modprobe.d/igb.conf. Add the following: options igb max_vfs=4

root@ryzen:~# cat /etc/modprobe.d/igb.conf
options igb max_vfs=4

file /etc/modprobe.d/ixgbe.conf. Add the following: options ixgbe max_vfs=8

root@ryzen:~# cat /etc/modprobe.d/ixgbe.conf
options ixgbe max_vfs=8

Edit /etc/modules to add configuration for vfio modules
- file /etc/modules. Add the following lines:

root@ryzen:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Update initramfs

root@ryzen:~# update-initramfs -u -k all

Edit /etc/default/grub Note that the process of GRUB command line update needs to be done in two steps. If only IOMMU and Passthrough are enabled in GRUB, the operating system creates a number of IOMMU groups that mix devices that we want separated in order to passthrough them independently. Let's see it:
- Initially the line GRUB_CMDLINE_LINUX_DEFAUL. should be edited to look like this: GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
  - amd_iommu=on This is required to enable IOMMU support in the kernel.
  - iommu=pt This is required to enable IOMMU passthrough
- edit grub file

root@ryzen:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
....
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""
....

Update grub

root@ryzen:~# update-grub

Reboot Proxmox server

root@ryzen:~# reboot

After reboot. Let's check that IOMMU is active and the IOMMU groups generated

root@ryzen:~# dmesg | grep  -e IOMMU -e AMD-Vi
[    0.765091] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.766199] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.766200] pci 0000:00:00.2: AMD-Vi: Extended features (0xf77ef22294ada):
[    0.766203] AMD-Vi: Interrupt remapping enabled
[    0.766203] AMD-Vi: Virtual APIC enabled
[    0.766365] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.766528] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

Let's check the IOMMU groups generated by the kernel

root@ryzen:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 0 01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E12 NVMe Controller [1987:5012] (rev 01)
IOMMU Group 0 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset USB 3.1 xHCI Controller [1022:43bb] (rev 02)
IOMMU Group 0 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset SATA Controller [1022:43b7] (rev 02)
IOMMU Group 0 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)
IOMMU Group 0 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 0 03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 0 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 0 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0c)
IOMMU Group 0 05:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 0 05:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 0 06:10.0 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 0 06:10.1 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 0 06:10.2 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 0 06:10.3 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 0 06:10.4 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 0 06:10.5 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 0 06:10.6 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 0 06:10.7 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 0 07:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
IOMMU Group 0 07:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
IOMMU Group 0 08:10.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:10.2 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:10.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:10.4 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:10.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:10.6 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:10.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:11.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:11.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:11.2 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:11.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:11.4 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:11.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:11.6 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 0 08:11.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 2 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
IOMMU Group 2 09:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU Group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 4 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU Group 4 0a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 4 0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 5 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU Group 5 0b:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 5 0b:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU Group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU Group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]

TO BE CONTINUED.

daisychain · Feb 28, 2021

CONTINUATION.

As it is shown there are a limited number of IOMMU groups (up to seven) which are mixing a number of devices. In order to have a much more granular control and allow to have devices or virtual functions to be passed through independently we need to update grub with an extra parameter. This is required because automatically the operating system does not create independent IOMMU groups for each of the devices (NICs and virtual Function NICs). If IOMMU groups hold several devices together then PCI passthrough does not work and in my case I was getting a proxmox locked with kernel errors and hanging.
pcie_acs_override=downstream,multifunction,id:8086:10ca,id:8086:10ed,id:8086:10fb,id:8086:10c9 The values 8086:10ca, etc. are vendorID (VID) and deviceID (DID) of the physical and virtual functions of each of the two NICs. You can get the values of these by checking output of command

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;

E1G42ET - Intel 82576
- Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
- Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
X520-DA2 - Intel 82599ES
- Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
- Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
Edit /etc/default/grub . with this new parameter and the values of VID: DID

root@ryzen:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'
....
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction,id:8086:10ca,id:8086:10ed,id:8086:10fb,id:8086:10c9"
GRUB_CMDLINE_LINUX=""
....

Update grub

root@ryzen:~# update-grub

Reboot Proxmox server

root@ryzen:~# reboot

Let's check the IOMMU groups generated by the kernel

root@ryzen:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 10 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 11 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 11 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 12 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 12 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 12 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 12 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 12 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 12 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU Group 12 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 13 01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E12 NVMe Controller [1987:5012] (rev 01)
IOMMU Group 14 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset USB 3.1 xHCI Controller [1022:43bb] (rev 02)
IOMMU Group 15 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset SATA Controller [1022:43b7] (rev 02)
IOMMU Group 16 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)
IOMMU Group 17 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 18 03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 19 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 20 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0c)
IOMMU Group 21 05:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 22 05:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 23 07:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
IOMMU Group 24 07:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
IOMMU Group 25 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
IOMMU Group 26 09:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU Group 27 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU Group 28 0a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 29 0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 30 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU Group 31 0b:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 32 0b:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU Group 33 06:10.0 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 34 06:10.2 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 35 06:10.4 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 36 06:10.6 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 37 08:10.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 38 08:10.2 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 39 08:10.4 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 40 08:10.6 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 41 08:11.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 42 08:11.2 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 43 08:11.4 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 44 08:11.6 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 45 06:10.1 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 46 06:10.3 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 47 06:10.5 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 48 06:10.7 Ethernet controller [0200]: Intel Corporation 82576 Virtual Function [8086:10ca] (rev 01)
IOMMU Group 49 08:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 50 08:10.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 51 08:10.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 52 08:10.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 53 08:11.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 54 08:11.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 55 08:11.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 56 08:11.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]

As you can see now we have many more IOMMU groups and we are able to do passthrough of the independent virtual functions
We can for example pass a dedicated virtual NICS to a VM by adding them as PCI Hardware:

And select the one we are interested for the list of virtual functions

AS you can see the new NICs are passed to the VM with this example of four VFs:

I hope this can help.
END OF TUTORIAL

Ramalama · Feb 28, 2021

Can you ping from your kvm vm (vf) any lxc container (linux bridge)? xD

daisychain · Feb 28, 2021

I am fairly new to proxmox

In my setup what I see is that virtual functions can be passed to VMs as PCI devices. However if you want to use the linux bridge functionality of proxmox I can only attach to them physical interfaces (virtual interfaces do not work/route traffic).
Example.

Consider container: lxc1.

As you can see LXC is in vmbr2 bridged to the physical port enp5s0f1

Consider VM: mxLinuxDesktop

As you can see the virtual function enp5s0f1v3 is passed directly to the VM

root@ryzen:~# grep PCI_SLOT_NAME /sys/class/net/enp5s0*/device/uevent
/sys/class/net/enp5s0f0/device/uevent:PCI_SLOT_NAME=0000:05:00.0
/sys/class/net/enp5s0f0v0/device/uevent:PCI_SLOT_NAME=0000:06:10.0
/sys/class/net/enp5s0f0v1/device/uevent:PCI_SLOT_NAME=0000:06:10.2
/sys/class/net/enp5s0f0v2/device/uevent:PCI_SLOT_NAME=0000:06:10.4
/sys/class/net/enp5s0f0v3/device/uevent:PCI_SLOT_NAME=0000:06:10.6
/sys/class/net/enp5s0f1/device/uevent:PCI_SLOT_NAME=0000:05:00.1
/sys/class/net/enp5s0f1v0/device/uevent:PCI_SLOT_NAME=0000:06:10.1
/sys/class/net/enp5s0f1v1/device/uevent:PCI_SLOT_NAME=0000:06:10.3
/sys/class/net/enp5s0f1v2/device/uevent:PCI_SLOT_NAME=0000:06:10.5

enp5s0f1v3 is missing in the list above taken in the PVE host, so only available to the VM.

We can ping each other (on the left teh VM and on the right the LXC):

To me this is good enough... I use virtual function to provide direct access to VMs that require direct access to the NIC driver for performance reasons. For the rest of VMs and all LXC I do not use virtual Functions but just linux bridges and proxmox firewall to the physical interfaces

Ramalama · Feb 28, 2021

You understood me wrong.
Normally you use virtual functions for VM's and linux bridge for LXC.

But Normally those 2 aren't able to communicate with each other, due to a intel or Linux Bug.
Thats why i simply asked you, if it works for you, or you have the same issue. There is a workaround with addong mac adresses to the phy interface.

But seems for me, like everything is working for you xD

daisychain · Feb 28, 2021

Ah ok. As you can see above it looks that in Proxmox 6.3-4 with igb and ixgb drivers pinging between them works.

daisychain · Apr 1, 2021

As completion of this thread and following @Ramalama comment, please see thread https://forum.proxmox.com/threads/communication-issue-between-sriov-vm-vf-and-ct-on-pf-bridge.68638/ which describes the problem of not communication between a VM with a VF and a CT in a bridge in the PF. There is a nice script to update the bridge database should you encounter this issue.

bluegogo · Aug 27, 2021

Dude, thank you very much, your tutorial is very useful to me, I have followed your tutorial sr-iov and it has been able to work normally

My hardware platform is:
cpu: i3 9100
Motherboard: asrock b365m-itx
Network card: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2
pve version: 7.0-8

daisychain · Aug 28, 2021

@bluegogo . Cool!

Search

Search

[TUTORIAL] SR-IOV in AMD motherboard for two intel NICs: 1Gbps (E1G42ET - Intel 82576) 10 Gbps (X520-DA2 - Intel 82599ES) in Proxmox 6.3-4

daisychain

Member

daisychain

Member

Ramalama

Well-Known Member

daisychain

Member

Ramalama

Well-Known Member

daisychain

Member

daisychain

Member

bluegogo

New Member

daisychain

Member