[Support] Issues with SR-IOV on Proxmox 8.2.2

UnixChump · Jun 20, 2024

Hi all,

I've used the below method to setup SR-IOV for my Intel X550T on Proxmox 7-x installs, but I can't seem to get it working on 8-x installs. Does anyone see what I'm missing or doing wrong?

Noting I'm using systemd boot since I have ZFS, no GRUB.

Code:

root@pve-labby:~# pveversion --verbose
proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.4-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.0-1
proxmox-backup-file-restore: 3.2.0-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.1
pve-cluster: 8.0.6
pve-container: 5.0.10
pve-docs: 8.2.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.5
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2

bootctl output.

Code:

System:
      Firmware: UEFI 2.40 (American Megatrends 5.11)
 Firmware Arch: x64
   Secure Boot: disabled (setup)
  TPM2 Support: no
root@pve-labby:~# cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt




  Boot into FW: supported


Current Boot Loader:
      Product: systemd-boot 252.22-1~deb12u1
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Load drop-in drivers
               ✓ Support Type #1 sort-key field
               ✓ Support @saved pseudo-entry
               ✓ Support Type #1 devicetree field
               ✓ Boot loader sets ESP information
          ESP: /dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI


Random Seed:
 Passed to OS: yes
 System Token: set
       Exists: yes


Available Boot Loaders on ESP:
          ESP: /boot/efi (/dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2)
         File: ├─/EFI/systemd/systemd-bootx64.efi (systemd-boot 252.22-1~deb12u1)
               └─/EFI/BOOT/BOOTX64.EFI (systemd-boot 252.22-1~deb12u1)


Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0001
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI


        Title: Linux Boot Manager
           ID: 0x0000
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/27f12307-9d22-4d77-b8d0-c41a32fd870d
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI


        Title: UEFI OS
           ID: 0x0006
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/27f12307-9d22-4d77-b8d0-c41a32fd870d
         File: └─/EFI/BOOT/BOOTX64.EFI


        Title: UEFI OS
           ID: 0x0007
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2
         File: └─/EFI/BOOT/BOOTX64.EFI


Boot Loader Entries:
        $BOOT: /boot/efi (/dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2)


Default Boot Loader Entry:
         type: Boot Loader Specification Type #1 (.conf)
        title: Proxmox Virtual Environment
           id: proxmox-6.8.4-2-pve.conf
       source: /boot/efi/loader/entries/proxmox-6.8.4-2-pve.conf
      version: 6.8.4-2-pve
        linux: /EFI/proxmox/6.8.4-2-pve/vmlinuz-6.8.4-2-pve
       initrd: /EFI/proxmox/6.8.4-2-pve/initrd.img-6.8.4-2-pve
      options: root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt

Creation of VFs on boot.

Code:

root@pve-labby:~# cat /etc/systemd/system/sriov.service
[Unit]
Description=Script to enable SR-IOV on boot


#Replace the PCIe device, and desired amount of VFs as needed. Typically, you would create 1 VF per VM.
[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c '/usr/bin/echo 4 > /sys/class/net/enp2s0f1/device/sriov_numvfs'


[Install]
WantedBy=multi-user.target

Ethernet Controller PF and VF IDs and info.

Code:

root@pve-labby:~# lspci | grep Ethernet
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X550 (rev 01)
02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X550 (rev 01)
02:10.1 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.3 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.5 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.7 Ethernet controller: Intel Corporation X550 Virtual Function

The PF is set to autostart on boot.

Code:

root@pve-labby:~# cat /etc/network/interfaces
auto enp2s0f1
iface enp2s0f1 inet manual

#I loaded the vfio modules.

Code:

root@pve-labby:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.


vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

#Test VM Config. This is a Centos 7 guest.

Code:

root@pve-labby:~# qm config 100
balloon: 0
boot: order=scsi0
cores: 20
cpu: host
hostpci0: 0000:02:10.1,pcie=1
machine: q35
memory: 24576
meta: creation-qemu=8.1.5,ctime=1718907736
name: SRIOV-TEST
numa: 0
ostype: l26
scsi0: local-zfs:vm-100-disk-0,iothread=1,size=128G
scsihw: virtio-scsi-single
smbios1: uuid=d0fde6f9-22cc-46b0-85f4-e89b68cc3f6b
sockets: 1
tags: dr.giggletouch
vmgenid: e2610704-5f09-4a35-9e0b-b7fdfe60fe8d

gianpaoloracca · Jun 27, 2024

I think you have to pin an older kernel (6.5), because the driver won't work with 6.8 kernels.
More here (just at the top of the page).

UnixChump · Jun 27, 2024

gianpaoloracca said:
I think you have to pin an older kernel (6.5), because the driver won't work with 6.8 kernels.
More here (just at the top of the page).

Thanks! I tried this, but I can't ping out of the VM. I see it created the virtual functions.

Bash:

root@pve-labby:~# apt-get update
root@pve-labby:~# apt install -y pve-kernel-6.5.13-5

Bash:

root@pve-labby:~# proxmox-boot-tool kernel pin 6.5.13-5-pve

Bash:

root@pve-labby:~# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
6.5.13-5-pve
6.8.4-2-pve
6.8.8-1-pve

Pinned kernel:
6.5.13-5-pve

Bash:

root@pve-labby:~# reboot now

Bash:

root@pve-labby:~# uname -a
Linux pve-labby 6.5.13-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-5 (2024-04-05T11:03Z) x86_64 GNU/Linux

#Disable and/or remove the VF creation systemd service.

Bash:

root@pve-labby:~# systemctl disable sriov.service

#Adjust kernel cmdline to create VFs.

Bash:

root@pve-labby:~# vi /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=8

Bash:

root@pve-labby:~# proxmox-boot-tool refresh

Bash:

root@pve-labby:~# bootctl install

Bash:

root@pve-labby:~# reboot now

Bash:

root@pve-labby:~# bootctl
System:
      Firmware: UEFI 2.40 (American Megatrends 5.11)
 Firmware Arch: x64
   Secure Boot: disabled (setup)
  TPM2 Support: no
  Boot into FW: supported


Current Boot Loader:
      Product: systemd-boot 252.22-1~deb12u1
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Load drop-in drivers
               ✓ Support Type #1 sort-key field
               ✓ Support @saved pseudo-entry
               ✓ Support Type #1 devicetree field
               ✓ Boot loader sets ESP information
          ESP: /dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI


Random Seed:
 Passed to OS: yes
 System Token: set
       Exists: yes


Available Boot Loaders on ESP:
          ESP: /boot/efi (/dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2)
         File: ├─/EFI/systemd/systemd-bootx64.efi (systemd-boot 252.22-1~deb12u1)
               └─/EFI/BOOT/BOOTX64.EFI (systemd-boot 252.22-1~deb12u1)


Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0001
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI


        Title: Linux Boot Manager
           ID: 0x0000
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/27f12307-9d22-4d77-b8d0-c41a32fd870d
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI


        Title: UEFI OS
           ID: 0x0006
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/27f12307-9d22-4d77-b8d0-c41a32fd870d
         File: └─/EFI/BOOT/BOOTX64.EFI


        Title: UEFI OS
           ID: 0x0007
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2
         File: └─/EFI/BOOT/BOOTX64.EFI


Boot Loader Entries:
        $BOOT: /boot/efi (/dev/disk/by-partuuid/a3a7f1ba-1682-40d1-8ef7-80406053a4a2)


Default Boot Loader Entry:
         type: Boot Loader Specification Type #1 (.conf)
        title: Proxmox Virtual Environment (6.5.13-5-pve)
           id: proxmox-6.5.13-5-pve.conf
       source: /boot/efi/loader/entries/proxmox-6.5.13-5-pve.conf
      version: 6.5.13-5-pve
        linux: /EFI/proxmox/6.5.13-5-pve/vmlinuz-6.5.13-5-pve
       initrd: /EFI/proxmox/6.5.13-5-pve/initrd.img-6.5.13-5-pve
      options: root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=8

Bash:

root@pve-labby:~# lspci | grep Ethernet
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X550 (rev 01)
02:00.1 Ethernet controller: Intel Corporation Ethernet Controller X550 (rev 01)
02:10.0 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.1 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.2 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.3 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.4 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.5 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.6 Ethernet controller: Intel Corporation X550 Virtual Function
02:10.7 Ethernet controller: Intel Corporation X550 Virtual Function
02:11.0 Ethernet controller: Intel Corporation X550 Virtual Function
02:11.1 Ethernet controller: Intel Corporation X550 Virtual Function
02:11.2 Ethernet controller: Intel Corporation X550 Virtual Function
02:11.3 Ethernet controller: Intel Corporation X550 Virtual Function
02:11.4 Ethernet controller: Intel Corporation X550 Virtual Function
02:11.5 Ethernet controller: Intel Corporation X550 Virtual Function
02:11.6 Ethernet controller: Intel Corporation X550 Virtual Function
02:11.7 Ethernet controller: Intel Corporation X550 Virtual Function

EpiJunkie · Jul 7, 2024

It's great that you got it working on kernel 6.5. Here is how I got SRIOV working on 6.8. I figure getting on 6.8 earlier will prevent further issues down the road with future updates.

I recently updated from 8.1.4 to 8.2.2 and my SR-IOV networking stop working. I have Intel X710 cards. I could boot on 8.2.2 with the 6.5.x kernel fine but the 6.8 kernel would break the networking. First issue was the was the 'npX' being appended to the ports where 'X' was the port number (known issue for NIC cards that use the i40e driver). I also forgot that I was previously using the upstream Intel driver module, instead of the included i40e driver module. I found that SRIOV was faster and more stable using the upstream Intel driver module. So, I rebuilt the Intel i40e driver module, version 2.25.9 and updated the firmware from version 9.40 to 9.50. Before I updated the Intel driver module, the VFs would not pass traffic. I would suggest doing the same process for your X550 (driver and firmware).

Use the upstream Intel driver:

Bash:

modinfo ixgbe
wget https://downloadmirror.intel.com/825849/ixgbe-5.20.9.tar.gz
tar xvzf ixgbe-5.20.9.tar.gz
cd ixgbe-5.20.9/src
apt install pve-headers
make install
reboot
modinfo ixgbe #check against first output

Update NIC firmware:

Bash:

screen
wget https://downloadmirror.intel.com/727466/X550_NVMUpdatePackage_v3_60_Linux.tar.gz
tar xvzf X550_NVMUpdatePackage_v3_60_Linux.tar.gz
cd X550/Linux_x64
chmod +x nvmupdate64e
./nvmupdate64e
# Select the card to update
# Agree to backup existing firmware
# Wait for process to finish (make sure to have stable power)
reboot

Notes:

You need the make tools installed to build the Intel driver module, which can be installed with apt-get install build-essential.
The firmware update failed using the included i40e driver module. After updating to use the upstream Intel driver module, I was able to update the firmware quickly.
Make sure to update the firmware on a physical terminal or in a screen/tmux session. There is PCIe reset that occurs during the update process which will cause SSH to drop and potentially cause issues with the process to finish the update properly.
After updating the driver module, you can check that it worked by running modinfo ixgbe and checking the version number (literal 'version: 5.20.9.', not 'vermagic'). The 'version:' field will be missing with the include driver. Only the included driver will be signed and have a 'signature:' field.

anime · Aug 3, 2024

EpiJunkie said:
It's great that you got it working on kernel 6.5. Here is how I got SRIOV working on 6.8. I figure getting on 6.8 earlier will prevent further issues down the road with future updates.

I recently updated from 8.1.4 to 8.2.2 and my SR-IOV networking stop working. I have Intel X710 cards. I could boot on 8.2.2 with the 6.5.x kernel fine but the 6.8 kernel would break the networking. First issue was the was the 'npX' being appended to the ports where 'X' was the port number (known issue for NIC cards that use the i40e driver). I also forgot that I was previously using the upstream Intel driver module, instead of the included i40e driver module. I found that SRIOV was faster and more stable using the upstream Intel driver module. So, I rebuilt the Intel i40e driver module, version 2.25.9 and updated the firmware from version 9.40 to 9.50. Before I updated the Intel driver module, the VFs would not pass traffic. I would suggest doing the same process for your X550 (driver and firmware).

Use the upstream Intel driver:

Bash:

modinfo ixgbe wget https://downloadmirror.intel.com/825849/ixgbe-5.20.9.tar.gz tar xvzf ixgbe-5.20.9.tar.gz cd ixgbe-5.20.9/src apt install pve-headers make install reboot modinfo ixgbe #check against first output

Update NIC firmware:

Bash:

screen wget https://downloadmirror.intel.com/727466/X550_NVMUpdatePackage_v3_60_Linux.tar.gz tar xvzf X550_NVMUpdatePackage_v3_60_Linux.tar.gz cd X550/Linux_x64 chmod +x nvmupdate64e ./nvmupdate64e # Select the card to update # Agree to backup existing firmware # Wait for process to finish (make sure to have stable power) reboot

Notes:

You need the make tools installed to build the Intel driver module, which can be installed with apt-get install build-essential.

The firmware update failed using the included i40e driver module. After updating to use the upstream Intel driver module, I was able to update the firmware quickly.

Make sure to update the firmware on a physical terminal or in a screen/tmux session. There is PCIe reset that occurs during the update process which will cause SSH to drop and potentially cause issues with the process to finish the update properly.

After updating the driver module, you can check that it worked by running modinfo ixgbe and checking the version number (literal 'version: 5.20.9.', not 'vermagic'). The 'version:' field will be missing with the include driver. Only the included driver will be signed and have a 'signature:' field.

Thanks @EpiJunkie for posting crystal clear instructions. I'm using Intel X710-DA2 as well and on a latest PVE kernel 6.8.8-4 and while I'm able to create multiple VF for some reason VFIO drivers are not getting loaded and once I pass the VF to VM, the NIC lose network connection (host is alive, does not crash). I followed your instructions - updated driver and firmware to no avail.

I guessing this is another kernel temper tantrum, any ideas how to go about fixing this? I hate pinning to older kernel.

Thanks!

anime · Aug 4, 2024

Want to add what triggers the behavior -- VMBR on same NIC using PF can no longer ping the host once the VF is assigned the MAC address. This is new behavior from PVE/Kernel 6.8.x. Any input to address this is much appreciated.

Code:

root@rack:~# ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.646 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=0.307 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=64 time=0.209 ms
^C
--- 192.168.10.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2040ms
rtt min/avg/max/mdev = 0.209/0.387/0.646/0.187 ms

root@rack:~# /usr/bin/ip link set enp1s0f0np0 vf 0 mac 51:c5:b8:48:8a:00
root@rack:~# ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
^C
--- 192.168.10.1 ping statistics ---
40 packets transmitted, 0 received, 100% packet loss, time 39915ms

Newly created VF is reported DOWN by ip addr

Code:

enp1s0f0v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 51:c5:b8:48:8a:00 brd ff:ff:ff:ff:ff:ff

SmashedSqwurl · Aug 20, 2024

anime said:

Want to add what triggers the behavior -- VMBR on same NIC using PF can no longer ping the host once the VF is assigned the MAC address. This is new behavior from PVE/Kernel 6.8.x. Any input to address this is much appreciated.

Code:

root@rack:~# ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.646 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=0.307 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=64 time=0.209 ms
^C
--- 192.168.10.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2040ms
rtt min/avg/max/mdev = 0.209/0.387/0.646/0.187 ms

root@rack:~# /usr/bin/ip link set enp1s0f0np0 vf 0 mac 51:c5:b8:48:8a:00
root@rack:~# ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
^C
--- 192.168.10.1 ping statistics ---
40 packets transmitted, 0 received, 100% packet loss, time 39915ms

Newly created VF is reported DOWN by ip addr

Code:

enp1s0f0v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 51:c5:b8:48:8a:00 brd ff:ff:ff:ff:ff:ff

I was having the same issue until I found this thread: https://forum.proxmox.com/threads/communication-issue-between-sriov-vm-vf-and-ct-on-pf-bridge.68638/

Using the hookscript at https://github.com/jdlayman/pve-hookscript-sriov for my VMs and CTs fixed the problem.

Search

Search

[Support] Issues with SR-IOV on Proxmox 8.2.2

UnixChump

New Member

gianpaoloracca

New Member

UnixChump

New Member

EpiJunkie

New Member

anime

New Member

anime

New Member

SmashedSqwurl

Member