Unable to get SR-IOV to work; perhaps due to installation?

[ajtatum]

So, I'm following the instructions found here: https://www.reddit.com/r/Proxmox/comments/cm81tc/tutorial_enabling_sriov_for_intel_nic_x550t2_on/

And I've updated my grub and /etc/modules file, but when I try to verify that they're functioning, dmesg doesn't return anything.

When running update-grub, the following is given:

update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.2.16-9-pve
Found initrd image: /boot/initrd.img-6.2.16-9-pve
Found linux image: /boot/vmlinuz-6.2.16-5-pve
Found initrd image: /boot/initrd.img-6.2.16-5-pve
Found linux image: /boot/vmlinuz-6.2.16-4-pve
Found initrd image: /boot/initrd.img-6.2.16-4-pve
Found linux image: /boot/vmlinuz-5.15.108-1-pve
Found initrd image: /boot/initrd.img-5.15.108-1-pve
Found linux image: /boot/vmlinuz-5.15.74-1-pve
Found initrd image: /boot/initrd.img-5.15.74-1-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Adding boot menu entry for UEFI Firmware Settings ...
done

When I run undate-intramfs, I get:

update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-6.2.16-9-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-6.2.16-5-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-6.2.16-4-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.108-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.74-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

Seeing as there appears to be multiple kernels, I ran the Kernel Cleanup script found on https://tteck.github.io/Proxmox/. It initially reported the old kernels and I had them removed and when I run it now it shows that only 6.2.16-9 is installed. Yet, after running the above again it shows all the old kernels.

Digging deeper, I run proxmox-boot-tool:

proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
E: /etc/kernel/proxmox-boot-uuids does not exist.

I run lsblk and I can see the "/" mount point:

lsblk
NAME                                     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                                        8:0    0   7.3T  0 disk
sdb                                        8:16   0   7.3T  0 disk
sdc                                        8:32   0 465.8G  0 disk
├─sdc1                                     8:33   0  1007K  0 part
├─sdc2                                     8:34   0   512M  0 part /boot/efi
└─sdc3                                     8:35   0 465.3G  0 part
  ├─pve-swap                             253:0    0     8G  0 lvm  [SWAP]
  └─pve-root                             253:1    0 457.3G  0 lvm  /
nvme2n1                                  259:0    0   1.8T  0 disk
├─nvme2tb-nvme2tb_tmeta                  253:4    0  15.8G  0 lvm 
│ └─nvme2tb-nvme2tb-tpool                253:11   0   1.8T  0 lvm 
│   ├─nvme2tb-nvme2tb                    253:12   0   1.8T  1 lvm 
│   ├─nvme2tb-vm--100--disk--0           253:13   0     4M  0 lvm 
│   ├─nvme2tb-vm--100--disk--1           253:14   0    32G  0 lvm 
│   ├─nvme2tb-vm--101--disk--0           253:15   0   128G  0 lvm 
│   ├─nvme2tb-vm--103--disk--0           253:16   0     4M  0 lvm 
│   ├─nvme2tb-vm--103--disk--1           253:17   0   127G  0 lvm 
│   ├─nvme2tb-vm--105--disk--0           253:18   0    64G  0 lvm 
│   ├─nvme2tb-vm--106--disk--0           253:19   0   128G  0 lvm 
│   ├─nvme2tb-vm--106--state--AppRestore 253:20   0  24.5G  0 lvm 
│   ├─nvme2tb-vm--109--disk--0           253:21   0   256G  0 lvm 
│   ├─nvme2tb-vm--102--disk--0           253:22   0    64G  0 lvm 
│   ├─nvme2tb-vm--102--disk--1           253:23   0   256G  0 lvm 
│   ├─nvme2tb-vm--108--disk--0           253:24   0    64G  0 lvm 
│   ├─nvme2tb-vm--115--disk--0           253:25   0   128G  0 lvm 
│   ├─nvme2tb-vm--111--disk--0           253:26   0   128G  0 lvm 
│   └─nvme2tb-vm--112--disk--0           253:27   0   128G  0 lvm 
└─nvme2tb-nvme2tb_tdata                  253:5    0   1.8T  0 lvm 
  └─nvme2tb-nvme2tb-tpool                253:11   0   1.8T  0 lvm 
    ├─nvme2tb-nvme2tb                    253:12   0   1.8T  1 lvm 
    ├─nvme2tb-vm--100--disk--0           253:13   0     4M  0 lvm 
    ├─nvme2tb-vm--100--disk--1           253:14   0    32G  0 lvm 
    ├─nvme2tb-vm--101--disk--0           253:15   0   128G  0 lvm 
    ├─nvme2tb-vm--103--disk--0           253:16   0     4M  0 lvm 
    ├─nvme2tb-vm--103--disk--1           253:17   0   127G  0 lvm 
    ├─nvme2tb-vm--105--disk--0           253:18   0    64G  0 lvm 
    ├─nvme2tb-vm--106--disk--0           253:19   0   128G  0 lvm 
    ├─nvme2tb-vm--106--state--AppRestore 253:20   0  24.5G  0 lvm 
    ├─nvme2tb-vm--109--disk--0           253:21   0   256G  0 lvm 
    ├─nvme2tb-vm--102--disk--0           253:22   0    64G  0 lvm 
    ├─nvme2tb-vm--102--disk--1           253:23   0   256G  0 lvm 
    ├─nvme2tb-vm--108--disk--0           253:24   0    64G  0 lvm 
    ├─nvme2tb-vm--115--disk--0           253:25   0   128G  0 lvm 
    ├─nvme2tb-vm--111--disk--0           253:26   0   128G  0 lvm 
    └─nvme2tb-vm--112--disk--0           253:27   0   128G  0 lvm 
nvme1n1                                  259:1    0 931.5G  0 disk
├─nvme980--1tb-nvme980--1tb_tmeta        253:2    0   9.3G  0 lvm 
│ └─nvme980--1tb-nvme980--1tb-tpool      253:6    0 912.8G  0 lvm 
│   ├─nvme980--1tb-nvme980--1tb          253:7    0 912.8G  1 lvm 
│   ├─nvme980--1tb-vm--107--disk--0      253:8    0   256G  0 lvm 
│   ├─nvme980--1tb-vm--100--disk--0      253:9    0   150G  0 lvm 
│   └─nvme980--1tb-vm--110--disk--0      253:10   0   256G  0 lvm 
└─nvme980--1tb-nvme980--1tb_tdata        253:3    0 912.8G  0 lvm 
  └─nvme980--1tb-nvme980--1tb-tpool      253:6    0 912.8G  0 lvm 
    ├─nvme980--1tb-nvme980--1tb          253:7    0 912.8G  1 lvm 
    ├─nvme980--1tb-vm--107--disk--0      253:8    0   256G  0 lvm 
    ├─nvme980--1tb-vm--100--disk--0      253:9    0   150G  0 lvm 
    └─nvme980--1tb-vm--110--disk--0      253:10   0   256G  0 lvm 
nvme0n1                                  259:2    0 953.9G  0 disk

So, it made me curious about maybe I messed something up during the initial install, which would suck. I found this article: https://pve.proxmox.com/wiki/ZFS:_Switch_Legacy-Boot_to_Proxmox_Boot_Tool but I'm afraid of running anything as I don't want to lose my setup with VMs and Containers.

Proxmox isn't meant to be a storage server for me, it's main purpose was to be for VMs and containers and offload backups, etc to my Synology NAS, and I didn't think I needed ZFS during install as I was concerned about memory usage.

Anyway, I was hoping someone can help me figure out what I can do to solve the first issue here with the update-grub and whatnot so that I can continue to try and get SR-IOV working on my Intel T2-X550 ethernet cards.

Any advice would be greatly appreciated. Please let me know if you need any additional information.

Thanks,
AJ

 

[ajtatum]

Anyone able to help me figure this out? I'd really appreciate it!

 

[leesteken]

The various kernel versions and the warning about not using ESP boot partitions/proxmox-boot-tool is not related to SR-IOV, and I don't think you have to worry about this.
What problems are you having with SR-IOV? What did you (attempt to) change about the initramfs? You can check the current active kernel parameters with cat /proc/cmdline. Are you trying to do PCI(e) passthrough? Maybe you can share more details about what is working and what is not working in that regard.

 

[ajtatum]

The various kernel versions and the warning about not using ESP boot partitions/proxmox-boot-tool is not related to SR-IOV, and I don't think you have to worry about this.
What problems are you having with SR-IOV? What did you (attempt to) change about the initramfs? You can check the current active kernel parameters with cat /proc/cmdline. Are you trying to do PCI(e) passthrough? Maybe you can share more details about what is working and what is not working in that regard.

Sorry for the late response, I'm trying to get virtual functions to work with my X550-T2 ethernet card so that I can have more "ports" than what I actually have.

The output from the command you gave above is:

BOOT_IMAGE=/boot/vmlinuz-6.2.16-9-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt

The update I made to /etc/modules was adding the following modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

According to the Reddit post (and there's a similar thread here), after rebooting Proxmox, I should receive a message by running

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

Which actually now returns something:

[    0.004665] ACPI: DMAR 0x000000006175B000 000088 (v02 INTEL  Dell Inc 00000002      01000013)
[    0.004700] ACPI: Reserving DMAR table memory at [mem 0x6175b000-0x6175b087]
[    0.135569] DMAR: IOMMU enabled
[    0.313318] DMAR: Host address width 39
[    0.313318] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.313322] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e
[    0.313323] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.313326] DMAR: dmar1: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[    0.313327] DMAR: RMRR base: 0x0000006c000000 end: 0x000000703fffff
[    0.313329] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.313330] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.313330] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.314855] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.865658] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    1.354989] DMAR: No ATSR found
[    1.354990] DMAR: No SATC found
[    1.354991] DMAR: IOMMU feature fl1gp_support inconsistent
[    1.354991] DMAR: IOMMU feature pgsel_inv inconsistent
[    1.354992] DMAR: IOMMU feature nwfs inconsistent
[    1.354992] DMAR: IOMMU feature dit inconsistent
[    1.354992] DMAR: IOMMU feature sc_support inconsistent
[    1.354993] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    1.354993] DMAR: dmar0: Using Queued invalidation
[    1.354995] DMAR: dmar1: Using Queued invalidation
[    1.355793] DMAR: Intel(R) Virtualization Technology for Directed I/O

Per article, I should now be able to set SR-IOV by calling

echo 2 > /sys/class/net/enp2s0f0/device/sriov_numvfs

enp2s0f0 is one of the network interfaces; however, there is no sriov_numvfs folder.

However, when I go into the directory

/sys/class/net/enp2s0f0/device

I see that the driver is mapped to ixgbe, whereas it should, I believe, be ixgbevf. When I do a search for that driver via

find / -name ixgbevf

, it brings up ones for various kernels, but has one located at

/usr/lib/modules/6.2.16-9-pve/kernel/drivers/net/ethernet/intel/ixgbevf

That's unfortunately as far as my Linux knowledge takes me as I don't know how to tell it to use a different driver or how to install that driver.

Also, just in case it helps, here's what's in /sys/class/net/enp2s0f0/device...

root@pve:/sys/class/net/enp2s0f0/device# ls -la
total 0
drwxr-xr-x  7 root root       0 Aug 18 08:38 .
drwxr-xr-x 12 root root       0 Aug 18 08:38 ..
-r--r--r--  1 root root    4096 Aug 21 16:57 aer_dev_correctable
-r--r--r--  1 root root    4096 Aug 21 16:57 aer_dev_fatal
-r--r--r--  1 root root    4096 Aug 21 16:57 aer_dev_nonfatal
-r--r--r--  1 root root    4096 Aug 18 08:38 ari_enabled
-rw-r--r--  1 root root    4096 Aug 21 16:57 broken_parity_status
-r--r--r--  1 root root    4096 Aug 20 03:14 class
-rw-r--r--  1 root root    4096 Aug 18 08:38 config
-r--r--r--  1 root root    4096 Aug 21 16:57 consistent_dma_mask_bits
-r--r--r--  1 root root    4096 Aug 21 16:57 current_link_speed
-r--r--r--  1 root root    4096 Aug 21 16:57 current_link_width
-rw-r--r--  1 root root    4096 Aug 21 16:57 d3cold_allowed
-r--r--r--  1 root root    4096 Aug 18 08:38 device
-r--r--r--  1 root root    4096 Aug 21 16:57 dma_mask_bits
lrwxrwxrwx  1 root root       0 Aug 18 08:38 driver -> ../../../../bus/pci/drivers/ixgbe
-rw-r--r--  1 root root    4096 Aug 21 16:57 driver_override
-rw-r--r--  1 root root    4096 Aug 21 16:57 enable
lrwxrwxrwx  1 root root       0 Aug 21 16:57 firmware_node -> ../../../LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:55/device:56
lrwxrwxrwx  1 root root       0 Aug 21 16:57 iommu -> ../../../virtual/iommu/dmar1
lrwxrwxrwx  1 root root       0 Aug 21 08:31 iommu_group -> ../../../../kernel/iommu_groups/15
-r--r--r--  1 root root    4096 Aug 21 08:31 irq
drwxr-xr-x  2 root root       0 Aug 21 16:57 link
-r--r--r--  1 root root    4096 Aug 21 16:57 local_cpulist
-r--r--r--  1 root root    4096 Aug 21 16:57 local_cpus
-r--r--r--  1 root root    4096 Aug 21 16:57 max_link_speed
-r--r--r--  1 root root    4096 Aug 21 16:57 max_link_width
-r--r--r--  1 root root    4096 Aug 21 08:31 modalias
-rw-r--r--  1 root root    4096 Aug 21 16:57 msi_bus
drwxr-xr-x  2 root root       0 Aug 21 16:57 msi_irqs
drwxr-xr-x  3 root root       0 Aug 18 08:38 net
-rw-r--r--  1 root root    4096 Aug 21 08:31 numa_node
drwxr-xr-x  2 root root       0 Aug 21 16:57 power
-r--r--r--  1 root root    4096 Aug 21 16:57 power_state
drwxr-xr-x  3 root root       0 Aug 18 08:39 ptp
--w--w----  1 root root    4096 Aug 21 16:57 remove
--w-------  1 root root    4096 Aug 21 16:57 rescan
--w-------  1 root root    4096 Aug 21 16:57 reset
-rw-r--r--  1 root root    4096 Aug 21 16:57 reset_method
-r--r--r--  1 root root    4096 Aug 21 08:31 resource
-rw-------  1 root root 4194304 Aug 21 16:57 resource0
-rw-------  1 root root 4194304 Aug 21 16:57 resource0_wc
-rw-------  1 root root   16384 Aug 21 16:57 resource4
-rw-------  1 root root   16384 Aug 21 16:57 resource4_wc
-r--r--r--  1 root root    4096 Aug 18 08:39 revision
-rw-------  1 root root  524288 Aug 20 03:14 rom
lrwxrwxrwx  1 root root       0 Aug 20 03:14 subsystem -> ../../../../bus/pci
-r--r--r--  1 root root    4096 Aug 20 03:14 subsystem_device
-r--r--r--  1 root root    4096 Aug 18 08:39 subsystem_vendor
-rw-r--r--  1 root root    4096 Aug 20 03:14 uevent
-r--r--r--  1 root root    4096 Aug 18 08:38 vendor
-rw-------  1 root root       0 Aug 21 16:57 vpd

I don't know why or how the instructions are different, given that they're the exact same Intel X550-T2s... the best reasoning I can think of is that it's somehow driver related.