DELL R7625 with an EPYC CPU and an A2 card no mdevctl types

[mle3]

Hello,
I have a DELL R7625 with an EPYC CPU and an A2 card.
I ran proxmox 8.2.4 with Kernel 6.8.8-2 and managed to install NVIDIA-Linux-x86_64-550.90.05-vgpu-kvm.run

Installation with https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE
without errors

Problem:
This results in mdevctl types being empty and no mdev folders have been created under /sys/bus/pci/devices/

This don't me allow to add the vGPU on VM's


root@pve2:~# nvidia-smi
Thu Jul 11 16:55:26 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.05 Driver Version: 550.90.05 CUDA Version: N/A |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A2 On | 00000000:21:00.0 Off | 0 |
| 0% 43C P8 8W / 60W | 0MiB / 15356MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

root@pve2:~# dmesg | grep -e IOMMU -e AMD-Vi
[ 0.358275] AMD-Vi: Using global IVHD EFR:0x25bf732fa2295afe, EFR2:0x1d
[ 0.964277] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.967932] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.971888] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.976541] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.979412] pci 0000:e0:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.981814] pci 0000:c0:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.984203] pci 0000:80:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.986690] pci 0000:a0:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.989177] AMD-Vi: Extended features (0x25bf732fa2295afe, 0x1d): PPR X2APIC NX GT [5] IA GA PC GA_vAPIC
[ 0.989187] AMD-Vi: Interrupt remapping enabled
[ 0.989188] AMD-Vi: X2APIC enabled
[ 0.990415] AMD-Vi: Virtual APIC enabled
[ 0.992592] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 0.992599] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[ 0.992606] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[ 0.992613] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[ 0.992620] perf/amd_iommu: Detected AMD IOMMU #4 (2 banks, 4 counters/bank).
[ 0.992627] perf/amd_iommu: Detected AMD IOMMU #5 (2 banks, 4 counters/bank).
[ 0.992633] perf/amd_iommu: Detected AMD IOMMU #6 (2 banks, 4 counters/bank).
[ 0.992640] perf/amd_iommu: Detected AMD IOMMU #7 (2 banks, 4 counters/bank).

root@pve2:~# lsmod | grep -i nvidia
nvidia_vgpu_vfio 114688 10
nvidia 54288384 3
mdev 24576 1 nvidia_vgpu_vfio
kvm 1372160 2 kvm_amd,nvidia_vgpu_vfio
vfio_pci_core 86016 2 nvidia_vgpu_vfio,vfio_pci
irqbypass 12288 3 vfio_pci_core,nvidia_vgpu_vfio,kvm
vfio 69632 4 vfio_pci_core,nvidia_vgpu_vfio,vfio_iommu_type1,vfio_pci

root@pve2:~# /usr/lib/nvidia/sriov-manage -e 0000:21:00.0 ALL
GPU at 0000:21:00.0 already has VFs enabled.

 

[INFOLOGEEK]

Hello,

I have the same graphic card A2 ! and the latest 17.3 nvidia_vgpu works only with the kernel 6.5 .
The nvidia vgpu driver is not yet compatible with kernel 6.8, downgrade to kernel 6.5 .

Remove Nvidia vgpu driver 
./NVIDIA-Linux-x86_64-550.90.05-vgpu-kvm.run --uninstall

Downgrade kernel
pve-efiboot-tool kernel list
pve-efiboot-tool kernel pin kernel_you_want

reboot en reinstall the driver

mdevctl types will no longer be empty !!!!