Nvidia VGPU - No virtual devices

V

vertigo8

New Member
Mar 3, 2024
2
0
1
Hi - I am having issues seeing virtual vgpu devices under lspci. I just see the default number. This is my configuration:

  1. Latest version - environment 8.4.1 with kernel 6.8.12-10-pve
  2. I have a Quadro RTX 5000 running with patched vgpu-kvm drivers, version 550.90.05. Patching was done with the guide by polloloco (https://gitlab.com/polloloco/vgpu-proxmox)
  3. nvidia-smi shows expected output
  4. nvidia-smi vgpu shows expected output
  5. mdevctl types shows expected output
  6. Nouveau driver has been blacklisted
  7. Iommu confirmed as working
Here are some command/config outputs. I should be seeing more devices for NVIDIA surely?

Thanks in advance

Code: 
root@pve:~# lspci | grep NVIDIA
51:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 5000] (rev a1)
51:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
51:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
51:00.3 Serial bus controller: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

Code: 
root@pve:~# dmesg | grep -e DMAR -e IOMMU
[    0.022829] ACPI: DMAR 0x0000000067569000 0001BC (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
[    0.022876] ACPI: Reserving DMAR table memory at [mem 0x67569000-0x675691bb]
[    0.222177] DMAR: IOMMU enabled
[    0.557801] DMAR: Host address width 46
[    0.557802] DMAR: DRHD base: 0x000000bb7fc000 flags: 0x0
[    0.557813] DMAR: dmar0: reg_base_addr bb7fc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[    0.557817] DMAR: DRHD base: 0x000000d0ffc000 flags: 0x0
[    0.557825] DMAR: dmar1: reg_base_addr d0ffc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[    0.557827] DMAR: DRHD base: 0x000000e67fc000 flags: 0x0
[    0.557831] DMAR: dmar2: reg_base_addr e67fc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[    0.557833] DMAR: DRHD base: 0x000000fb7fc000 flags: 0x0
[    0.557839] DMAR: dmar3: reg_base_addr fb7fc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[    0.557841] DMAR: DRHD base: 0x000000a5ffc000 flags: 0x1
[    0.557844] DMAR: dmar4: reg_base_addr a5ffc000 ver 4:0 cap 8ed008c40780466 ecap 60000f050df
[    0.557846] DMAR: RMRR base: 0x0000006b985000 end: 0x0000006b9a8fff
[    0.557848] DMAR: RMRR base: 0x0000006a297000 end: 0x0000006a4e0fff
[    0.557850] DMAR: ATSR flags: 0x0
[    0.557852] DMAR: RHSA base: 0x000000a5ffc000 proximity domain: 0x0
[    0.557854] DMAR: RHSA base: 0x000000bb7fc000 proximity domain: 0x0
[    0.557855] DMAR: RHSA base: 0x000000d0ffc000 proximity domain: 0x0
[    0.557856] DMAR: RHSA base: 0x000000e67fc000 proximity domain: 0x0
[    0.557857] DMAR: RHSA base: 0x000000fb7fc000 proximity domain: 0x0
[    0.557859] DMAR-IR: IOAPIC id 8 under DRHD base  0xa5ffc000 IOMMU 4
[    0.557861] DMAR-IR: HPET id 0 under DRHD base 0xa5ffc000
[    0.557863] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.559587] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.738167] DMAR: No SATC found
[    1.738170] DMAR: dmar3: Using Queued invalidation
[    1.738174] DMAR: dmar2: Using Queued invalidation
[    1.738176] DMAR: dmar1: Using Queued invalidation
[    1.738180] DMAR: dmar0: Using Queued invalidation
[    1.738185] DMAR: dmar4: Using Queued invalidation
[    1.743229] DMAR: Intel(R) Virtualization Technology for Directed I/O

Code: 
root@pve:~# nvidia-smi
Sat May 10 21:12:42 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.05              Driver Version: 550.90.05      CUDA Version: N/A      |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 5000                On  |   00000000:51:00.0 Off |                  Off |
| 34%   41C    P8             17W /  230W |      99MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                        
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Code: 
root@pve:~# nvidia-smi vgpu
Sat May 10 21:13:05 2025       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.05              Driver Version: 550.90.05                 |
|---------------------------------+------------------------------+------------+
| GPU  Name                       | Bus-Id                       | GPU-Util   |
|      vGPU ID     Name           | VM ID     VM Name            | vGPU-Util  |
|=================================+==============================+============|
|   0  Quadro RTX 5000            | 00000000:51:00.0             |   0%       |
+---------------------------------+------------------------------+------------+

Code: 
root@pve:~# mdevctl types
0000:51:00.0
  nvidia-256
    Available instances: 24
    Device API: vfio-pci
    Name: GRID RTX6000-1Q
    Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-257
    Available instances: 12
    Device API: vfio-pci
    Name: GRID RTX6000-2Q
    Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
  nvidia-258
    Available instances: 8
    Device API: vfio-pci
    Name: GRID RTX6000-3Q
    Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
  nvidia-259
    Available instances: 6
    Device API: vfio-pci
    Name: GRID RTX6000-4Q
    Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
  nvidia-260
    Available instances: 4
    Device API: vfio-pci
    Name: GRID RTX6000-6Q
    Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
  nvidia-261
    Available instances: 3
    Device API: vfio-pci
    Name: GRID RTX6000-8Q
    Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
  nvidia-262
    Available instances: 2
    Device API: vfio-pci
    Name: GRID RTX6000-12Q
    Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
  nvidia-263
    Available instances: 1
    Device API: vfio-pci
    Name: GRID RTX6000-24Q
    Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
  nvidia-435
    Available instances: 24
    Device API: vfio-pci
    Name: GRID RTX6000-1B
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-436
    Available instances: 12
    Device API: vfio-pci
    Name: GRID RTX6000-2B
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-437
    Available instances: 24
    Device API: vfio-pci
    Name: GRID RTX6000-1A
    Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
  nvidia-438
    Available instances: 12
    Device API: vfio-pci
    Name: GRID RTX6000-2A
    Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
  nvidia-439
    Available instances: 8
    Device API: vfio-pci
    Name: GRID RTX6000-3A
    Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
  nvidia-440
    Available instances: 6
    Device API: vfio-pci
    Name: GRID RTX6000-4A
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
  nvidia-441
    Available instances: 4
    Device API: vfio-pci
    Name: GRID RTX6000-6A
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
  nvidia-442
    Available instances: 3
    Device API: vfio-pci
    Name: GRID RTX6000-8A
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
  nvidia-443
    Available instances: 2
    Device API: vfio-pci
    Name: GRID RTX6000-12A
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
  nvidia-444
    Available instances: 1
    Device API: vfio-pci
    Name: GRID RTX6000-24A
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
 
mdev devices are different than pci functions, so you can't see them with lspci, but you can see them with "mdevctl list"

you simply need to pass the mdev (nvidia-xxx) in the pci passthrough gui 1746890344426.png
 
Last edited:
You must log in or register to reply here.
Top Bottom