nVIDIA vGPU mdev setup not working (as per wiki)

[proxwolfe]

Hi,

I am trying to set up a NVIDIA A5000 as vGPU as per this wiki article: https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE_7.x

Got to the point where I enabled SRIOV and can list the virtual functions via lspci.

But then it says under guest configuration that I should pick an mdev in the GUI but for me the selection menu remains empty. (My GUI differentiates between a mapped device and a raw device. the mdev selection menu is shown right to the mapped device radio button (but, as said, remains empty). Under the selection menu of the raw device, I now see a number of virtual devices but - unlike in the wiki - without any information about their individual configuration (like ram, max. instances etc.).

Is this behaviour normal (because the GUI has changed compared to the wiki) or have I misconfigured something and should change anything to get to the point the wiki describes?

Thanks!

 

[proxwolfe]

So I just tried to pass one of the vGPU devices to a VM but I get this error message:

TASK ERROR: Cannot bind 0000:02:00.0 to vfio

Interestingly, after this error, mdevctl types returns nothing. Before the error, it returns a list of mdevs.

So something is off. But what?

 

[torxed]

I've bumped into this issue too.
In a previous version of proxmox the mdev type dropdown showed all available mdev supported devices as per https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Guest_Configuration

# cat /sys/bus/pci/devices/0000\:c7\:00.4/mdev_supported_types/nvidia-710/name
NVIDIA A16-2B
# cat /sys/bus/pci/devices/0000\:c7\:00.4/mdev_supported_types/nvidia-710/description
num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=8

I'd love to submit a fix for this if it's a lesser used feature, but trying to find the handling of this as a beginner to the project is a bit tricky which is fine. But the dropdown appears to be here: https://github.com/proxmox/pve-mana...228d003/www/manager6/form/MDevSelector.js#L80
And calling: https://proxmox:8006/api2/json/nodes/node-01/hardware/pci/0000:c7:00.4/mdev

And that returns

{
    "data": [
        {
            "description": "num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=8\n",
            "available": 0,
            "name": "NVIDIA A16-2Q",
            "type": "nvidia-712"
        },
        {
            "available": 0,
            "description": "num_heads=1, frl_config=60, framebuffer=16384M, max_resolution=1280x1024, max_instance=1\n",
            "type": "nvidia-720",
            "name": "NVIDIA A16-16A"
        },

It's a convenience factor for sure, but editing /etc/pve/qemu-server/<machine-id>.conf is fine too until I figure this one out.
I'll also note that

# mdevctl types --dumpjson

works perfectly fine too, all the different mdev types show up fine in CLI.

I think I started figuring out the code, although Pearl and I are at best crisis buddies.
But the /mdev endpoint appears to be located here: https://github.com/proxmox/pve-mana...e28228d003/PVE/API2/Hardware/PCI.pm#L172-L174

And it calls: https://github.com/proxmox/pve-comm...0d526a3da4a7d1cf52/src/PVE/SysFSTools.pm#L155

Which then used somewhere here (and there's probably a few steps in between that I haven't found/poked on yet):
And the JavaScript in the frontend is assuming: https://github.com/proxmox/pve-mana...4e8504bec02a/www/manager6/qemu/PCIEdit.js#L63

Where `pciDev.data.mdev` won't exist because the pciDev.data is actually:

{
    "checks": [],
    "id": "A16-VGPU",
    "digest": "7ffa7e4a371321d77",
    "map": [
        "id=10de:25b6,iommugroup=133,node=node-01,path=0000:c7:00.4,subsystem-id=10de:0000"
    ],
    "type": "pci"
}

And that appears to be because this is never called: https://github.com/proxmox/pve-mana...dee/www/manager6/form/MDevSelector.js#L76-L83

Which in turn is because me.pciid isn't set here: https://github.com/proxmox/pve-mana...dee/www/manager6/form/MDevSelector.js#L94-L96

 

[torxed]

This is by no means perfect, but at least it gets the dropdown working again on pve-manager/8.1.4/ec5affc9e41f1d79:
pvemanagerlib.js

--- pvemanagerlib.js.orig    2024-03-19 23:20:55.108008038 +0100
+++ pvemanagerlib.js    2024-03-19 23:25:43.843702312 +0100
@@ -6436,21 +6436,44 @@
         type: 'proxmox',
         url: '/api2/json/nodes/' + me.nodename + '/hardware/pci/' + me.pciid + '/mdev',
     });
-    me.store.load();
+    me.store.load({
+                callback: (recs, op, success) => me.processResponse(recs, op, success)
+        });
+    },
+
+    processResponse: function(records, _op, success) {
+        let me = this;
+        if (!success) {
+            return;
+        }
+
+        let store = me.getStore();
+
+        let recs = []
+        records.forEach((rec) => {
+            recs.push(rec)
+        });
+
+        me.setDisabled(false)
+
+        me.suspendEvent('change');
+        me.setSelection();
+        me.setSelection(recs);
+        me.resumeEvent('change');
     },

     initComponent: function() {
-    var me = this;
+        var me = this;

-    if (!me.nodename) {
-        throw 'no node name specified';
-    }
+        if (!me.nodename) {
+            throw 'no node name specified';
+        }

         me.callParent();

-    if (me.pciid) {
-        me.setPciID(me.pciid, true);
-    }
+        if (me.pciid) {
+            me.setPciID(me.pciid, true);
+        }
     },
 });

@@ -32472,7 +32495,6 @@
         return;
         }

-
         let deviceId = `${device.vendor}:${device.device}`.replace(/0x/g, '');
         let subId = `${device.subsystem_vendor}:${device.subsystem_device}`.replace(/0x/g, '');

@@ -50112,6 +50134,8 @@
         }
         }

+            mdevfield.setPciID(path);
+
         if (pciDev.data.mdev) {
         mdevfield.setPciID(path);
         }

 

[torxed]

Below is a picture of the patch above in action, and a bugzilla bug has been submitted: https://bugzilla.proxmox.com/show_bug.cgi?id=5321
First two images describes the issue, third picture is the patched version.