Automatically assign UUID to a vGPU instance?

[pixerati]

Hi there,

Nvidia vGPU requires that a host node assign each vGPU instance a UUID, which then needs to be included in a given VM's config.

Is there a way to dynamically assign an "open" UUID.

For example, if I go to spin up a VM on a host server, and that host server already has clients using UUID#1 and UUID#3, can I have the VM's config rewritten before it spins up to use UUID#2, the UUID of the remaining free vGPU instance?

I'm assuming I'd need to script such a thing (which I have no problem with), I just want to make sure there's a mechanism for writing such a script. (Even better if someone's already dealt with this and there's an existing solution.)

Thanks,

Damian Allen

 

[dcsapak]

if you use the gui to select a vgpu model (called 'mediated device') we generate the vgpu on the start (and destroy it on vm stop)
the uuid we generate are 'hostpciindex-0000-0000-0000-vmid'
so the mediated device on vm '100' with a vgpu on hostpci1 will have the uuid: '00000001-0000-0000-0000-000000000100'

but i think the uuid of the vm must not always correspond with the uuid of the vgpu ? (what happens if you assign 2 vgpus to a vm?)

 

[pixerati]

Thanks for the quick response. Actually, these UUIDs are not the same as a VM's main UUID.

Nvidia requires that you create unique UUID's for each "partition" that you slice your GPU into. For example, if you have a 24GB Quadro RTX 6000, you might partition it into 8 x 3GB vGPUs. Each is given a UUID at boot on the hypervisor.

So when you spin up a VM, it needs to be assigned (in the hardware config definition) a UUID.

E.g. in a KVM VM config file:

<device>
...
<hostdev mode='subsystem' type='mdev' model='vfio-pci'>
<source>
<address uuid='uuid'/>
</source>
</hostdev>


My plan is to assign the same 8 x UUIDs on each Compute node (all with identical Nvidia cards), then just before a VM spins up on a given host, the vGPU UUID in its config file is replaced with a one of the 8 UUIDs on that host that's unused.

So my question: is there a way to script this in ProxMox?

I would need to:
1. Keep track of which UUIDs are currently "checked out" on all Compute nodes (obviously this can be done in my code, regardless of ProxMox API).
2. Do a text replace of the config file of a VM about to spin up.
3. Be notified if a VM quits so I can register the corresponding vGPU UUID as now "free."

Hope that makes sense.

 

[dcsapak]

i am not completely sure i understand.. who decides the uuids?
pve can automatically create the mdev vgpus via the kernel interface and there we use the uuids i described earlier
(and we remove it automatically again on vm stop)

maybe you do not need to do all that manually, but simply select the correct mediated device type on the passthrough edit window?

otherwise, you'd need to check how we attach such a gpu on the qemu commandline (assign a mdev and call 'qm showcmd ID --pretty')
and add it yourself via the 'args' parameter of vms (check 'man qm' or 'man cm.conf' for details)

 

[pixerati]

Thanks for being so responsive Dominik. Already liking the idea of signing on for enterprise support (we're currently evaluating ProxMox).

The UUIDs are decided by the user, but they're set on the host hypervisor, so to automate spinning up a new VM, I either need to change the UUID on the VM to match the one on the host, or change the one on the host to match the VM.

In a high availability situation I need to be able to spin up a VM on an arbitrary host and make sure it isn't using the same UUID vGPU as one of the other VMs already on that host.

Is there a simple "preflight" callback or something that I can use to issue the qemu commands? I haven't dug into the API at all yet, so I don't know its limits...

 

[dcsapak]

i don't think you fully understood my earlier message:

PVE can and will automatically create a mediated device (vGPU) if you configure it to do (pci passhtrough edit window in the gui -> select the correct gpu -> select the mdev type)
this will create a cluster unique uuid for that vgpu device (like i explained above)

where do you need to set the uuid manually then ?

 

[pixerati]

Yes, it sounds like I am missing the point

But to confirm: are we both talking about partitioned virtualization (sharing a single card between multiple VMs) or direct passthrough of a single GPU to a single VM? (I'm talking about the former, but I'm guessing that's what the "mediated" refers to anyway.)

So you're saying that PVE automatically assigns a UUID instance on the host OS and then assigns it to the provisioned VM profile? If so, then that is fantastic. Also, does it then "garbage collect" the instance when the VM powers off?

Forgive me if it's front and center in the docs and I missed it...

 

[dcsapak]

yes i am talking about nvidias vGPU (i believe it was called GRID at some point?) where you have multiple vgpus on one real gpu. this is called 'mediated devices' in the kernel (intel has that feature too on some iGPUs, they call it kvmgt)

So you're saying that PVE automatically assigns a UUID instance on the host OS and then assigns it to the provisioned VM profile? If so, then that is fantastic. Also, does it then "garbage collect" the instance when the VM powers off?

Forgive me if it's front and center in the docs and I missed it...

yes, you can read about it in https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_mediated_devices_vgpu_gvt_g

 

[pixerati]

Brilliant, thanks. And thanks again for being so patient with my ramblings; if this is what the community support looks like, I'm looking forward to enterprise support once we ramp up our data center