VirGL configuration with multiple GPUs

[maskoff]

Hey guys!

I am currently having troubles distributing multiple VirGL VMs over two GPUs, mostly because I was unable to find any documentation for the tool.

As per tests, VirGl is looking for a renderer in /dev/dri/ and uses it for the VM. After installing two cards (that are visible in lspci and nvidia-smi) I've encountered an issue that no matter how many (currently it's 7) VMs I launch, all of these are being allocated to the latest card whilst the first one is chilling with 0% load.

I wish to understand where can I get a better understanding of VirGL internals and maybe some advice or directions on what to take a better look at.

Cheers!

 

[maskoff]

Alright, answering myself. After further digging I got here, line 113. As far as I understand, VirGL is not intended for the use of multiple GPUs at all and is using the first one it finds. The next question is how to properly upgrade the built-in version to the patched one, but that should be straghtforward.

 

[dcsapak]

it should also be possible to give the render node via the '-display' parameter
(we don't do this by default)

we append

-display egl-headless,gl=core

and there one could give

-display egl-headless,rendernode=/dev/dri/renderD....,gl=core

you could try to append this via the 'args' parameter in the config
and feel free to open a feature request here: https://bugzilla.proxmox.com

i could see that this might be interesting for other people too

 

[maskoff]

it should also be possible to give the render node via the '-display' parameter
(we don't do this by default)

we append

-display egl-headless,gl=core

and there one could give

-display egl-headless,rendernode=/dev/dri/renderD....,gl=core

you could try to append this via the 'args' parameter in the config
and feel free to open a feature request here: https://bugzilla.proxmox.com

i could see that this might be interesting for other people too

Hello!

Thanks for the suggestion, I've just tested this approach and it works like a charm. After initial observation I got an idea that virglrenderer exclusively controls the GPU choice and it will need to be patched/upgraded with a balancer, but configurability is great and I wish it was better documented to eliminate this untangling.

I'll create a feature request asap.

I appreciate your assistance.

 

[denishumen]

Can you tell me where to specify these parameters?
Example above
I don't really understand the point, thanks.

-display egl-headless,rendernode=/dev/dri/renderD....,gl=core

 

[dcsapak]

you'd have to put that in the 'args' part e.g. with

qm set ID -args '-display egl-headless....'

 

[maskoff]

Can you tell me where to specify these parameters?
Example above
I don't really understand the point, thanks.

-display egl-headless,rendernode=/dev/dri/renderD....,gl=core

/etc/pve/qemu-server/{ID}.conf

args: -display egl-headless,rendernode=/dev/dri/renderD128,gl=core

As a side note on the matter I'd like to report that there's a certain trick to kvm process when it's used with an NVIDIA GPU. The RAM overhead is overall expected but my observations show that when the VM is launched with an NVIDIA GPU renderer with an NVIDIA driver, the overhead is around 1.5 Gb of RAM per process. Alternatively, when the AMD GPU is used, the same VM will have just around 0.3 Gb extra. It's not something I was able to google, so I'll just leave it here since when these VMs are multiplied the leaked memory becomes a headache.

 

[r4lix]

Hey, I was wondering if you could help me, I have virgl as display but my VM still feels really slow and nvidia-smi report 0% GPU usage.

 

[dcsapak]

Hey, I was wondering if you could help me, I have virgl as display but my VM still feels really slow and nvidia-smi report 0% GPU usage.

please open a new thread with a bit more information in it, like the vm config, pveversion etc.

thanks