[SOLVED] GPU passthrough and WSLg

[Kaman]

Hi,

I am running PVE 7.3 on a Dell Preceision T5820. Created a Windows 11 VM and passthrough an NVidia RTX 3060 to it. The latest NVidia drivers are installed on the VM and pretty much everything working great, except the WSLg.

In the VM, I installed the default Ubuntu distribution. But none of the GUI apps are working. I mean, I can run xterm, for example, which supposed would bring up a new xterm window, but the window never come up. I tried other guild apps all the same behaviour. The same VM, if I remove the GPU passthrough, the gui apps will work. This issue happens regardless if I am remote desktop to it or through the monitor (passthrough).

Is there some configuration I need to change to make gui app work with the GPU passthrough?

Thanks,

Kaman

 

[noel.]

Hi,

Have you had a look at GPU acceleration in WSL yet?

If that doesn't help, you might have to look at windows error logs in Event Viewer under Windows Logs->System

Also, I have no idea what your setup is or anything, but wouldn't it be easier to set up a linux vm in proxmox alongside the windows wm and just switch between the two? you can send any files from one machine to another via ssh.

Hope this helps

 

[Kaman]

Thanks for the reply and sorry for the delay.

Can you repost the link? It doesn't seem to work.

I did a bit more testing using different drivers and configurations (thank goodness for VM snapshots) and found that the only configuration where the GUI apps will work is by installing the STANDARD nVidia drivers with ONLY the drivers, meaning no PhysX, no HD audio and GeForce Experiece options. But nVidia has switched to DCH drivers since over a year ago so I don't really want to be using the old drivers. Besides, even though the GUI apps works with the standard drivers, but I don't know if GPU acceleration is being used (how do I find that?).

I also installed Windows 11 directly (removing Proxmox) and the GUI apps are working, even using the latest DCH drivers. So it seems to be with VM?

This is how I use Windows if it is installed on a dedicated PC. Setting up Proxmox is for fun so I want to retain the same features if possible. I am actually running a MacOS VM as my primary PC. My intention is to get Windows 11 running in VM with dedicated GPU passthrough and all the features I use as in my dedicated PC. And if that is successful, I will retire the dedicated PC.

 

[noel.]

Thanks for the reply and sorry for the delay.

No worries

Can you repost the link? It doesn't seem to work.

Yeah I posted a wrong url, should be fixed now. This link might be useful as well as well as the github page of wslq.

Besides, even though the GUI apps works with the standard drivers, but I don't know if GPU acceleration is being used (how do I find that?).

nvidia-smi in your wsl install gives you all your gpu info (temps, how much vram is being used, wattage, etc) as well as which programms are running. You can also run watch -n 1 nvidia-smi to refresh the info once a second.

(PS: glxinfo -B is the semi-equivalent command for openGL info)

Best of luck with your endeavours. CUDA can be a pain in the ass to get work correctly but with a little patience and some vm snapshots/restores it should be doable.

 

[Kaman]

Cool. Thanks. I've been to those links already and still reading. I actually posted my question in github to see if anyone has any idea. Based on the way the gui apps not returning and without errors, it seems they are sending the result to another display server. But I've checked my $DISPLAY and it is set to :0, which is supposed to be correct.

Yeah, nvidia-smi shows me the GPU info, in both windows and wsl. It is just any other gui apps will not return as if they are working but the wayland/x server isn't show them on my screen.

Here is the nvidia-smi info and glxinfo -B:


Notice glxinfo doesn't return any results, I had to ctrl-c to kill it. The first part of nvidia-smi I can understand but not the Process part of it. Does it look right?

 

[noel.]

I'll have to get back to you with this next week, in case no one helps you until then.

 

[Kaman]

I believe it is solved. Reading through question 10 of this link: http://vfio.blogspot.com/2014/08/vfiovga-faq.html, it darn on me that, maybe nVidia didn't completely fix the issue and took a stab at it by adding:
args: -cpu host,kvm=off

to the configuration file and boom, it worked. GUI apps are working, running the TensorFlow-DirectML samples works.

With that change, you can use the latest nVidia drivers with all options installed: PhysX, GeForce Experience, HD Audio.

How do I mark this thread solved?

 

[noel.]

Just tested it on my machine, adding args: -cpu host,kvm=off to the .conf file of the vm also worked for me. However, as the post mentions, this will lead to lower performance.
For me cuda worked in wsl without having to pass the kvm=off option. So depending on what you need wsl for, if it is exclusively tensorflow, it might give better performance to not pass kvm=off and just live without a gui?
Alternatively, if you want to be even faster and bypass all virtualization, you could also install a Desktop environment (such as gnome or xfce) on your proxmox instance (this works perfectly fine as proxmox is just a debian fork), run all your tensorflow stuff in proxmox and then start your windows vm and share the tensorflow data between proxmox and the windows vm (wg with a zfs pool or an nfs share or something). But this might not be what you want.

Glad you found a solution for your problem and I hope the setup works well for you

 

[Kaman]

Very much appreciated that you are still following up on this.

For me, without kvm=off, cuda didn't work until I install the cudnn in my wsl distro following this link:
https://forums.developer.nvidia.com...ont-detect-my-cuda-enabled-gpu-in-wsl2/157880

I never thought of installing a Desktop on the Proxmox instance. It is great to know and I might try that sometime. Again it is all for fun. For years prior to my discovery of bare-metal virtualization (I did use VirtualBox before), I only run Windows on my PC. Now that I found it, and able to have both MacOS and Windows with dedicated GPU, I don't think I can go back to a plain vanilla PC anymore.

Thanks again for your help.

 

[Kaman]

Update: I think I found a solution without having to turn off KVM. instead of kvm=off, replace it with -hypervisor. So the new line for args is as follow:

args: -cpu host,-hypervisor

I had used this option before when installing WSL in a VM, in a different computer with different CPU. Without that option, WSL will not even install. However for my current computer none of the Windows VMs I created requires this option and WSL will install, albeit without GUI apps working. So just today I decided to try this option and boom, WSLg is working with GUI apps.

I think I understand what that -hypervisor flag is for, but what I don't understand is, why WSL in a VM can be installed (ignoring the GUI apps) in some PCs and not in others. But I think I will just included it for all my Windows VMs from now on, for all CPUs that supports it.

Anyway, hope this helps to prevent someone making the same mistakes I made.