virglrenderer for 3d support

[FGD-Garuda]

Right so pve-qemu-kvm 6.2.0-5 with qemu-server 7.1.5!

root@proxmox:~# qm set 101 --vga virtio-gl
update VM 101: -vga virtio-gl

Sounds good up to that point.

I was wondering if I'd be missing something and yes I am.

Starting the VM (which runs fine without the virtio-gl) fails with

()
MESA-LOADER: failed to open radeonsi: /usr/lib/dri/radeonsi_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
failed to load driver: radeonsi
MESA-LOADER: failed to open kms_swrast: /usr/lib/dri/kms_swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
failed to load driver: kms_swrast
MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
failed to load swrast driver
kvm: egl: gbm_create_device failed
kvm: egl: render node init failed
TASK ERROR: start failed: QEMU exited with code 1

I am using open source video drivers with my AMD iGPU. I am using default amdgpu drivers installed by Proxmox during installation.

Now, Comment #6 in the bugzilla mentions some dependencies with mesa package, not sure if that's where I am having this issue.
If the fix for the above error has been discussed already in this thread here, sorry for my ignorance of not being able to understand what has been discussed, which would explain why I missed it.

Tnx

 

[chchia]

i am not using AMD, but i think you may check if:

1. AMD VGA driver is installed with correctly and latest.
2. installed latest MESA, if required you can try the daily build.
3. make sure your AMD VGA driver did not get blacklisted previously due to trying to direct passthrough to VM.

 

[t.lamprecht]

That's because we don't force the actual openGL/GPU driver to be installed via a hard dependency, that is mainly done due to:

• there sadly isn't a sane open source driver for every GPU
• the free drivers that, e.g., libgl1 pulls in, come with quite a few additional dependencies and lots of Proxmox VE users do not run desktop workload in their VMs, so it'd be unecesarry balast for quite a good chunk of the user base.

You should be able to get you going with: apt install libgl1 as that pulls legacy and modern openGL support in via package dependencies.

 

[FGD-Garuda]

That's because we don't force the actual openGL/GPU driver to be installed via a hard dependency, that is mainly done due to:

• there sadly isn't a sane open source driver for every GPU
• the free drivers that, e.g., libgl1 pulls in, come with quite a few additional dependencies and lots of Proxmox VE users do not run desktop workload in their VMs, so it'd be unecesarry balast for quite a good chunk of the user base.

Make total sense to me.

You should be able to get you going with: apt install libgl1 as that pulls legacy and modern openGL support in via package dependencies.

This installs 30 packages, which again as said before it makes sense they are not installed by default.

However another package is required: apt install libegl1
This will install the required libEGL.so librairies.

And then it BOOTS!!

FIRST IMPRESSIONS
Good ones for starters!
I can confirm openGL works cuz the KDE Desktop Effects that are totally disabled and unavailable when not using openGL are now back and can be enabled if need be and do work.

One thing interesting, it is possible to run Alacritty, which is a terminal app using only the GPU and its memory. This does not work at all under VirtualBox but with Proxmox openGL it does just like on a real machine.

I am now trying to make the VM faster although it's very acceptable already. As of now I am using noVNC and it's still a bit slow but far from without openGL So It's still a few steps further than without and I would not come back to non-openGL.

Unless I cannot make it work with SPICE and/or RDP... noVNC has always been slower than SPICE or RDP for me and limits me within the boundaries of the browser, meaning there is no window auto-resize feature whenever you resize the window using its borders. I have to manually change the display settings within the VM and find one that matches somewhat my browser window size, which is impossible cuz it's between available choices, which anyway it doesn't keep when resizing the window afterwards or when loging out to the SDDM screen. And then there's key bindings, noVNC doesn't grab all keys so when using the Function keys (vastly used with F12 for my drop-down terminal), ALT or CTRL you need to temporarily enable the key from the noVNC GUI (Functions keys I couldn't find at all how to pass them through). Ah well, that's how noVNC has always worked for me, no change here, definitely not the most convenient console for me. But I need to start testing somewhere.

- SPICE is not available in the Proxmox Console button, even after installing the Arch Linux xf86-video-qxl and spice-vdagent packages, which is also what we see here https://www.spice-space.org/download.html. My guess is this is expected at this point.
- RDP, well I am unable to make the Linux QEMU Guest Agent work in order to get the IP but I can get it manually and I can connect RDP to the VM. This though disables the openGL features... which is probably expected at this point too.

I did not find graphical bugs, nothing out of the ordinary though. Very good for a first start!

Next step is to test Windows VM. On this one I am expecting a performance regression, cuz I have to run it through noVNC and Windows doesn't use or leverage acceleration as much as some Linux distros, so I am expecting the noVNC performance impact to be greater than virgl's benefits. It will be interesting to see whether or not my theory stands.

 

[t.lamprecht]

Thanks for your feedback!

- SPICE is not available in the Proxmox Console button, even after installing the Arch Linux xf86-video-qxl and spice-vdagent packages, which is also what we see here https://www.spice-space.org/download.html. My guess is this is expected at this point.

I evaluated it and SPICE with VirGL on bullseye software stack and it works quite well, so it'll be enabled with the next qemu-server bump

Next step is to test Windows VM.

IIRC there are no VirGL VirtIO drivers yet for Windows, but that may be outdated info now (didn't checked for a while).

 

[FGD-Garuda]

You're right Tom, Windows can be configured in a way that's it pretty fast and near real-machine performance so GPU acceleration has less benefits there.

Sounds VERY promising on the SPICE implementation! That will be another huge step fwd.

While it gets bumped, I kept on testing more distros/flavors.

GNOME 42
Since 42 uses GPU acceleration within its native menus it's another good comparison. And there is a performance increase pretty much everywhere. To be frank, that virgl implementation makes Linux desktop-user use cases perfectly usable! Without acceleration you have to use occasionally the VM or be totally insensitive to speed in order to use the VMs for desktop use cases. But now it's perfectly usable everyday for any desktop use.

WAYFIRE
I also tested Wayfire which uses Compiz and Wayland. This one is the absolute best example of what virgl can do. With virgl ON it works pretty good. Slower than non-Wayfire distros, either due to noVNC or the fact it leverages more GPU acceleration which puts more load on virgl I don't know. But with virgl OFF, when it boots it totally fails to load the Greater Deamon and you are stuck without the possibility to use it. So this particular flavor requires 100% virgl, if you ain't got it, you ain't gonna run Wayfire.

 

[speatzle_]

Hi, I ran some tests using Spice on an Nvidia Quadro K4200 Comparing VirGL and QXL.

To get the VM to start using VirGL I had to install libegl1, libglvnd-dev and Nvidia driver Version 470.103.01. For Spice to work I installed the latest pve-test repo and applied the latest Spice related commits manually.

For the VM's i used the Fedora 36 Beta to get Gnome 42 on Wayland.

Generally, both Worked well but:
- QXL Breaks Spice when the lock screen kicks in
- QXL doesn't Auto resize.
- VirGL has Strange Vertical Tearing When moving Windows.

Benchmarks:
- SuperTuxKart won't start a race in QXL (Crashes) In VirGL it Works with about 3 FPS
- glmark2 QXL Score 119 FPS ~ 130 (glmark2 threw lots of GLX errors on QXL) VirGL Score 80 FPS ~100
- glxgears QXL FPS ~300 VirGL FPS ~20

TLDR:
Spice is Very Usable using VirGL and Performance is the Same or Worse than QXL for now but VirGL can run more Applications.

 

[FGD-Garuda]

and applied the latest Spice related commits manually.

Where are those commits located?
The ones in the bugzilla don't seem to specify anything for SPICE.

Also, have you tried benchmarking SPICE virgl vs noVNC virgl?

 

[t.lamprecht]

For the VM's i used the Fedora 36 Beta to get Gnome 42 on Wayland.

Note that virt-viewer/SPICE can sometimes still be quite more performant using X11.

or Worse than QXL

Looks rather wrong IMO and doesn't make much sense to be that way, QXL cannot do 3D acceleration.

I'm getting around 70 FPS with VirGL here on the iGPU of the i7-12700K, while with QXL even getting from the menu into the actual game takes ages and then clocks in at 2 FPS.

 

[speatzle_]

Looks rather wrong IMO and doesn't make much sense to be that way

Since I also thought it was Strange I tried an AMD GPU Today and it seems like this is a Classical fuck you Nvidia Moment (insert Linus Here).

VM Fedora 36 Beta (Wayland)

Test	Standard VGA (Default)	QXL	Nvidia Quadro K4200 VirGL	AMD R7 340 2GB VirGL
SuperTuxKart Spice		Crashes	Runs ~ 3 FPS at half Speed	Runs ~10 FPS in Realtime
SuperTuxKart NoVNC	Crashes			Runs ~10 FPS in Realtime
glxgears Spice		FPS 300	FPS 20	FPS 48
glxgears NoVNC	FPS 1000			FPS 10 (Spikes to 70)
glmark2 Spice		Score 119 FPS ~130	Score 80 FPS ~100	Score 1151 FPS ~2000
glmark2 NoVNC	Score 256 FPS ~300			Score 1067 FPS ~1900
alacritty time tree / Spice		Time 4.9s		Time 14.5s
alacritty time tree / NoVNC	Time 10s	Time 13s		Time 4s

Also, have you tried benchmarking SPICE virgl vs noVNC virgl?

Spice is generally a much better Experience than NoVNC in Every way even if the Numbers say otherwise. I am guessing that the Number difference is due to Spice limiting the Framerate and these aren't really good benchmarks.

Where are those commits located?

In /usr/share/perl5/PVE/QemuServer.pm Replace if ($qxlnum) { with if ($qxlnum || $vga->{type} eq 'virtio-gl') {
And in /usr/share/perl5/PVE/API2/Qemu.pm Replace $status->{spice} = 1 if PVE::QemuServer::vga_conf_has_spice($conf->{vga}); With

if ($conf->{vga}) {
            my $vga = PVE::QemuServer::parse_vga($conf->{vga});
            $status->{spice} = 1
                if $vga->{type} eq 'virtio-gl' || PVE::QemuServer::vga_conf_has_spice($conf->{vga});
        }

https://git.proxmox.com/?p=qemu-ser...;hpb=6f070e39dec2e6d1cc60718cc022b89c9a2b7355
https://git.proxmox.com/?p=qemu-ser...;hpb=463bb05f933fa79e6ae58d56ce2095ad5205d1c4

TLDR 2.0: VirGL is Fastest in Everything Except for Nvidia (might be card/ Driver Specific), also if possible use Spice.

 

[FGD-Garuda]

Love your tests, @speatzle_ !
Tnx for the commits, I would never have found those alone. If they aren't pushed tonight in pvetest I will apply them and try SPICE.
I am assuming SPICE will then show up in the Console drop-down list while still using video = virtio-virgl. I wonder if the amount of vid memory will be limited, as it is with the standard SPICE (no virgl). I think it's SPICE that doesn't go over 128megs or 256.

 

[FGD-Garuda]

Ok I'm getting video memory errors, even when running noVNC, I guess I didn't apply the patches properly so I'll wait for the whole package to bump into pvetest.

 

[FGD-Garuda]

Long time no see.

With June 12th updates (from TEST repo) I continued my tests.

Very promising! And definitely usable better than without 3D acceleration.

SPICE is working as expected with auto-resize feature, both with Xorg and Wayland.

Still under SPICE, Xorg is faster than Wayland, but has occasional glitches (black squares behind icons, large black vertical bars on screen for half a second to a second, etc.). Wayland has very few glitches. All those glitches (Xorg and Wayland) go away while you move mouse around and work within the Linux OS, but new ones appear here and there all the time and fade away and etc.

OS seems to detect 3D acceleration as it should and enables everything you should have with such acceleration.

I did notice that the higher the resolution the laggier it gets. For example, 4K rez lags quite more than 2K and is not recommended to use at this time if you need something with decent speed.

I guess with time all this will improve, but it's definitely on the right track!

 

[janssensm]

TLDR 2.0: VirGL is Fastest in Everything Except for Nvidia (might be card/ Driver Specific), also if possible use Spice.

@speatzle_
Thanks for your earlier testing.
One thing I noticed was that the results with glmark2 had a lot of variation.
Seems to depend on the spice window size. Default glmark2 test on a small spice window results in much higher fps than a large window.

 

[SInisterPisces]

That's because we don't force the actual openGL/GPU driver to be installed via a hard dependency, that is mainly done due to:

• there sadly isn't a sane open source driver for every GPU
• the free drivers that, e.g., libgl1 pulls in, come with quite a few additional dependencies and lots of Proxmox VE users do not run desktop workload in their VMs, so it'd be unecesarry balast for quite a good chunk of the user base.

You should be able to get you going with: apt install libgl1 as that pulls legacy and modern openGL support in via package dependencies.

Thanks for this info. The massive number of dependencies made me realize I definitely wanted to check before I installed anything.

Are there any known downsides to installing these things aside from just generally increased resource usage? I'm concerned about possibly introducing instability to the node.

PC is running a Ryzen 5900HX mobile processor with integrated GPU.

 

[t.lamprecht]

Are there any known downsides to installing these things aside from just generally increased resource usage? I'm concerned about possibly introducing instability to the node.

The answer for that isn't exactly straight forward, as it depends on quite some things, and extra software can have some impact and instability is a way to general word. But, I'd in general say that 1. (most of) the set of software installed is pretty standard on Desktop Linux system, so it's really not like some completely experimental niche software but gets quite some exposure, and 2. if it holds up initial experimentation, you shouldn't expect issues later on, and if any you can always remove it again.

 

[ness1602]

Thomas,when will live-migration for virgl be available?

 

[SInisterPisces]

That's because we don't force the actual openGL/GPU driver to be installed via a hard dependency, that is mainly done due to:

• there sadly isn't a sane open source driver for every GPU
• the free drivers that, e.g., libgl1 pulls in, come with quite a few additional dependencies and lots of Proxmox VE users do not run desktop workload in their VMs, so it'd be unecesarry balast for quite a good chunk of the user base.

You should be able to get you going with: apt install libgl1 as that pulls legacy and modern openGL support in via package dependencies.

I just got VirGL working in one of my VMs for the first time.
The VM went from pegging all 4 vCores at 80+ percent for 2d graphics operations to 38 percent on all cores for the same operations (there's some heavy computation going on there, too.)

I'm thrilled. Thanks for your note on this that gave me the push to try it.
For anyone who finds this later, I had enable Compton in Lubuntu to actually get a usable desktop.

Does it matter if I'm using a machine type that does PCIe or not?
I'm in a SeaBIOS-based, i440fx machine, and i noticed that VirtIO-PCI is the driver being used.

Would I get better performance with a PCIe machine?EDIT: Is there a way to monitor how much of the physical GPU is being used (AMD integrated GPU). My little mini PC really pumps its fans when I put the graphics app in full screen, even inside the noVNC console window in the browser.

 

[janssensm]

Does it matter if I'm using a machine type that does PCIe or not?

Not necessarily , for pcie you have to choose q35. Just try and see what works best.

Is there a way to monitor how much of the physical GPU is being used

Type lspci -k on your host and see what driver you are using, probably amdgpu.
You can setup lm-sensors by installing it: apt install lm-sensors
When you run sensors, normally amdgpu will list some hardware sensors items, f.e. on my system amdgpu-pci-0800
To monitor every second you could run: watch -n 1 sensors amdgpu-pci-0800, replace with your detected item.

And with amdgpu you could also read sysfs, f.e:
cat /sys/kernel/debug/dri/1/amdgpu_pm_info
See for more info:
https://wiki.archlinux.org/title/AMDGPU
Side note: it could be that you need to install pve kernel 5.19 to get these sysfs parameters, I think they were added in later kernels then current pve kernel 5.15.

 

[dcsapak]

Is there a way to monitor how much of the physical GPU is being used (AMD integrated GPU).

you could try 'radeontop'