NVIDIA driver to use virtio-gl

gcobr

New Member
Mar 24, 2024
2
0
1
Hi, All,

My Proxmox host has an NVIDIA Quadro K2200.
I installed NVIDIA drivers with the intention of using VirGL (virtio-gl) with some of my VMs.
I installed the drivers using the binaries provided by NVIDIA themselves: https://www.nvidia.com/download/driverResults.aspx/223426/en-us/

The download is in fact an executable that compiles and then installs the drivers.
Right at the start, it says it is not compatible with the nuveau driver; hence, I added it to /etc/modprobe.d/blacklist.conf:
Code:
blacklist nouveau

After the installation, I have:
Code:
# lspci -k
Code:
03:00.0 VGA compatible controller: NVIDIA Corporation GM107GL [Quadro K2200] (rev a2)
    Subsystem: NVIDIA Corporation GM107GL [Quadro K2200]
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

But then, when I try to start a VM that uses VirGL, I get:

Code:
Header
[...]
swtpm_setup: Not overwriting existing state file.
src/nv_gbm.c:288: GBM-DRV error (nv_gbm_create_device_native): nv_common_gbm_create_device failed (ret=-1)

MESA-LOADER: failed to open nouveau: /usr/lib/dri/nouveau_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
failed to load driver: nouveau
MESA-LOADER: failed to open zink: /usr/lib/dri/zink_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
failed to load driver: zink
MESA-LOADER: failed to open kms_swrast: /usr/lib/dri/kms_swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
failed to load driver: kms_swrast
MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
failed to load swrast driver
kvm: egl: gbm_create_device failed
kvm: egl: render node init failed
stopping swtpm instance (pid 9805) due to QEMU startup error
TASK ERROR: start failed: QEMU exited with code 1

I am running:

Code:
# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.13-3-pve)
pve-manager: 8.1.8 (running version: 8.1.8/d29041d9f87575d0)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5: 6.5.13-3
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph-fuse: 17.2.7-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.3
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.5
libpve-cluster-perl: 8.0.5
libpve-common-perl: 8.1.1
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.6
libpve-network-perl: 0.9.6
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.1.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.5-1
proxmox-backup-file-restore: 3.1.5-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.5
pve-cluster: 8.0.5
pve-container: 5.0.9
pve-docs: 8.1.5
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.1.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve1

Any ideas as to what I am doing wrong?
 
Hey, did you solve it ? I have an NVIDIA Quadro K4200, I installed drivers and set VirGL as display but I don't think the GPU is getting used by the VM, nvidia-smi report 0% usage.
 
@r4lix Does lspci show the card being used by anything?

Also, does dmesg show anything interesting in relation to the card? eg output showing modules being loaded or something
 
I have 2 GPU, one for passthrough the other one for the host

Code:
01:00.0 VGA compatible controller: NVIDIA Corporation TU117GL [T400 4GB] (rev a1)
        Subsystem: Dell TU117GL [T400 4GB]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device: NVIDIA Corporation Device 10fa (rev a1)
        Subsystem: Dell Device 1613
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
05:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K4200] (rev a1)
        Subsystem: NVIDIA Corporation GK104GL [Quadro K4200]
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
05:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
        Subsystem: NVIDIA Corporation GK104 HDMI Audio Controller
        Kernel modules: snd_hda_intel

dmesg show nothing about the gpu.

And nvidia-smi report 0% usage

Code:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.239.06   Driver Version: 470.239.06   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro K4200        Off  | 00000000:05:00.0 Off |                  N/A |
| 35%   47C    P0    26W / 110W |      0MiB /  4037MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
 
I have 2 GPU, one for passthrough the other one for the host

Code:
01:00.0 VGA compatible controller: NVIDIA Corporation TU117GL [T400 4GB] (rev a1)
        Subsystem: Dell TU117GL [T400 4GB]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device: NVIDIA Corporation Device 10fa (rev a1)
        Subsystem: Dell Device 1613
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
05:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K4200] (rev a1)
        Subsystem: NVIDIA Corporation GK104GL [Quadro K4200]
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
05:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
        Subsystem: NVIDIA Corporation GK104 HDMI Audio Controller
        Kernel modules: snd_hda_intel

dmesg show nothing about the gpu.

And nvidia-smi report 0% usage

Code:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.239.06   Driver Version: 470.239.06   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro K4200        Off  | 00000000:05:00.0 Off |                  N/A |
| 35%   47C    P0    26W / 110W |      0MiB /  4037MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
hi, I seem to have same issue. Any progress on this problem?
 
@gcobr The error messages here are probably what's causing the problem.

As part of the loading process for whatever program is generating these errors, the system tried to load these files that it was compiled to depend upon:

Bash:
MESA-LOADER: failed to open nouveau: /usr/lib/dri/nouveau_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
failed to load driver: nouveau
MESA-LOADER: failed to open zink: /usr/lib/dri/zink_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
failed to load driver: zink
MESA-LOADER: failed to open kms_swrast: /usr/lib/dri/kms_swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
failed to load driver: kms_swrast
MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
failed to load swrast driver

It didn't find those files (maybe not installed, or just outright missing?) so the application itself had errors.

At least, that's how most other programs work. Even if it's not using a particular feature, if it can't find some libraries at the load time the whole program just won't run.

That being said, I'm not personally all that familiar with Mesa stuff. Mesa might have its own specialised code to load stuff differently.
 
Last edited:
Nope, I ended up using my iGPU
I had solved my problem.
Before I start vm, find the log like this:1719979035227.png
and in vm, virgl has no effect.
1719979146956.png

After i uninstall my host NVIDIA driver(NVIDIA-Linux-x86_64-550.90.07), then install driver NVIDIA-Linux-x86_64-535.104.05, it finally work!1719981949396.png
 
After i uninstall my host NVIDIA driver(NVIDIA-Linux-x86_64-550.90.07), then install driver NVIDIA-Linux-x86_64-535.104.05, it finally work!
With those driver versions, was the Nvidia 550.90.07 driver directly downloaded from the Nvidia site?

With the 535.104.05 one, is that from the Debian 12 repositories?
 
Ahhh. I wonder if the 535 driver from the Debian 12 repo is better than the one from the website?

They're both 535, so could just be effectively the same. :)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!