[SOLVED] vGPU just stopped working randomly (solution includes 6.14, pascal fixes for 17.5, changing mock p4 to A5500 thanks to GreenDam )

the line 376 in lib.rs is the only one you should have to change, my guess the second number doesnt align or you need to manually restart the service
this can be done with systemctl restart nvidia-vgpu-mgr.service i have had ot use this to get mdevctl types to work before, which if thats the case you usually have to use it after every reboot to get it to work again, you could try making an issue on github too, to see if the person who edited it has a fix
 
Looks like I figured it out.

I uninstalled the 17.5 driver and vgpu-unlock-rs.

I rebooted to start 'clean'.

Installed 17.6 driver NVIDIA-Linux-x86_64-550.163.02-vgpu-kvm-custom.run after patching it with 550.163.02.patch.

Installed vgpu-unlock-rs from https://github.com/mbilker/vgpu_unlock-rs.

Rebooted, and viola, have P40 profiles :)

Code:
 nvidia-smi
Wed Nov 12 08:12:24 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.163.02             Driver Version: 550.163.02     CUDA Version: N/A      |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA TITAN Xp                On  |   00000000:03:00.0 Off |                  N/A |
| 23%   31C    P8             18W /  250W |      47MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro P6000                   On  |   00000000:81:00.0 Off |                  Off |
| 26%   43C    P8             19W /  250W |   24434MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    1   N/A  N/A    128883    C+G   vgpu                                         8126MiB |
|    1   N/A  N/A    129032    C+G   vgpu                                         8126MiB |
|    1   N/A  N/A    129049    C+G   vgpu                                         8126MiB |
+-----------------------------------------------------------------------------------------+
 
that is good to hear you got it working, also for these new drivers you do not want p4/p40 profiles anymore if you meant p40 and not t40, you need the t4/t40 or the newer drivers wont install. p4/p40 is limited to 535 drivers as the maximum supported version.
 
Yeah, couldn't install the 17.6 client drivers, so I installed the 16.12 client drivers and they work fine. I doubt newer drivers will bring any improvements to these old cards... The 17.6 driver on proxmox installed fine, only problem is the client driver.

I might try and be advetureous and see if I can get the unlocker to fake my Pascal cards into a V100 so that newer drivers can be installed as the guy suggested in the repo:

https://github.com/mbilker/vgpu_unl...632c327e683d642ecaa93#commitcomment-170322337
 
Yeah, couldn't install the 17.6 client drivers, so I installed the 16.12 client drivers and they work fine. I doubt newer drivers will bring any improvements to these old cards... The 17.6 driver on proxmox installed fine, only problem is the client driver.

I might try and be advetureous and see if I can get the unlocker to fake my Pascal cards into a V100 so that newer drivers can be installed as the guy suggested in the repo:

https://github.com/mbilker/vgpu_unl...632c327e683d642ecaa93#commitcomment-170322337
it is good to change it to something that supports the newer drivers, in my experience too linux can get pretty picky of client drivers do not match the host exactly. lol

they do bring some improvements, since optimizations and all work on any cards, as do new features, if you do anything with AI, etc, the system fallback setting is in the new drivers but not on 535, as well as many improvements for newer games, ai, etc, the only reason the old cards do not work is because nvidia changed a few lines of code to remove them from the supported devices lists to phase them out. nothing is stopping them from doing most the things newer cards do. pre-pascal are kind of pretty bad but pascal and newer are all pretty comparable besides the tensor cores and built in ai features, newer cards are just of course much faster at the same advanced compute tasks and often have more vram. as long as you don't mind it being slower then your card can do everything a newer card can do that doesn't specifically use tensor cores.
 
  • Like
Reactions: shpitz461
I have been able to install the nvidia-grdd guest driver in rhel10 VM after lots of reading and troubleshooting. The xserver issue was easy as rhel10 does not even have xserver or xorg stuff (it was removed in rhel10) but the the hardcoded .run installer was just complaining as false alarm. I also patched the driver using instructions at gridd-unlock-patcher. Downloaed the client token on client VM using the following command

Code:
wget --no-check-certificate -O /etc/nvidia/ClientConfigToken/client_configuration_token_$(date '+%d-%m-%Y-%H-%M-%S').tok https://10.10.1.135/-/client-token


Now the only issue remains is licencing. Running
Code:
systemctl status nvidia-gridd.service
gives the following erros:

Sep 09 22:14:20 localhost.localdomain nvidia-gridd[3148]: vGPU Software package (0)
Sep 09 22:14:20 localhost.localdomain nvidia-gridd[3148]: Ignore service provider and node-locked licensing
Sep 09 22:14:20 localhost.localdomain nvidia-gridd[3148]: NLS initialized
Sep 09 22:14:20 localhost.localdomain nvidia-gridd[3148]: Acquiring license. (Info: 10.10.1.135; NVIDIA Virtual Applications)
Sep 09 22:14:22 localhost.localdomain nvidia-gridd[3148]: Mismatch between client and server with respect to licenses held. Returning the licenses
Sep 09 22:14:22 localhost.localdomain nvidia-gridd[3148]: License returned successfully. (Info: 10.10.1.135)
Sep 09 22:14:22 localhost.localdomain nvidia-gridd[3148]: Failed to verify signature (error:02000068:rsa routines::bad signature)
Sep 09 22:14:22 localhost.localdomain nvidia-gridd[3148]: Failed to verify signature on lease response
Sep 09 22:14:22 localhost.localdomain nvidia-gridd[3148]: Failed to validate lease response
Sep 09 22:14:22 localhost.localdomain nvidia-gridd[3148]: Failed to acquire license from 10.10.1.135


I have checked and confirmed that the time zone on both the fastapi license server and client VM is identical. Fastapi-DLS server is running on an lxc container (Ubuntu 24.04). In fact I also recreated the webserver.crt and webserver.key and then patched the original nvidia-gridd file again.

What could be wrong. After so much of effort still not working.
HI, how exactly do you run the gridd-unlocker-patcher? which file should I download?
 
I'm trying to use my Quadro P6000 on 7.0.14-28-pve for AI.

I installed openwebui using community-script, but even though nvidia-smi shows the gpu in the LXC, ollama uses the cpu and the gpu shows no use.

vGPU works for me with GRID v19.6 (driver v580.178.05).

Was also able to patch the license using https://git.collinwebdesigns.de/vgpu/gridd-unlock-patcher

Any idea how to get Ollama LXC to use the P6000?

Thanks!
 
I'm trying to use my Quadro P6000 on 7.0.14-28-pve for AI.

I installed openwebui using community-script, but even though nvidia-smi shows the gpu in the LXC, ollama uses the cpu and the gpu shows no use.

vGPU works for me with GRID v19.6 (driver v580.178.05).

Was also able to patch the license using https://git.collinwebdesigns.de/vgpu/gridd-unlock-patcher

Any idea how to get Ollama LXC to use the P6000?

Thanks!
unfortunately, the vGPU driver lacks Cuda and other functionality necessary for LXCs to properly utilize the cards. so while its possible with tinkering to make the LXC see the card, there is nothing there to use.

so currently the only way is to

A: use a vm with the vGPU driver so you can use split memory, etc

B: use the standard workstation non kvm driver and forego vGPU functionality so it can work in LXCs.

its potentially possible to merge them (the standard driver and the kvm.), i got that half working once but it is complicated and in my tinkering, doesnt work right at all as the KVM driver is just not designed to work that way.
 
  • Like
Reactions: shpitz461
Ah, so running it separately in a VM is the way to go then, I'll give that a shot, good call!
yeah, that seems to be the best option for the moment, that is how i am running mine.

hugepages helps by the way, since ram bandwidth plays a role, unless your models fit purely in vram anyway.
 
it's odd, when I add the gpu to the archlinux VM (i440fx), it borks the proxmox console video vnc, but the VM does run.
Also does it when i add a VF from the Mellanox card.
 
Still doesn't work, ollama reverts to using 100% cpu.
I see this in journalctl:
Code:
Nov 06 20:54:27 ollama ollama[12662]: time=2026-11-06T20:54:27.254-05:00 level=INFO source=llama_server.go:295 msg="skipping CUDA device — compute capability not in compiled architectures" device="Quadro P6000" cc=610 archs="[750 800 860 870 880 890 900 1000 1030 1100 1200 1210]" libDirs="[/usr/lib/ollama /usr/lib/ollama/cuda_v13]"
Is it the CUDA version?
 
Got it to work, uninstalled everything then reinstalled ollama using official script:

curl -fsSL https://ollama.com/install.sh | sh

Code:
ollama ps
NAME           ID              SIZE      PROCESSOR    CONTEXT    RUNNER      UNTIL
gemma3:270m    e7d36fb2c3b3    414 MB    100% GPU     32768      llamacpp    4 minutes from now

Code:
nvidia-smi
Fri Nov  6 21:46:12 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.196.07             Driver Version: 580.196.07     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro P6000                   Off |   00000000:00:10.0 Off |                  Off |
| 26%   46C    P8             11W /  250W |     661MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            1212      C   ...local/lib/ollama/llama-server        658MiB |
+-----------------------------------------------------------------------------------------+