Older LTS nvidia vGPU host drivers with newer kernel / pve-manager

S

Stephane Chazelas

Member
May 23, 2022
6
0
21
Hello,

from https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Software_Versions I see now that version 595.58.02 from branch 20.0 is available and supported with

pve-managerKernelvGPU Software BranchNVIDIA Host drivers
9.1.87.0.0-1-rc6-pve20.0595.58.02
9.1.86.17.13-2-pve20.0595.58.02

While the previous LTS one (branch 19, 580.126.08) is supported for

pve-managerKernelvGPU Software BranchNVIDIA Host drivers
9.1.46.17.4-1-pve19.4580.126.08

Does that mean that the LTS one (580.126.08) is not supported on newer kernels or versions of pve-manager and that we need to switch to the non-LTS version (20.0) in order to benefit from PVE/kernel updates?
 
Now, same issue again.

https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE#Software_Versions

Has:

pve-managerKernelvGPU Software BranchNVIDIA Host drivers
8.4.16.8.12-11-pve18.3570.158.02
8.4.16.11.11-2-pve18.3570.158.02
8.4.16.14.5-1-bpo12-pve18.3570.158.02
9.2.27.0.2-6-pve19.5580.159.01
9.2.107.0.14-12-pve20.2595.91.04


Does that means that PVE has dropped support for the LTS version (19.6 being the latest on that branch) and we need to move to the non-LTS 20.x branch?
 
hi,

Stephane Chazelas said:
Does that means that PVE has dropped support for the LTS version (19.6 being the latest on that branch) and we need to move to the non-LTS 20.x branch?
Click to expand...
no, the list just represents the versions explicitely tested in our lab

In general newer versions of the driver also work on the tested kernel (minor versions). kernel patch versions are in general also compatible (except some security fixes might need adaption of some interface, but this should be rare)

Really the only problem that can happen is newer minor kernel versions (e.g. a new 7.3) or major nvidia driver versions.

minor kernel + nvidia versions (e.g. 7.0 and 20.x) should all be compatible if tested with one version.
(Sometimes nvidia only updates their driver for newer kernel version later on, so x.0 might not compatible with kernel y.0 but x.1 might)

Overall, newer versions of the driver should never be a problem from the kernel perspective, and newer kernel patch versions (7.0.12 vs 7.0.14 for example) should also not change the compatibilty.

I agree that the wording can be improved on the wiki page, but we did not find a better short explanation for the wiki page yet ;)
 
Thanks a lot for getting back to me on that.

The reason I'm asking is that we've ran with compatibilities issues in the past and would like to minimise the risk of running into similar issues by adopting best practices in that regard.

Specifically, back in May, around the 7.0 kernel release, we had to hold the proxmox-default-headers and proxmox-default-kernel packages as the latest nvidia LTS driver would not compile or run on it.

I understand from the wiki, Proxmox recommend using the latest stable version from nvidia, but we would like to stay on the LTS driver version (19.x ATM) if possible at least for the time being, so would like to clarify whether that's going to be supported by Proxmox going forward or if we'll have to move over to non-LTS versions (latest 20.x ATM).

From your previous comment, I understand we should be fine moving to 19.6 even if that hasn't been tested in Proxmox labs, even with pve-manager 9.2.10 as long as we stay on 7.0.x kernels.

Now if PVE decides to move on to 7.1 or newer and assuming nvidia has not released a 19.x yet compatible with that kernel (like was the case for the move to 6.17 to 7.0), what are our options? Would PVE carry on maintaining a 7.0 kernel branch so we can carry on using 19.x nvidia drivers? Or would PVE defer updating to 7.1 until nvidia LTS drivers are available for it?
 
Complain to NVidia that their driver is not yet compatible with a recent (and common because it's based on Ubuntu) Linux kernel version? It's not like Proxmox runs the very latest Linux kernel version. Proxmox usually supports the previous kernel for a while (especially when it is the default kernel of the previous major version, while it's not EOL). Eventually, you'll have to decide between NVidia LTS and Proxmox kernel security fixes.
 
dcsapak said:
Really the only problem that can happen is newer minor kernel versions (e.g. a new 7.3) or major nvidia driver versions.

minor kernel + nvidia versions (e.g. 7.0 and 20.x) should all be compatible if tested with one version.
(Sometimes nvidia only updates their driver for newer kernel version later on, so x.0 might not compatible with kernel y.0 but x.1 might)

Overall, newer versions of the driver should never be a problem from the kernel perspective, and newer kernel patch versions (7.0.12 vs 7.0.14 for example) should also not change the compatibilty.
Click to expand...
Unfortunately, kernel patch versions after 7.0.14-19-pve are not compatible with even the newest Nvidia driver (615.78.08). Lots of "error: field ‘cxlds’ has incomplete type" when compiling the module.
 
leesteken said:
Only NVidia can update their closed/proprietary driver... I guess you are stuck on outdated and vulnerable kernel versions then.
Click to expand...
Actually Blackwell requires the use of the MIT/GPL module, the proprietary does only support up to Ada. So someone could fix this provisionally.
We'll be moving the GPU to VM pass-through though, to decouple driver incompatibilities such as this one from the used Proxmox kernel.
 
  • Like
Reactions: leesteken
skf10 said:
Unfortunately, kernel patch versions after 7.0.14-19-pve are not compatible with even the newest Nvidia driver (615.78.08). Lots of "error: field ‘cxlds’ has incomplete type" when compiling the module.
Click to expand...
this was a wrong/incomplete backport it seems and was fixed a few days ago: https://git.proxmox.com/?p=pve-kernel.git;a=commit;h=2892a0d2a69c915c398e9e91580e6ad81f11c951

and should be fixed with 7.0.14-26

Stephane Chazelas said:
From your previous comment, I understand we should be fine moving to 19.6 even if that hasn't been tested in Proxmox labs, even with pve-manager 9.2.10 as long as we stay on 7.0.x kernels.
Click to expand...
yes this should be fine

Stephane Chazelas said:
Now if PVE decides to move on to 7.1 or newer and assuming nvidia has not released a 19.x yet compatible with that kernel (like was the case for the move to 6.17 to 7.0), what are our options? Would PVE carry on maintaining a 7.0 kernel branch so we can carry on using 19.x nvidia drivers? Or would PVE defer updating to 7.1 until nvidia LTS drivers are available for it?
Click to expand...
usually we provide some (critical) fixes for a while, but aside from asking nvidia to support newer kernels faster staying on the older kernels is the only option then

skf10 said:
Actually Blackwell requires the use of the MIT/GPL module, the proprietary does only support up to Ada. So someone could fix this provisionally.
We'll be moving the GPU to VM pass-through though, to decouple driver incompatibilities such as this one from the used Proxmox kernel.
Click to expand...
while the kernel part is open source, the vgpu driver is a different beast with some (at least it looks like) proprietary parts that need to be copied in the open driver (when compiling it manually). Also even when someone fixes that, this does not guarantee that the userspace (still proprietary) works with that custom-patched kernel part...
 
You must log in or register to reply here.
Top Bottom
Back