Upgrade to 8.1 broke all GPU passthrough

[caymann]

I upgraded today to 8,1, everything went well, rebotted and now lost nvidia GPU passthrough
This seems like a common theme in every proxmox upgrade.

#nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

 

[LnxBil]

This seems like a common theme in every proxmox upgrade.

In every Linux kernel update, yes. Therefore I gave up on pcie passthrough. I was also sick of it.

 

[fabian]

the upgrade to 8.1 brought a new major kernel release by default (6.5). some third party or proprietary driver modules might need to be updated, but that is outside of our control.

 

[cjghvieira]

I upgraded today to 8,1, everything went well, rebotted and now lost nvidia GPU passthrough
This seems like a common theme in every proxmox upgrade.

#nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Hi. Did you try to update nvidia drivers to the newest version?
Normaly I have to do that in every kernel update.

 

[Ivan Dimitrov]

I am not sure what will be the relevance of the kernel driver for the PCIe passthrough? I have all my passed devices added to the ignore list as recommended by the passthrough guide.
Anyway for me GTX1660 Supper and Quadro A4000 worked without changes after the 8.1 upgrade.

 

[failurefile]

I have noticed that my igpu passthrough is no working after upgrading as well. Is there native support for igpu passthroughs?

 

[briansbits]

Hello. I'm in a similar situation, however I'm new to using Proxmox VE. Installed my 3 node cluster only a few weeks ago using the 8.1 installer, no upgrade. No pre-existing GPU passthrough configured. I'm just now trying to get GPU passthrough working so I can do hardware transcoding with Jellyfin.

Prior to Proxmox VE I was using ESXi, and was able to get GPU passthrough working relatively easily.

Here's what I did so far.

On the PVE Node:

1. On the node with the GPU installed in it I followed the "PCI Passthrough" documentation. Made sure the drivers were blacklisted on the node.
2. Created a new VM. UEFI BIOS. PCI Passthrough raw device and selected the GPU. Checked the "All functions" and "PCI Express" check boxes.

In the VM I performed the following tasks

1. Installed Debian 12 Bookworm in the VM
2. lspci command shows the nVidia GPU
3. Installed 'nvidia-detect' and ran it. It see's my GPU and recommends installing the 'nvidia-driver' package
4. Installed the 'nvidia-driver' package. Gave me a warning about the nouveau driver causing a conflict and a restart should fix it.
5. Rebooted the VM after the driver was installed
6. Ran 'nvidia-smi' and I received the error "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."

Not sure what I should be doing at this point. Any advice or assistance would be appreciated.

Thank you.

 

[Docop2]

the gfx is detected, this is a wrong setting in your linux vm. Boot over a live arch or popos and they have native nvidia pre-installed. Quadro lineup do run perfectly over pass. And as new version bring new setting.. a server is not a phone to update every 3 min.

 

[LnxBil]

Prior to Proxmox VE I was using ESXi, and was able to get GPU passthrough working relatively easily.

You did it also very easiy. The problem you have looks like a guest problem.

Ran 'nvidia-smi' and I received the error "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."

Did you load the nvidia driver manually and test again?

 

[briansbits]

Solved the issue.

It had to do with secure boot being enabled on the VMs BIOS. Disabling Secure Boot in the BIOS allowed the driver to be loaded.

No where in the documentation is this mentioned. Might be nice to include that for others in the future.

 

[LnxBil]

It had to do with secure boot being enabled on the VMs BIOS. Disabling Secure Boot in the BIOS allowed the driver to be loaded.

Oh yeah, that could be a problem.

Compiling your own kernel modules is unfortunately still VERY complicated with secure boot enabled. I think there has been a discussion about enrolling your own crypto keys into the stock keyring for secureboot a few months/years back to be able to have a proper secured setup.

 

[fabian]

it's actually rather streamlined nowadays if you use the (default) shim-based setup. at least on Debian-based distros, dkms will generate a key that you need to import/enroll once, and all dkms-built modules will be signed with that and trusted by the kernel.

 

[LnxBil]

it's actually rather streamlined nowadays if you use the (default) shim-based setup. at least on Debian-based distros, dkms will generate a key that you need to import/enroll once, and all dkms-built modules will be signed with that and trusted by the kernel.

Yes, but this approach is not sufficient if you have a lot of machines and need to use a module on all of them and want to provide deb-packages for it. You need to enroll your own keys in order the distribute pre-compiled kernel modules.