Proxmox 8.1 +Secure Boot + A40 (Supported Nvidia vGPU)

[deepcloud]

Hi Folks,

Got Proxmox 8.1 Running with Secure Boot and CEPH, added an Nvidia A40 GPU, this supports vGPU

was following the below guide - i dont need vgpu-unlock as its a supported card.
https://gitlab.com/polloloco/vgpu-proxmox

iommu is enabled - verified by below command
--------------------------------------------------------------------------------------------
dmesg | grep -e DMAR -e IOMMU
[ 8.463826] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported
[ 8.467158] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[ 8.470901] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported
[ 8.474805] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 8.482426] pci 0000:e0:00.2: AMD-Vi: IOMMU performance counters supported
[ 8.485921] pci 0000:c0:00.2: AMD-Vi: IOMMU performance counters supported
[ 8.489575] pci 0000:a0:00.2: AMD-Vi: IOMMU performance counters supported
[ 8.492256] pci 0000:80:00.2: AMD-Vi: IOMMU performance counters supported
[ 8.495073] pci 0000:60:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 8.495101] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 8.495124] pci 0000:20:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 8.495145] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 8.495167] pci 0000:e0:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 8.495188] pci 0000:c0:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 8.495210] pci 0000:a0:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 8.495231] pci 0000:80:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 8.503694] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 8.503775] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[ 8.503853] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[ 8.503926] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[ 8.504001] perf/amd_iommu: Detected AMD IOMMU #4 (2 banks, 4 counters/bank).
[ 8.504076] perf/amd_iommu: Detected AMD IOMMU #5 (2 banks, 4 counters/bank).
[ 8.504161] perf/amd_iommu: Detected AMD IOMMU #6 (2 banks, 4 counters/bank).
[ 8.504230] perf/amd_iommu: Detected AMD IOMMU #7 (2 banks, 4 counters/bank).

-----------------------------------------------------------------------------------------------------------------------------

I successfully installed the driver with the below command

./NVIDIA-Linux-x86_64-535.154.02-vgpu-kvm.run --module-signing-public-key=vgpu_debian_publickey.pub --dkms

vgpu_debian_publickey.pub is from the nvidia driver (Signing_Keys folder inside the zip file NVIDIA-GRID-Linux-KVM-535.154.02-535.154.05-538.15)

after reboot

I am unable to get output from nvidia-smi

# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

------------------------------------------------------------------------------------------------------------------------------
if i run dmesg - the last line says

Loading of module with unavailable key is rejected

which means that its a issue with the nvidia driver not getting signed properly and hence not getting loaded properly

any ideas

 

[dcsapak]

did you already see this part of the wiki? https://pve.proxmox.com/wiki/Secure_Boot_Setup#DKMS https://pve.proxmox.com/wiki/Secure_Boot_Setup#DKMS_2

 

[deepcloud]

the driver install process is

 

[dcsapak]

yes, as the guid i linked mentions and the nvidia instructions too, the kernel module is signed, but you have to enroll the keys in to the kernel so it know to trust those
you can do this with mokutil as written in the linked guide

 

[deepcloud]

Sorry for my late reply, i was busy with something else, I am fully confused seeing this post. In my understanidng the post is to sign the boot files during upgrade from 8 to 8.1.

Can somebody give me the list of commands to be run, I don't want to break a working cluster as it's a 3 node proxmox 8.1 cluster with ceph

 

[deepcloud]

like i need to do ---

mokutil --import /usr/share/nvidia/nvidia-modsign-crt-DAE5E38B.der
or
mokutil --import /var/lib/dkms/mok.pub

The guide is not nvidia specific but generic from a dkms signing perspective, something more specific would be more helpful

 

[oxbx08]

like i need to do ---

mokutil --import /usr/share/nvidia/nvidia-modsign-crt-DAE5E38B.der
or
mokutil --import /var/lib/dkms/mok.pub

The guide is not nvidia specific but generic from a dkms signing perspective, something more specific would be more helpful

I'm trying to do the same thing as you and just got mine working with a Tesla p4. I followed these instructions, https://askubuntu.com/questions/1023036/how-to-install-nvidia-driver-with-secure-boot-enabled, but you should be able to use your existing key based on your prior screenshots of the Nvidia install process.

The ask Ubuntu forum also gives this link, but it was very useful to understand the screens you need to follow once you run the mokutil command and set the password. https://sourceware.org/systemtap/wiki/SecureBoot

 

[deepcloud]

Got it working, thanks for the help folks