[TUTORIAL] NVIDIA drivers instalation Proxmox and CT

[bialykostek]

Hi! I post step by step tutorial how to install NVIDIA drivers on Promox server. I hope it will be usefull for someone:

1. Blacklist nouveau:
vi /etc/modprobe.d/blacklist-nouveau.conf

2. Paste, save and quit:
blacklist nouveau
options nouveau modeset=0

3. Update initramfs:
update-initramfs -u

4. Check if nouveau is enabled:
lsmod | grep nouveau

5. Disable nouveau and verify:
rmmod nouveau
lsmod | grep nouveau

6. Ensure GPU is visible:
lspci | grep NVIDIA

7. Download driver (check for most recent version compatible with nvidia-utils-xxx-server):
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.90.07/NVIDIA-Linux-x86_64-550.90.07.run
chmod +x NVIDIA-Linux-x86_64-550.90.07.run

8.Install build packages:
apt install build-essential pve-headers-$(uname -r)

9. Run instalation:
./NVIDIA-Linux-x86_64-550.90.07.run

10. Check if instalation was successfull:
nvidia-smi

11. [optional] Turn on persistane mode if necessary (lowers IDLE power consumption):
https://docs.nvidia.com/deploy/driver-persistence/index.html
nvidia-smi --persistence-mode=1 #only for current session
nvidia-persistenced

12. After creating CT shut it down and edit lxc configuration file (location might be diffrent):
vi /etc/pve/nodes/pve/lxc/10001.conf

13. Paste, save and quit. If you have more than one GPU, change /dev/nvidia0 /dev/nvidia0 to /dev/nvidia<GPU ID> /dev/nvidia<GPU ID>:
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 243:* rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file

14. Start your CT and install (on the CT):
apt install nvidia-utils-550-server

15. Verify installation (on the CT)
nvidia-smi

 

[SoMoney]

using the NVIDIA run scripts has been hit/miss with me.
When I get to step7, I much rather goto NVIDIA's cuda page https://developer.nvidia.com/cuda-downloads, and select nvidia/cuda drivers for the OS I'm building.

 

[tirk]

Thanks for the contribution. I've been trying to install the NVIDIA GPU in Proxmox for a while now, but without success. I'm stuck on step 9, which isn't working. I'm probably doing something wrong, but I'm stuck.

 

[bialykostek]

using the NVIDIA run scripts has been hit/miss with me.
When I get to step7, I much rather goto NVIDIA's cuda page https://developer.nvidia.com/cuda-downloads, and select nvidia/cuda drivers for the OS I'm building.

Sure, that was the most recent at the time. Currently I'm using 550.127.05 on all my machines, but you have to be careful as for example 550.127.08 didn't support RTX devices.

 

[bialykostek]

As mentioned in previous comment, due to lack of compability for all devices, currently I'm using 550.127.05 everywhere. It is tricky because there was a problem with libnvidia-compute dependancy, so I'm just using local files instead. I have gathered all necessary files here:

https://drive.google.com/drive/folders/1GaiN_2FC1HJCYGAdqiE3CeX5Vu41Qofi?usp=drive_link

To update instructions, in step 7 you don't need to download file, just use the one from the drive. Then instead of step 14 run:

dpkg -i libnvidia-compute-550-server_550.127.05-0ubuntu0.22.04.1_amd64.deb
dpkg -i nvidia-utils-550-server_550.127.05-0ubuntu0.22.04.1_amd64.deb

Everything should work just fine

 

[bialykostek]

Thanks for the contribution. I've been trying to install the NVIDIA GPU in Proxmox for a while now, but without success. I'm stuck on step 9, which isn't working. I'm probably doing something wrong, but I'm stuck.

Please tell me you GPU model and what error occurs, I'll try to help. You may also try drivers from my previous reply.

 

[tirk]

I'm trying to install the NVIDIA TESLA P4 GPU in proxmox but I can't get it to work, I've installed the drivers
wget https://es.download.nvidia.com/tesl...local-repo-debian12-570.86.15_1.0-1_amd64.deb

I'm stuck, I don't know what to do to get the VMs to detect it and work

 

[bialykostek]

I'm trying to install the NVIDIA TESLA P4 GPU in proxmox but I can't get it to work, I've installed the drivers
wget https://es.download.nvidia.com/tesl...local-repo-debian12-570.86.15_1.0-1_amd64.deb

I'm stuck, I don't know what to do to get the VMs to detect it and work

Wait, if you are trying to make it work in VM, I'm not sure if that's possible. This steps works on CT (lxc) only

 

[tirk]

https://gitlab.com/polloloco/vgpu-proxmox

 

[tirk]

Step 1: Edit GRUB Execute: nano /etc/default/grub Change this line from GRUB_CMDLINE_LINUX_DEFAULT="quiet" to GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff" Save file and exit the text editor
Step 2: Update GRUB Execute the command: update-grub
Step 3: Edit the module files Execute: nano /etc/modules Add these lines: vfio vfio_iommu_type1 vfio_pci vfio_virqfd Save file and exit the text editor
Step 4: IOMMU remapping a) Execute: nano /etc/modprobe.d/iommu_unsafe_interrupts.conf Add this line: options vfio_iommu_type1 allow_unsafe_interrupts=1 Save file and exit the text editor b) Execute: nano /etc/modprobe.d/kvm.conf Add this line: options kvm ignore_msrs=1 Save file and exit the text editor
Step 5: Blacklist the GPU drivers Execute: nano /etc/modprobe.d/blacklist.conf Add these lines: blacklist radeon blacklist nouveau blacklist nvidia blacklist nvidiafb Save file and exit the text editor
Step 6: Adding GPU to VFIO a) Execute: lspci -v Look for your GPU and take note of the first set of numbers b) Execute: lspci -n -s (PCI card address) This command gives you the GPU vendors number. c) Execute: nano /etc/modprobe.d/vfio.conf Add this line with your GPU number and Audio number: options vfio-pci ids=(GPU number,Audio number) disable_vga=1 Save file and exit the text editor
Step 7: Command to update everything and Restart a) Execute: update-initramfs -u b) Then restart the your Proxmox Node