GPU passthrough to Container (LXC)
- Thread starter pkr
- Start date
-
- Tags
- container gpu lxc ubuntu 20.04
there is no detailed guide in the wiki AFAIK, but there is some guide e.g. for jellyfin: https://jellyfin.org/docs/general/administration/hardware-acceleration/intel#lxc-on-proxmox
basically you have to install the correct nvidia driver on the host and then follow the guide
(the things there are not only specific to jellyfin, but rather generally how to bind mount the gpu device nodes into the container)
basically you have to install the correct nvidia driver on the host and then follow the guide
(the things there are not only specific to jellyfin, but rather generally how to bind mount the gpu device nodes into the container)
I support my lab colleagues running CUDA workloads on Docker. They want Docker and CUDA, I want to also run VMs.
On oVirt/RHV I solve that with passing the V100 GPUs through to one or two VMs, which then run the Nvidia augmented Docker for their CI-CD workloads.
A previous attempt to run Docker + CUDA side-by-side with oVirt/RHV failed because Docker and oVirt were righting over the network.
Likewise a previous attempt to run CUDA inside an OpenVZ container failed because the CUDA runtime is looking for certain devices files and OpenVZ is hiding them from containers. I tried patching the OpenVZ kernel drivers but without source code to the CUDA runtime, that effort seemed too much to pursue.
I am a bit curious if LXC works with CUDA out of the box, but not enough to try it on my proxmox test machines, where I have gone with passing the GPU to a VM and toggling forth and back is rather involved.
And the user demand is Docker, ...which should work inside LXC, but might not want to. And the chances of LXC+Docker+CUDA working without messing with KVM seem almost remote.
If you can get it to work, I'd like to hear how it went.
But I can assure you that CUDA on a VM with GPU pass-through works extremely easy also on Proxmox or KVM in general and might be a better approach ...once you get the pass-through GPU working.
I dislike "fat" VM virtualization when containers should do the job already. But when CUDA workloads suffer next to no VM overhead because of pass-through, pragmatism wins.
On oVirt/RHV I solve that with passing the V100 GPUs through to one or two VMs, which then run the Nvidia augmented Docker for their CI-CD workloads.
A previous attempt to run Docker + CUDA side-by-side with oVirt/RHV failed because Docker and oVirt were righting over the network.
Likewise a previous attempt to run CUDA inside an OpenVZ container failed because the CUDA runtime is looking for certain devices files and OpenVZ is hiding them from containers. I tried patching the OpenVZ kernel drivers but without source code to the CUDA runtime, that effort seemed too much to pursue.
I am a bit curious if LXC works with CUDA out of the box, but not enough to try it on my proxmox test machines, where I have gone with passing the GPU to a VM and toggling forth and back is rather involved.
And the user demand is Docker, ...which should work inside LXC, but might not want to. And the chances of LXC+Docker+CUDA working without messing with KVM seem almost remote.
If you can get it to work, I'd like to hear how it went.
But I can assure you that CUDA on a VM with GPU pass-through works extremely easy also on Proxmox or KVM in general and might be a better approach ...once you get the pass-through GPU working.
I dislike "fat" VM virtualization when containers should do the job already. But when CUDA workloads suffer next to no VM overhead because of pass-through, pragmatism wins.
Has anyone had luck actually installing the "proper" NVIDIA drivers on the host?
I.e. On Proxmox 8 I download the appropriate .run file (for my Quadro P600) and the installation aborts, and tells me to use the Debian package manager to install instead. This is even after blacklisting nouveau, and 'apt remove --purge libnvcuvid1 libnvidia-encode1' (the two packages specified in the Jellyfin install instructions for Debian systems)
Trying later to just use those packages does not work. I end up with only /dev/nvram, and not the other 5 items in the /dev directory shown in most tuturials.
I.e. On Proxmox 8 I download the appropriate .run file (for my Quadro P600) and the installation aborts, and tells me to use the Debian package manager to install instead. This is even after blacklisting nouveau, and 'apt remove --purge libnvcuvid1 libnvidia-encode1' (the two packages specified in the Jellyfin install instructions for Debian systems)
Trying later to just use those packages does not work. I end up with only /dev/nvram, and not the other 5 items in the /dev directory shown in most tuturials.
The biggest problem with luck is that it's not reliable.
Proxmox and Nvidia software are constantly evolving and even if you should be so lucky as to get things to work, any package update might possibly undo things and without mainline support for the combination your luck might run out.
I've fallen to the same temptation often enough, e.g. tried to force CUDA to run on OpenVZ/Virtuozzo and found myself rewriting kernel code to offset what I'd still label 'CUDA runtime stupidities'... I ran out of time before it got it to work, but at least I learned something.
Containers are designed to share better, what was designed to be shared already.
Computers weren't designed to be shared initially, not even memory with data and code before John von Neumann's memo spread the idea, but in the case of storage, CPUs, memory, and even networks (which I'd argue have never been 'natural' in Unix) resources became much easier to share thanks to replacing (slow) human operators by (fast) software operating systems and design retrofits such as mapping (storage), resumable traps (CPU), memory mapping and virtualization (RAM), and 'better' networking.
But GPUs have changed their functionality and mission so rapidly over the last decades, that proper architecture, hardware abstractions, software and hardware co-design haven't quite caught up and I'm afraid they'll only ever will, once GPUs slow down their evolution and become boring.
And one such aspect IMHO can be seen in the fact that many want to virtualize the GPU for transcoding... a VPU functionality that, until streaming and remote gaming became a thing, wound up with the GPUs more or less by accident or marketing pushing ideas over thought.
VPU engines are fixed function blocks and evidently GPUs have them in varying numbers, relatively unrelated to the 'size' of a GPU. Sharing those should be pretty much like sharing a set of printers (limited time exclusive round-robin), and even if Microsoft still struggles with it, that hasn't been rocket science for a long time.
Except that on x86 VPUs are tied to the GPUs for said accident and lack of proper abstractions, both in hardware and in software.
But it's much worse with the proper GPU (graphics, not video), which just isn't designed to be shared: at best is has partition support.
So putting it under the exclusive control of a dedicated VM may not just be far easier now that at least PCIe pass-through mechanics have been devised and established, it's the better way of dealing with a resource not designed for sharing.
IMHO keeping it out of Proxmox' hair, is just a more efficient way of applying everyones' resources until xPUs, chipsets and operating systems have caught up.
I had the same issue as you. I received the message about using the Debian package manager to install instead. To fix this, I did apt remove purge nvidia*
You have to remove all nvidia packages before running the installer.
I've found this amazing solution as well: https://sluijsjes.nl/2024/05/18/cor...to-install-frigate-video-surveillance-server/
This made my live easy and my Frigate NVR is now blazing fast!
This made my live easy and my Frigate NVR is now blazing fast!
Found an absolutely great tutorial on how to do this. You can find the webpage here https://digitalspaceport.com/proxmox-lxc-gpu-passthru-setup-guide/https://digitalspaceport.com/proxmox-lxc-gpu-passthru-setup-guide/ and it has a corresponding YouTube video (at the top of that page) which walks you through it.
The only thing not mentioned in the video or the tutorial to keep in mind is that you need to turn off UEFI Secure Boot options unless you have a solution for signing an NVIDIA driver.
The only thing not mentioned in the video or the tutorial to keep in mind is that you need to turn off UEFI Secure Boot options unless you have a solution for signing an NVIDIA driver.
I've hit the same problem, and looking for a clean way to do this. I need to pass the gpu through to lxc, then to docker. I know this isn't the recommended setup, but it can work, and its the least performance hit and easiest maintenance, in my opinion.
Both of these tutorials do work and do exactly that:
- https://sluijsjes.nl/2024/05/18/cor...to-install-frigate-video-surveillance-server/
- https://digitalspaceport.com/proxmox-lxc-gpu-passthru-setup-guide/
However, this requires manually installing the drivers on the host and all lxc containers that need the gpu. I need to pass this to more than one lxc. Maintaining this will be a nightmare, especially since I (and both of those tutorials) installed it manually, not through apt. I will probably never even update, out of fear of it falling apart.
Does anyone know of a verified method that would only have the drivers on the host, and everything is passed through lxc containers?
I've tried deleting the drivers from the lxc, and adding passthroughs like this (in different combinations of .so files):
I can post more details if someone if willing to have a look, but i was wondering is there a working tutorial out there that I missed. I've spent hours on this, and abusing chatgpt to somehow solve it, but failed in all attempts. Mostly, it can't find
Side note: in both of these tutorials the mapping can be simplified, to map the devices through the webui, without doing manual mount entries, cgroups etc. At least it works for me, provided that I install the no-kernel driver on the lxc (same as the tutorials do).
Both of these tutorials do work and do exactly that:
- https://sluijsjes.nl/2024/05/18/cor...to-install-frigate-video-surveillance-server/
- https://digitalspaceport.com/proxmox-lxc-gpu-passthru-setup-guide/
However, this requires manually installing the drivers on the host and all lxc containers that need the gpu. I need to pass this to more than one lxc. Maintaining this will be a nightmare, especially since I (and both of those tutorials) installed it manually, not through apt. I will probably never even update, out of fear of it falling apart.
Does anyone know of a verified method that would only have the drivers on the host, and everything is passed through lxc containers?
I've tried deleting the drivers from the lxc, and adding passthroughs like this (in different combinations of .so files):
Code:
lxc.mount.entry: /usr/bin/nvidia-smi usr/bin/nvidia-smi none bind,optional,create=file
lxc.mount.entry: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so usr/lib/x86_64-linux-gnu/libnvidia-ml.so none bind,optional,create=file
lxc.mount.entry: /usr/lib/x86_64-linux-gnu/libcuda.so usr/lib/x86_64-linux-gnu/libcuda.so none bind,optional,create=file
lxc.environment: LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnuI can post more details if someone if willing to have a look, but i was wondering is there a working tutorial out there that I missed. I've spent hours on this, and abusing chatgpt to somehow solve it, but failed in all attempts. Mostly, it can't find
libnvidia-ml.so event though its mounted.Side note: in both of these tutorials the mapping can be simplified, to map the devices through the webui, without doing manual mount entries, cgroups etc. At least it works for me, provided that I install the no-kernel driver on the lxc (same as the tutorials do).
Code:
dev0: /dev/nvidia0
dev1: /dev/nvidiactl
dev2: /dev/nvidia-uvm
dev3: /dev/nvidia-uvm-tools
dev4: /dev/nvidia-caps/nvidia-cap1
dev5: /dev/nvidia-caps/nvidia-cap2
Last edited: