[SOLVED] LXC Nvidia Passthrough

T

t3dc0x

New Member
Jun 23, 2023
8
2
3
Hi there,

Proxmox 8.0.4 / Kernel 6.2.16-6-pve / Debian 12 (host and container)

I had this working for a while, but it's suddenly stopped.

I've got two different lxcs which use the gpu (P400). I'm noticing that both lxcs are now running into a "permission denied" error accessing the hardware. In the container, the relevant /dev/dri entries are all owned by nobody:nogroup. Interestingly, nvidia-smi works in both containers even though emby and stable diffusion both get permission denied trying to access the hardware.

I'm guessing the problem is related to device ownership / uid/gid mapping. So, in this case, I'm trying to map video (44) and render (104)

Once I setup mapping, I run into this error:

newgidmap failed to write mapping "newgidmap: gid range [44-45) -> [44-45) not allowed": newgidmap 245834 44 44 1
Click to expand...

I've read and attempted to follow several tutorials on proper id mapping without making any forward progress. All the iterations produce exactly the same error. At this point, I'm assuming it's necessary to map the full range of gids even though I only care about 44 and 104. Is that correct?

Any help would be greatly appreciated! Thank you.

/etc/subuid:
Code: 
root:100000:65536

/etc/subgid:
Code: 
root:100000:65536
root:44:1
root:104:1

lxc conf:
Code: 
arch: amd64
cores: 2
features: nesting=1
hostname: emby
memory: 2048
mp0: /storage/Media,mp=/storage/Media
net0: name=eth0,bridge=vmbr0,firewall=1,gw=XXX.XXX.XXX.XXX,hwaddr=XX:XX:XX:XX:XX:XX,ip=XXX.XXX.XXX.XXX/24,type=veth
ostype: debian
rootfs: local-lvm:vm-101-disk-0,size=32G
swap: 2048
tags: media
unprivileged: 1
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 59
lxc.idmap: g 104 106 1
lxc.idmap: g 105 65430
lxc.mount.entry: /dev/dri/card1 dev/dri/card1 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD129 dev/dri/renderD129 none bind,optional,create=file
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 509:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file

Log snippet:
lxc-start 101 20230809101519.488 INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWNS
lxc-start 101 20230809101519.488 INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWPID
lxc-start 101 20230809101519.488 INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWUTS
lxc-start 101 20230809101519.488 INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWIPC
lxc-start 101 20230809101519.488 INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWCGROUP
lxc-start 101 20230809101519.488 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved user namespace via fd 17 and stashed path as user:/proc/245816/fd/17
lxc-start 101 20230809101519.488 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 18 and stashed path as mnt:/proc/245816/fd/18
lxc-start 101 20230809101519.488 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 19 and stashed path as pid:/proc/245816/fd/19
lxc-start 101 20230809101519.488 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 20 and stashed path as uts:/proc/245816/fd/20
lxc-start 101 20230809101519.488 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 21 and stashed path as ipc:/proc/245816/fd/21
lxc-start 101 20230809101519.488 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 22 and stashed path as cgroup:/proc/245816/fd/22
lxc-start 101 20230809101519.488 DEBUG conf - ../src/lxc/conf.c:idmaptool_on_path_and_privileged:3549 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start 101 20230809101519.488 DEBUG conf - ../src/lxc/conf.c:idmaptool_on_path_and_privileged:3549 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start 101 20230809101519.488 DEBUG conf - ../src/lxc/conf.c:lxc_map_ids:3634 - Functional newuidmap and newgidmap binary found
lxc-start 101 20230809101519.490 ERROR conf - ../src/lxc/conf.c:lxc_map_ids:3701 - newgidmap failed to write mapping "newgidmap: gid range [44-45) -> [44-45) not allowed": newgidmap 245834 44 44 1
lxc-start 101 20230809101519.490 ERROR start - ../src/lxc/start.c:lxc_spawn:1788 - Failed to set up id mapping.
lxc-start 101 20230809101519.490 DEBUG network - ../src/lxc/network.c:lxc_delete_network:4173 - Deleted network devices
lxc-start 101 20230809101519.490 ERROR start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "101"
Click to expand...
 
Thank you @t3dc0x

The script is exactly what I needed to fix Ollama problems with the GPU - despite the correct nvidia-smi output, it wasn't using my NVIDIA cards and the below fixes this issue. I also moved the script into the /etc/init.d/ folder so it runs on every boot.

Code: 
#!/bin/bash
## Script to initialize nvidia device nodes.
## https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#runfile-verifications

/sbin/modprobe nvidia

if [ "$?" -eq 0 ]; then
  # Count the number of NVIDIA controllers found.
  NVDEVS=`lspci | grep -i NVIDIA`
  N3D=`echo "$NVDEVS" | grep "3D controller" | wc -l`
  NVGA=`echo "$NVDEVS" | grep "VGA compatible controller" | wc -l`

  N=`expr $N3D + $NVGA - 1`
  for i in `seq 0 $N`; do
    mknod -m 666 /dev/nvidia$i c 195 $i
  done

  mknod -m 666 /dev/nvidiactl c 195 255

else
  exit 1
fi

/sbin/modprobe nvidia-uvm

if [ "$?" -eq 0 ]; then
  # Find out the major device number used by the nvidia-uvm driver
  D=`grep nvidia-uvm /proc/devices | awk '{print $1}'`

  mknod -m 666 /dev/nvidia-uvm c $D 0
  mknod -m 666 /dev/nvidia-uvm-tools c $D 0
else
  exit 1
fi
 
You must log in or register to reply here.
Top Bottom