Just done exactly that for a customer ;)
Not tested yet, bit nvidia-smi sees the card inside the container, so i think it will work
additional lxc-config entries:
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 234:* rwm
lxc.cgroup2.devices.allow: c 509:* rwm...