Docker in Unpriviledged LXC or Dedicated VM?

[kaivalagi]

Hi all, first post from me. After much internal debate and poc'ing where I can in hyperv between truenas 24.10 with docker versus proxmox with truenas vm and docker lxc, I went with the latter option on the grounds of better virtualisation and cluster support.

Currently I have my truenas vm using disks via hba in passthrough, with nfs shares made available through autofs mounts on the host, then bind mount those host folders in the lxc for use by my docker stacks, I used a privileged container to avoid all the uid/gid mappings for now. I also need igpu access for jellyfin etc.

Rather than having the user mapping can of worms with an unprivileged container I'm thinking of moving my docker stacks to a vm and having autofs setup within that for truenas access, better isolation and easier to manage that way leaving the host to be a host only I am a little concerned with dmesg activity around eth0 renaming on the host from the lxc proxy containers I have which is another reason I am looking at vm isolation.

I would normally go with debian but resource usage is obviously way more than the lxc container setup I have, any distros worth considering purely for this function instead ti reduce footprint? The lxc way is soooo lightweight it's spoilt me!

Alternatively is it worth biting the bullet and going unprivileged with lxc and sorting mappings out? I have all my docker containers running as either root for network stuff or as an apps user/group. I would also need to figure out the proxy stuff....I'm just thinking the vm approach simplifies it all at the cost of an extra core and a few gb of memory, something I can live with for now I guess.

Thanks for any insight

 

[LnxBil]

For best isolation, use QEMU/KVM VMs for Docker, as it is stated in the documentation. I recommend to use the OS of your choice for the Docker host and there are small Docker-centric distributions available, yet I don't know if you want to add another distribution to your list of supported distributions. I like to stick to one distribution to keep the complexity down.

 

[Johannes S]

truenas scale 24.10 has native docker support and this is what I would use. Up to now they only released the RC2, but the stable release is expected in the next weeks and I havn't experienced any issues with the RC.
Alternatively create a vm with your favourite Linux distribution and run docker/portainer from it.
Don't use docker inside an lxc, this isn't really supported and might break at any time (especially due to updates). Don't use any "helper scripts" either, if you don't understand, how they are working. It's never a good idea to deploy things, without understanding them and will lead to problems sooner or later.

 

[LnxBil]

Don't use any "helper scripts" either, if you don't understand, how they are working. It's never a good idea to deploy things, without understanding them and will lead to problems sooner or later.

This cannot be stressed enough! I love tutorials starting with

curl xyz | sudo bash

 

[Johannes S]

This cannot be stressed enough! I love tutorials starting with

curl xyz | sudo bash

Sadly this is even true for official documentation of some projects (not Proxmox!). And I'm doing a kubernetes introduction course right now whose "labs" mainly consist of copy/pasting of certain commands, downloading containers from a (hopefully trustworthy) source etc pp.
I'm glad I use some cloud instance for it (through Googles free trial offering) so if something goes wrong nothing important will be lost.

 

[kaivalagi]

As another option to explore I've also added in a spare disk to proxmox and will try messing with a file server setup using lxc or vm as a replacement for truenas fronting zfs pools instead of disks, but that will need some solutions for share setup, monitoring/notifying for zfs health (maybe grafana in lxc) and cloud backups (rclone scripts I guess), as that's what truenas gives me out of the box....another mini project to mess with even if it ends going nowhere

For best isolation, use QEMU/KVM VMs for Docker, as it is stated in the documentation. I recommend to use the OS of your choice for the Docker host and there are small Docker-centric distributions available, yet I don't know if you want to add another distribution to your list of supported distributions. I like to stick to one distribution to keep the complexity down.

Yeah, I think I just needed someone to state the obvious, thanks, I have read the recommendations before but given so many homelab'ers seem to use the lxc's for docker I thought why not.

For now I'm migrating my docker containers and related volume data to a debian 12 vm and giving it a go with some real world testing...I can always fallback to my lxc docker stacks after a quick dhcp/dns tweak

truenas scale 24.10 has native docker support and this is what I would use. Up to now they only released the RC2, but the stable release is expected in the next weeks and I havn't experienced any issues with the RC.
Alternatively create a vm with your favourite Linux distribution and run docker/portainer from it.
Don't use docker inside an lxc, this isn't really supported and might break at any time (especially due to updates). Don't use any "helper scripts" either, if you don't understand, how they are working. It's never a good idea to deploy things, without understanding them and will lead to problems sooner or later.

I am running truenas RC2 right now as that was my original choice for docker when there was no proxmox in the picture, but I want docker managed in proxmox now as I prefer the handling of pci/gpu passthrough in there (was even better when many lxc's could use the same resource but I disgress). So the vm route it is for now...

 

[Johannes S]

For now I'm migrating my docker containers and related volume data to a debian 12 vm and giving it a go with some real world testing...I can always fallback to my lxc docker stacks after a quick dhcp/dns tweak


I am running truenas RC2 right now as that was my original choice for docker when there was no proxmox in the picture, but I want docker managed in proxmox now as I prefer the handling of pci/gpu passthrough in there (was even better when many lxc's could use the same resource by anyway). So the vm route it is for now...

Well my own truenas rc2 I'm running as a vm in proxmox with a pcie hba sata adapter in passthrough mode. This works good enough and give me the benefit of proxmox backup function. Of course (since my old cpu can't do nested virtualization) I can't use VMs inside truenas but that's not really a problem: I still can run another vm on the same proxmox host.

 

[LnxBil]

Yeah, I think I just needed someone to state the obvious, thanks, I have read the recommendations before but given so many homelab'ers seem to use the lxc's for docker I thought why not.

I use it too, yet I know of the risks and problems. Most don't and come here to complain about it
I've Alpine LXC docker hosts with vim and bash of about 200M

 

[kaivalagi]

oh, now I remember why I wanted to use lxc's for docker, I can use the host igpu/gpu in jellyfin docker. No hardware transcoding will be an issue.

mmmmmm, I wonder, anything I need access to the igpu for I could create an lxc for maybe, jellyfin is possible with a script, immich too manually....I do like the ability to just stack with docker compose though

this is getting more complicated again

 

[Johannes S]

Well jellyfin is also available for install via the native distribution package system (e.g. apt on Debian/Ubuntu), so in that case it's propably more reasonable to set up a container with the distribution of your choice and afterwards install the needed packages: https://jellyfin.org/docs/general/installation/linux#debuntu-debian-ubuntu-and-derivatives-using-apt

If you had a dedicated GPU you could passthrough it to a vm btw. Another benefits of vms is that that they don't need to be stopped for migration or backup.

Your mileage may vary

 

[Johannes S]

Another thing to consider: Jellyfin allows to run ffmpeg on a remote host to use it's GPU: https://github.com/joshuaboniface/rffmpeg/
So you could run jellyin in a docker image on the nas and the actual ffmpeg in a small lxc container on proxmox or another host.
It might be worth to benchmark, which variant yields the best performance

 

[kaivalagi]

Another thing to consider: Jellyfin allows to run ffmpeg on a remote host to utilice it's GPU: https://github.com/joshuaboniface/rffmpeg/
So you could run jellyin in a docker image on the nas and the actual ffmpeg in a small lxc container on proxmox or another host.
It might be worth to benchmark, which variant yields the best performance

interesting, remote ffmpeg...something else to play with later, thanks