Best (config/storage) strategy for multiple lxc with docker?

[Meister177]

Hi,
I have to rebuild a node. I want it to run multiple LXCs with multiple docker container and am wondering on what would be the best strategy on this.

Here are my ideas so far.
Let's say I have two LXC called "network" and "media".
"network" should run something like adguard and wireguard.
"media" should run jellyfin and navidrome.

Now I create a dataset like /tank/container and mount it to all LXCs. Each LXC creates a folder with their hostname in it (e.g. /tank/container/network). And inside of this folder there are all compose files.
Ultimately I want to sync my compose files via git in an gitea instance. There I'll just edit all of them on my windows machine push changes and pull them on the different LXCs.

And now I could either set all paths in the compose files to my mountpoint so in addition to the compose files the data generated by the container will be saved on my dataset as well. Or I could save the container data inside the LXC itself and make use of the backup function inside of proxmox.

I'm not sure what would be the best practice here. I'm still very new to most of the stuff.

Thanks

 

[Madozu]

The simple answer is "If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox QEMU VM". That's not my invention, it is from the Proxmox Wiki.

Grouping similar apps into "functional LXC containers" like you plan to do for network and media makes perfect sense ... but I strongly recommend to install the applications (like adguard or wireguard) directly within the LXC container running on proxmox. I don't see much of a reason to get even started with the "nested container issues" you might (most probably will) face when running docker containers within LXC containers.

Some of reasons that brought me to migrate my docker containers away from docker LXCs to docker VMs are:

• Updates to (Proxmox) LXC containers can break your docker installation (happened in the past ... might happen again).
• Unprivileged LXCs cannot directly mount SMB/CIFS shares and therefore you cannot work with docker volumes stored on a SMB-attached NAS. The only workaround would be to launch the LXC container privileged ... which I would avoid if ever possible (for security reasons)
• I observed some sluggish behavior like long container startup times when running Docker within LXCs.
• Docker and docker images are well tested running on regular Hypervisors (Vmware, Proxmox, HyperV, Xen/XOA) but not much tested running within LXCs.
• You cannot live-migrate LXC containers between PVE hosts, but you can live-migrate VMs, resulting in no downtime for the Docker containers.

So, below the line, I recommend that you start with docker in a Linux VM, it's straight forward and the recommended way to go.