Docker on promox recommendations

[luison]

As others on this forum we've been considering for a while migrating some installations to docker containers and also like other I would hope to be able to do it within the Proxmox setup.

I was wondering if the new Proxmox 6 brings any updates or changes to this. As far as I understand Proxmox developers are not into supporting proxmox on containers while I also understand is technically possible by reducing the isolation of that LXC with Docker CE with "lxc.apparmor.profile" configuration. I understand this works but it's not recommended for a production environment due to those security reasons.

With that in mind, unless any changes in Proxmox 6 my options to have a docker system in production I believe would still be to install an isolated KVM machine running the Docker CE. My doubt in this case is the "cost" in efficiency of that system and the limitations as far as sharing host documents with other LXC containers.

Any updated reference to documentation regarding would be greatly appreciated.

 

[LnxBil]

With that in mind, unless any changes in Proxmox 6 my options to have a docker system in production I believe would still be to install an isolated KVM machine running the Docker CE.

Docker is still a PaaS solution and PVE an IaaS, so mixing both is unwise. Running in KVM still the (security-wise) best option.

Also, If you're running a PVE cluster, it is recommended to run a Docker Cluster aswell. You'll end up using orchestration as PaaS with Swarm/Kubernetes/$Whatsoever.

My doubt in this case is the "cost" in efficiency of that system and the limitations as far as sharing host documents with other LXC containers.

What do you mean by "sharing documents"?

One way to do this on a single node PVE is to use ZFS, create datasets and mount them directly via NFS to your KVM Docker host (can be done directly in Docker with compose). With this way, the datasets are present on the host and not in the VM, which is great for default ZFS replication/backup/snapshots etc. You can then share them, but without locks and such, this is very dangerous.

 

[luison]

Hi @LnxBil. Thanks for the feedback.

Understanding the differences I still think that Proxmox / LXC would benefit in the long term of supporting running Docker hosts on containers if that was properly isolated.

As for "sharing documents" more or less what you mentioned. We currently do via bind mounts of certain host directories made available to various different containers.

Turning all to ZFS is likely not an option for us at the moment unfortunately, unless we can be sure we can get hosting on that same setup, but we'll look into it as I understand PVE6 also improves its support. Regarding your suggestion of using NFS (which has surprised me) for sharing to the KVM it gives other overheads but I guess we'll also have to consider if it's the only option to share from PVE/LXC containers to a Docker container running within the KVM.

 

[LnxBil]

Understanding the differences I still think that Proxmox / LXC would benefit in the long term of supporting running Docker hosts on containers if that was properly isolated.

I can totally understand the point of the Proxmox staff not going to implement this. Most people that want such a feature don't understand what Docker is really about. They see just another easier container solution in comparison to LXC, but it is not that. Docker started out to be that, they also used LXC in their beginning, but Docker and its eco system evolved and reaches it full potential with autoscaling, orchestration etc. There are many solutions for this, but the (current) champion on this is kubernetes, which runs totally fine inside of KVM.
Proxmox VE also reaches it full potential in being a clusterized solution and the would need to reimplement everything from scratch and every other solution is man-decades or even man-centuries away. Every big company that needs a PaaS platform just goes to Kubernetes and the backing IaaS is totally exchangeable, so PVE is just right as any other solution to do that job. Administration is done with Kubernetes tools.

Running Docker containers and "normal" virtualization in PVE have nothing in common - the first is a PaaS solution, the other a IaaS solution. It just solves different problems. Even if you look at the big player VMware. They run Docker inside of a VM and have in their fully fledged vcenter a simple, non-integrated (into other services) web GUI for docker that shares nothing of the look&feel of the ordinary VM administration stuff. If you want that with PVE, just go with it and install e.g. portainer.

Regarding your suggestion of using NFS (which has surprised me)

The problem is the lack of alternatives. I don't like NFS, but it solves some problem and introduces new ones of course.

If all services would run in Docker, it would easier to share docker volumes and you would not need any other technology in between, but as I stated before, LXC and KVM live on another layer as Docker containers do. Mixing those technologies does not solve any problems, it introduces many new ones.