Docker support in Proxmox
- Thread starter aik.fiend
- Start date
Thanks for the setup. Indeed this works just fine, I tried it now.
However, now Docker Swarm doesn't seem to work regardless of what I do to convince it.
The same setup above works perfectly in Docker without swarm, but as soon as I initiate Swarm, with the new interfaces it creates, iptables seems to be go bananas and fails to publish ports.
Will investigate further, but no luck so far.
Thanks again!
Stop docker, create config:
Code:
# less /etc/docker/daemon.json
{
"storage-driver": "overlay2",
"iptables": true,
"ip-masq": true
}Run docker, save iptables generated rules in file:
Code:
iptables-save > /var/lib/iptables-docker-default.rulesStop docker, сhange config:
Code:
# less /etc/docker/daemon.json
{
"storage-driver": "overlay2",
"iptables": false,
"ip-masq": true
}Create file:
Code:
cat /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStartPost=/bin/sh -c "iptables-restore < /var/lib/iptables-docker-default.rules"Restart LXC container.
Iptables rules for ingress and for port forwarding is will be generate.
Profit
With this options set, docker won't even start. Removed it as I don't think it has any impact on the Iptables rules.
However, I followed the rest of the instructions step by step and it still doesn't work.
It seems like there's some Lxc/Proxmox magic going on that doesn't allow docker to run without specific rules...
Still now luck getting Docker Swarm to work in Lxc. Works just fine in VM, however, so not that big of an issue, just a minor inconvenience as this SHOULD work.
Thanks!
On proxmox host is need some modules preloaded:
Code:
# cat /etc/modules
bonding
ip_vs
ip_vs_dh
ip_vs_ftp
ip_vs_lblc
ip_vs_lblcr
ip_vs_lc
ip_vs_nq
ip_vs_rr
ip_vs_sed
ip_vs_sh
ip_vs_wlc
ip_vs_wrr
xfrm_user
#for Docker
overlay
nf_nat
br_netfilter
xt_conntrackAlso docker can't start in LXC on ZFS datasets:
1) overlay can't work on ZFS
2) docker ZFS storage driver can't manage ZFS inside LXC
But can use ZFS volume (without Proxmox UI, editing lxc-container configuration directly) with ext4/xfs and with overlay.
Thanks for the details. I tried them but I cannot seem to get these two load on PVE so that the LXC container sees the and passes to Docker the requirements. Docker still says that the kernel is missing these features.
I encountered this issue a few times before with br_netfilter and still couldn't find a fix for it. Modinfo on PVE doesn't show anything and modprobe is missing these modules...
Perhaps you need enable in config container Options --> Features "keyctl" and "Nesting" or add into config next params:Thanks for the details. I tried them but I cannot seem to get these two load on PVE so that the LXC container sees the and passes to Docker the requirements. Docker still says that the kernel is missing these features.
Code:
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rwbr_netfilter no need to load on pve kernels, because:
Code:
$ grep 'BRIDGE_NETFILTER' /boot/config-$(uname -r)
CONFIG_BRIDGE_NETFILTER=yPersonally I would NEVER permit someone put hands "programmatically" (the approach of Docker) on my Proxmox Host just to manage their Docker services!
The right solution indeed is to contain Docker itself or a Docker orchestration product inside a fence. Security need resources.
In my opinion the best way to achieve this approach would be using LXC as underlying base system. Perhaps, at the moment, KVM with RancherOS or Flatcar Container Linux (CoreOS drop-in replacement) or Fedora CoreOS or any other OS Docker capable are fine.
EDIT: defining a cloud-init template of a VM would be the best option to easily and quickly deploy multiple vms under a swarm or kubernetes cluster.
The right solution indeed is to contain Docker itself or a Docker orchestration product inside a fence. Security need resources.
In my opinion the best way to achieve this approach would be using LXC as underlying base system. Perhaps, at the moment, KVM with RancherOS or Flatcar Container Linux (CoreOS drop-in replacement) or Fedora CoreOS or any other OS Docker capable are fine.
EDIT: defining a cloud-init template of a VM would be the best option to easily and quickly deploy multiple vms under a swarm or kubernetes cluster.
Last edited:
What do we do now that RancherOS is dead? k3os? Fedora CoreOS? Ubuntu LXD? ... This landscape is so disappointing and yet we desperately need Docker support on Proxmox.
As I struggled to create a docker machine driver myself, I came up with another idea which I'm using right now to easily bootstrap docker in KVMs: https://gitlab.com/morph027/pve-cloud-init-creator
Probably the concept will work for @LnxBil driver too if he finds some time
Probably the concept will work for @LnxBil driver too if he finds some time
Yeah? Also dead, i didn't read/notice. That's a shame. All the projects are dying. Why don't you just migrate to a PaaS/Kubernetes solution? Docker is also kinda dead. In comparison before the bankruptcy, there is not much going on in the docker environment, less releases, less new stuff, etc. We're also in the progress of migrating more and more to podman.
That's also nice. I use a similar approach, but with fully-automatic PXE to install the nodes. That's the way I normally provision new nodes nowadays.
Probably the concept will work for @LnxBil driver too if he finds some time
Yeah, yeah ... I know ... shame on me. The overhead of starting is just too high for me right now. I still don't have a big test bench for all the settings and configurations.
Rancher is not dead (last release was in June) but I listened to a talk saying then most of company/user run docker without an orchestrator such as Kubernetes and that was a big mistake/misunderstanding of what is docker for. If I understood well, Since Docker joined the CNCF they split the project into a bunch of projects and docker itself, will focus on the daemon part while networking configuration and other parameters will be a relay to orchestrator such as Kubernetes.
Also, most of people who want docker instead of LXC are simply a bunch of lazy bunnies that don't seem to understand the nature of Docker and only want to run the application as fast a possible without any other consideration. In a production environment running docker because you don't go through the installation process of the application under LXC should not be considered a production environment.
Also, most of people who want docker instead of LXC are simply a bunch of lazy bunnies that don't seem to understand the nature of Docker and only want to run the application as fast a possible without any other consideration. In a production environment running docker because you don't go through the installation process of the application under LXC should not be considered a production environment.
I use Docker, Kubernetes (with Rancher) and some other container tools (but I'm not an expert, just a regular user).
So maybe let me try to comment, without making it a "religous" thing.
Docker was the first container tool (all in one: container runtime, cli tool, orchestrator, registry, etc). Docker was challenged from many sides and there were even attempts to create alternative container "formats" (like rkt). And there were tools, like Kubernetes, that built on top Docker's container runtime.
Today, there's still "Docker" as the "one in all" tool (actually the upstream community project is now called Moby). But some parts have moved to other projects, most notably the container runtime "containerd" (with the lower level part called "runc") is derrived from Docker and Docker itself uses it (but not for everything, for example both "containerd" and "moby" have separate code to pull container images). And others have created their own container runtimes (for example Red Hat created cri-o (with the lover level part "crun"). Kubernetes itself originally used Docker as its runtime (and can still use Docker as its backend, which in turn uses containerd), but it's more common that Kubernetes distributions use containerd or cri-o these days.
Today for end users there are two notable standards:
- the container format that Docker established (now known as OCI) - all tools can use and work with it
- the Kubernetes API for container orchestration
Some people just use one of them (there are many use cases where you don't need complex orchestration - for example to build/test software in CI runners). Some are using both (for example to run complex web applications with Kubernetes).
The Kubernetes API is now also supported by VMWare (vsphere / esxi). A bit simplified: You can just point a Kubernetes yaml at your vsphere cluster. Under the hood vsphere sets up and runs VMs with vmware's Kubernetes distribution for you. This is roughly how I wished Kubernetes would be supported by Proxmox as well.
TL'DR: I agree that it wouldn't make too much sense to treat Docker containers as VMs (that's what LXC is better suited for). But IMHO it would be great if PVE could one-click deploy/extend/shrink a Kubernetes cluster for you, both control plane and worker nodes running as VMs. I recently had the chance to test VMWare's Kubernetes features and I think they're doing a lot right here. Their customers don't need to go to another vendor to get a Kubernetes distro (like Rancher) and painfully integrate it into their existing environment. They can just use it right away. And for VMWare it means their customers have a reason to stay and pay (even though there are great and more cost effective alternatives to vsphere, like PVE). (And don't get my praise for vmware wrong - I've been into vmware licensing at a large org and it wasn't fun. Really appreciate that PVE exists).
(Just my opinion as a user - I'm not an expert and there might be other opinions).
So maybe let me try to comment, without making it a "religous" thing.
Docker was the first container tool (all in one: container runtime, cli tool, orchestrator, registry, etc). Docker was challenged from many sides and there were even attempts to create alternative container "formats" (like rkt). And there were tools, like Kubernetes, that built on top Docker's container runtime.
Today, there's still "Docker" as the "one in all" tool (actually the upstream community project is now called Moby). But some parts have moved to other projects, most notably the container runtime "containerd" (with the lower level part called "runc") is derrived from Docker and Docker itself uses it (but not for everything, for example both "containerd" and "moby" have separate code to pull container images). And others have created their own container runtimes (for example Red Hat created cri-o (with the lover level part "crun"). Kubernetes itself originally used Docker as its runtime (and can still use Docker as its backend, which in turn uses containerd), but it's more common that Kubernetes distributions use containerd or cri-o these days.
Today for end users there are two notable standards:
- the container format that Docker established (now known as OCI) - all tools can use and work with it
- the Kubernetes API for container orchestration
Some people just use one of them (there are many use cases where you don't need complex orchestration - for example to build/test software in CI runners). Some are using both (for example to run complex web applications with Kubernetes).
The Kubernetes API is now also supported by VMWare (vsphere / esxi). A bit simplified: You can just point a Kubernetes yaml at your vsphere cluster. Under the hood vsphere sets up and runs VMs with vmware's Kubernetes distribution for you. This is roughly how I wished Kubernetes would be supported by Proxmox as well.
TL'DR: I agree that it wouldn't make too much sense to treat Docker containers as VMs (that's what LXC is better suited for). But IMHO it would be great if PVE could one-click deploy/extend/shrink a Kubernetes cluster for you, both control plane and worker nodes running as VMs. I recently had the chance to test VMWare's Kubernetes features and I think they're doing a lot right here. Their customers don't need to go to another vendor to get a Kubernetes distro (like Rancher) and painfully integrate it into their existing environment. They can just use it right away. And for VMWare it means their customers have a reason to stay and pay (even though there are great and more cost effective alternatives to vsphere, like PVE). (And don't get my praise for vmware wrong - I've been into vmware licensing at a large org and it wasn't fun. Really appreciate that PVE exists).
(Just my opinion as a user - I'm not an expert and there might be other opinions).
definitely Docker democratizes and puts on the map the container technology, but in the end, Docker is a fork of LXC which is inspired by Jails (BSD). It is basically using Ubuntu instead of Debian; some people would consider that a more useful because the package is more up to date and it is easier with PPA, while others think it is Ubuntu is for noob only.
But one thing is sure, when you try to please everyone you often end pleasing no one. Maybe Proxmox should simply drop LXC, and or make it optional at the installation and/or as service then only focus on KVM. And who ever want containers technology could run his own technology inside a VM.
But one thing is sure, when you try to please everyone you often end pleasing no one. Maybe Proxmox should simply drop LXC, and or make it optional at the installation and/or as service then only focus on KVM. And who ever want containers technology could run his own technology inside a VM.
not sure that that’s accurate, but the point is: OCI containers are here to stay. It’s the essence of Docker that has surpassed Docker and is directly used in projects and products that are designed to challenge/compete/sunset Docker. Docker is already a legacy tool and it may not survive on the long run. But no one is challenging the OCI standard. It’s the common ground in the cloud native ecosystem.
Not sure what you mean. Docker had never anything directly to do with Ubuntu. You can build Moby using containers and run it on almost any flavor of Linux. Same for almost any other container runtimes and tools.
I think LXC support in Proxmox was a wise choice and I’ve heard of people using it in production. The way it’s usable on a high level provides a real benefit (saving performance/system resources) to those who want the overal“feeling” of a VM but don’t need strict isolation security wise,
Docker isn’t intended to be used in this way.
My point was more like: Look at what VMWare is doing to enable Kubernetes on vsphere/esxi. I’d love to see something like this for Proxmox. (I know, it’s not a realistic wish. So Rancher (RancherOS + RKE or any Linux distro + k3s) comes in handy).
(Technically it’s a totally different thing then LXC. Yes, containers but running in VMs).
of course, I'm unable to find the original source of my information but, at least I'm not the only one saying it
it is obviously related to what I mentioned before, such as Ubuntu is a fork of Debian.
I agree with you on this, while Docker is more popular, in the end LXC is easier since it is almost run as its own machine.
In the meanwhile my point is they have a more than 300 Linux distribution active, one of them could probably VirtualMachine and Docker together (I just think of 2 rights now Nethserver and OpenMediaVault)
also it is possible to run Docker inside a LXC container: https://discuss.linuxcontainers.org...in-lxc-unprivileged-container-in-proxmox/3828
It was, it was also completely rewritten (a couple of years ago) and not a fork (a fork means the same codebase stays). Nowadays the name 'container' is the same, the rest is not. Both (LXC and Docker) are based on cgroups and namespaces in the kernel, so they are both based on that and not on each other.
Yes, that's true. With podman, we have also an infrastructure in place that can be used instead of Docker and is 100% compatible and even more secure (no root required). But in the end, the orchestration is the thing that most people will use well more "ordinary containers" and more application vendors will move to
And in Docker you absolutetly don't want a whole machine, that is not was Docker is about. Most containers I use don't run a fully-fledged OS, just a binary and that is the beauty of it.
Possible and good are two things, running Docker in LXC in production is possible, but not good (just look at the forums).
You guys are missing the forest with all this obsession with trees.
docker :: LXC = Ubuntu :: Linux Kernel.
One enables the other. They are not replacements, they are not the same function. The big difference between how proxmox uses containers and how docker uses containers is the focus on programmatic access- lxc on proxmox is designed as lightweight virtualization; docker is meant to operate as git for executable applications. docker without an orchestrator isnt very useful but controlled centrally becomes a very powerful tool for ci/cd lifecycle at the system level.
docker :: LXC = Ubuntu :: Linux Kernel.
One enables the other. They are not replacements, they are not the same function. The big difference between how proxmox uses containers and how docker uses containers is the focus on programmatic access- lxc on proxmox is designed as lightweight virtualization; docker is meant to operate as git for executable applications. docker without an orchestrator isnt very useful but controlled centrally becomes a very powerful tool for ci/cd lifecycle at the system level.
It really depends on your use case. I use lxc containers for various applications all the time with great success; it cannot replace a VM for all purposes but then thats not the scope of containers... Much of the internet runs microservices on container these days
I think this discussion was lunched in 2016 means over 4 years now. It happend like some others said longtime ago. Docker got big and biger and I've no other choice anymore than use it due to the fakt that our Development depends on Parners just delivering their parts in form of Docker container. I loved Proxmox. I used VM's and LXC. Over 90% of all we use inbetween is able to run under Docker (even .NET Core) I can no longer justify the use of Proxmox.
In my circles I'm allready the 7th CIO wich is moving away from Proxmox. Probably would have been not to bad to listen to your Users/Customers.
With thees word I say farwell and good luck for you all.
In my circles I'm allready the 7th CIO wich is moving away from Proxmox. Probably would have been not to bad to listen to your Users/Customers.
With thees word I say farwell and good luck for you all.