Docker support in Proxmox

Sep 8, 2019
9
6
3
60
Seed, sounds like a nice setup. I'm not quite that automated. I just clone my Debian Buster image, add it to the firewall for DHCP then run the kubernetes command to add it to the cluster.
 
  • Like
Reactions: kifeo

Alibek

Member
Jan 13, 2017
81
10
13
41
Thanks for the setup. Indeed this works just fine, I tried it now.

However, now Docker Swarm doesn't seem to work regardless of what I do to convince it.

The same setup above works perfectly in Docker without swarm, but as soon as I initiate Swarm, with the new interfaces it creates, iptables seems to be go bananas and fails to publish ports. :(

Will investigate further, but no luck so far.

Thanks again!
Stop docker, create config:
Code:
# less /etc/docker/daemon.json
{
  "storage-driver": "overlay2",
  "iptables": true,
  "ip-masq": true
}
Run docker, save iptables generated rules in file:
Code:
iptables-save > /var/lib/iptables-docker-default.rules
Stop docker, сhange config:
Code:
# less /etc/docker/daemon.json
{
  "storage-driver": "overlay2",
  "iptables": false,
  "ip-masq": true
}
Create file:
Code:
cat /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStartPost=/bin/sh -c "iptables-restore < /var/lib/iptables-docker-default.rules"
Restart LXC container.
Iptables rules for ingress and for port forwarding is will be generate.


Profit
 
  • Like
Reactions: MadalinC and jebbam

MadalinC

New Member
Jan 28, 2020
14
0
1
"storage-driver": "overlay2",
With this options set, docker won't even start. Removed it as I don't think it has any impact on the Iptables rules.

However, I followed the rest of the instructions step by step and it still doesn't work.
It seems like there's some Lxc/Proxmox magic going on that doesn't allow docker to run without specific rules...

Still now luck getting Docker Swarm to work in Lxc. Works just fine in VM, however, so not that big of an issue, just a minor inconvenience as this SHOULD work.

Thanks!
 

Alibek

Member
Jan 13, 2017
81
10
13
41
With this options set, docker won't even start. Removed it as I don't think it has any impact on the Iptables rules.

However, I followed the rest of the instructions step by step and it still doesn't work.
It seems like there's some Lxc/Proxmox magic going on that doesn't allow docker to run without specific rules...

Still now luck getting Docker Swarm to work in Lxc. Works just fine in VM, however, so not that big of an issue, just a minor inconvenience as this SHOULD work.

Thanks!
On proxmox host is need some modules preloaded:
Code:
# cat /etc/modules
bonding
ip_vs
ip_vs_dh
ip_vs_ftp
ip_vs_lblc
ip_vs_lblcr
ip_vs_lc
ip_vs_nq
ip_vs_rr
ip_vs_sed
ip_vs_sh
ip_vs_wlc
ip_vs_wrr
xfrm_user

#for Docker
overlay
nf_nat
br_netfilter
xt_conntrack
Also docker can't start in LXC on ZFS datasets:
1) overlay can't work on ZFS
2) docker ZFS storage driver can't manage ZFS inside LXC

But can use ZFS volume (without Proxmox UI, editing lxc-container configuration directly) with ext4/xfs and with overlay.
 

MadalinC

New Member
Jan 28, 2020
14
0
1
nf_nat br_netfilter
Thanks for the details. I tried them but I cannot seem to get these two load on PVE so that the LXC container sees the and passes to Docker the requirements. Docker still says that the kernel is missing these features. :(

I encountered this issue a few times before with br_netfilter and still couldn't find a fix for it. Modinfo on PVE doesn't show anything and modprobe is missing these modules...
 

Alibek

Member
Jan 13, 2017
81
10
13
41
Thanks for the details. I tried them but I cannot seem to get these two load on PVE so that the LXC container sees the and passes to Docker the requirements. Docker still says that the kernel is missing these features. :(
Perhaps you need enable in config container Options --> Features "keyctl" and "Nesting" or add into config next params:
Code:
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop: 
lxc.mount.auto: proc:rw sys:rw
I encountered this issue a few times before with br_netfilter and still couldn't find a fix for it. Modinfo on PVE doesn't show anything and modprobe is missing these modules...
br_netfilter no need to load on pve kernels, because:
Code:
$ grep 'BRIDGE_NETFILTER' /boot/config-$(uname -r)
CONFIG_BRIDGE_NETFILTER=y
 

stefanobertoli

New Member
Apr 25, 2020
14
0
1
39
Personally I would NEVER permit someone put hands "programmatically" (the approach of Docker) on my Proxmox Host just to manage their Docker services!

The right solution indeed is to contain Docker itself or a Docker orchestration product inside a fence. Security need resources.
In my opinion the best way to achieve this approach would be using LXC as underlying base system. Perhaps, at the moment, KVM with RancherOS or Flatcar Container Linux (CoreOS drop-in replacement) or Fedora CoreOS or any other OS Docker capable are fine.

EDIT: defining a cloud-init template of a VM would be the best option to easily and quickly deploy multiple vms under a swarm or kubernetes cluster.
 
Last edited:

LnxBil

Famous Member
Feb 21, 2015
5,449
589
133
Germany
What do we do now that RancherOS is dead? k3os? Fedora CoreOS? Ubuntu LXD? ... This landscape is so disappointing and yet we desperately need Docker support on Proxmox.
Yeah? Also dead, i didn't read/notice. That's a shame. All the projects are dying. Why don't you just migrate to a PaaS/Kubernetes solution? Docker is also kinda dead. In comparison before the bankruptcy, there is not much going on in the docker environment, less releases, less new stuff, etc. We're also in the progress of migrating more and more to podman.

As I struggled to create a docker machine driver myself, I came up with another idea which I'm using right now to easily bootstrap docker in KVMs: https://gitlab.com/morph027/pve-cloud-init-creator
That's also nice. I use a similar approach, but with fully-automatic PXE to install the nodes. That's the way I normally provision new nodes nowadays.

Probably the concept will work for @LnxBil driver too if he finds some time ;)
Yeah, yeah ... I know ... shame on me. The overhead of starting is just too high for me right now. I still don't have a big test bench for all the settings and configurations.
 

chr1s

New Member
Sep 9, 2020
7
2
3
36
What do we do now that RancherOS is dead?
I don't think RancherOS is dead. There has been some movement in the CoreOS space, but CoreOS isn't dead either (lives on as paid-only Red Hat product, community upstream project Fedora Core OS and the Flatcar fork).
 

JOduMonT

Active Member
Jan 20, 2016
40
2
28
Bangkok
jdumont.consulting
Rancher is not dead (last release was in June) but I listened to a talk saying then most of company/user run docker without an orchestrator such as Kubernetes and that was a big mistake/misunderstanding of what is docker for. If I understood well, Since Docker joined the CNCF they split the project into a bunch of projects and docker itself, will focus on the daemon part while networking configuration and other parameters will be a relay to orchestrator such as Kubernetes.

Also, most of people who want docker instead of LXC are simply a bunch of lazy bunnies that don't seem to understand the nature of Docker and only want to run the application as fast a possible without any other consideration. In a production environment running docker because you don't go through the installation process of the application under LXC should not be considered a production environment.
 

chr1s

New Member
Sep 9, 2020
7
2
3
36
I use Docker, Kubernetes (with Rancher) and some other container tools (but I'm not an expert, just a regular user).
So maybe let me try to comment, without making it a "religous" thing.

If I understood well, Since Docker joined the CNCF they split the project into a bunch of projects and docker itself, will focus on the daemon part while networking configuration and other parameters will be a relay to orchestrator such as Kubernetes.
Docker was the first container tool (all in one: container runtime, cli tool, orchestrator, registry, etc). Docker was challenged from many sides and there were even attempts to create alternative container "formats" (like rkt). And there were tools, like Kubernetes, that built on top Docker's container runtime.

Today, there's still "Docker" as the "one in all" tool (actually the upstream community project is now called Moby). But some parts have moved to other projects, most notably the container runtime "containerd" (with the lower level part called "runc") is derrived from Docker and Docker itself uses it (but not for everything, for example both "containerd" and "moby" have separate code to pull container images). And others have created their own container runtimes (for example Red Hat created cri-o (with the lover level part "crun"). Kubernetes itself originally used Docker as its runtime (and can still use Docker as its backend, which in turn uses containerd), but it's more common that Kubernetes distributions use containerd or cri-o these days.

Today for end users there are two notable standards:
- the container format that Docker established (now known as OCI) - all tools can use and work with it
- the Kubernetes API for container orchestration

Some people just use one of them (there are many use cases where you don't need complex orchestration - for example to build/test software in CI runners). Some are using both (for example to run complex web applications with Kubernetes).

The Kubernetes API is now also supported by VMWare (vsphere / esxi). A bit simplified: You can just point a Kubernetes yaml at your vsphere cluster. Under the hood vsphere sets up and runs VMs with vmware's Kubernetes distribution for you. This is roughly how I wished Kubernetes would be supported by Proxmox as well.

TL'DR: I agree that it wouldn't make too much sense to treat Docker containers as VMs (that's what LXC is better suited for). But IMHO it would be great if PVE could one-click deploy/extend/shrink a Kubernetes cluster for you, both control plane and worker nodes running as VMs. I recently had the chance to test VMWare's Kubernetes features and I think they're doing a lot right here. Their customers don't need to go to another vendor to get a Kubernetes distro (like Rancher) and painfully integrate it into their existing environment. They can just use it right away. And for VMWare it means their customers have a reason to stay and pay (even though there are great and more cost effective alternatives to vsphere, like PVE). (And don't get my praise for vmware wrong - I've been into vmware licensing at a large org and it wasn't fun. Really appreciate that PVE exists).

(Just my opinion as a user - I'm not an expert and there might be other opinions).
 
  • Like
Reactions: JOduMonT

JOduMonT

Active Member
Jan 20, 2016
40
2
28
Bangkok
jdumont.consulting
definitely Docker democratizes and puts on the map the container technology, but in the end, Docker is a fork of LXC which is inspired by Jails (BSD). It is basically using Ubuntu instead of Debian; some people would consider that a more useful because the package is more up to date and it is easier with PPA, while others think it is Ubuntu is for noob only.

But one thing is sure, when you try to please everyone you often end pleasing no one. Maybe Proxmox should simply drop LXC, and or make it optional at the installation and/or as service then only focus on KVM. And who ever want containers technology could run his own technology inside a VM.
 

chr1s

New Member
Sep 9, 2020
7
2
3
36
definitely Docker democratizes and puts on the map the container technology, but in the end, Docker is a fork of LXC which is inspired by Jails (BSD).
not sure that that’s accurate, but the point is: OCI containers are here to stay. It’s the essence of Docker that has surpassed Docker and is directly used in projects and products that are designed to challenge/compete/sunset Docker. Docker is already a legacy tool and it may not survive on the long run. But no one is challenging the OCI standard. It’s the common ground in the cloud native ecosystem.

It is basically using Ubuntu instead of Debian; some people would consider that a more useful because the package is more up to date and it is easier with PPA, while others think it is Ubuntu is for noob only.

Not sure what you mean. Docker had never anything directly to do with Ubuntu. You can build Moby using containers and run it on almost any flavor of Linux. Same for almost any other container runtimes and tools.

But one thing is sure, when you try to please everyone you often end pleasing no one. Maybe Proxmox should simply drop LXC, and or make it optional at the installation and/or as service then only focus on KVM. And who ever want containers technology could run his own technology inside a VM.
I think LXC support in Proxmox was a wise choice and I’ve heard of people using it in production. The way it’s usable on a high level provides a real benefit (saving performance/system resources) to those who want the overal“feeling” of a VM but don’t need strict isolation security wise,

Docker isn’t intended to be used in this way.

My point was more like: Look at what VMWare is doing to enable Kubernetes on vsphere/esxi. I’d love to see something like this for Proxmox. (I know, it’s not a realistic wish. So Rancher (RancherOS + RKE or any Linux distro + k3s) comes in handy).

(Technically it’s a totally different thing then LXC. Yes, containers but running in VMs).
 

JOduMonT

Active Member
Jan 20, 2016
40
2
28
Bangkok
jdumont.consulting
not sure that that’s accurate,
of course, I'm unable to find the original source of my information but, at least I'm not the only one saying it
1600241378161.png

Not sure what you mean
it is obviously related to what I mentioned before, such as Ubuntu is a fork of Debian.

I think LXC support in Proxmox was a wise choice and I’ve heard of people using it in production.
I agree with you on this, while Docker is more popular, in the end LXC is easier since it is almost run as its own machine.

In the meanwhile my point is they have a more than 300 Linux distribution active, one of them could probably VirtualMachine and Docker together (I just think of 2 rights now Nethserver and OpenMediaVault)

also it is possible to run Docker inside a LXC container: https://discuss.linuxcontainers.org...in-lxc-unprivileged-container-in-proxmox/3828
 

LnxBil

Famous Member
Feb 21, 2015
5,449
589
133
Germany
of course, I'm unable to find the original source of my information but, at least I'm not the only one saying it
View attachment 19875


it is obviously related to what I mentioned before, such as Ubuntu is a fork of Debian.
It was, it was also completely rewritten (a couple of years ago) and not a fork (a fork means the same codebase stays). Nowadays the name 'container' is the same, the rest is not. Both (LXC and Docker) are based on cgroups and namespaces in the kernel, so they are both based on that and not on each other.

OCI containers are here to stay.
Yes, that's true. With podman, we have also an infrastructure in place that can be used instead of Docker and is 100% compatible and even more secure (no root required). But in the end, the orchestration is the thing that most people will use well more "ordinary containers" and more application vendors will move to


I agree with you on this, while Docker is more popular, in the end LXC is easier since it is almost run as its own machine.
And in Docker you absolutetly don't want a whole machine, that is not was Docker is about. Most containers I use don't run a fully-fledged OS, just a binary and that is the beauty of it.

Possible and good are two things, running Docker in LXC in production is possible, but not good (just look at the forums).
 

alexskysilk

Well-Known Member
Oct 16, 2015
652
71
48
Chatsworth, CA
www.skysilk.com
You guys are missing the forest with all this obsession with trees.

docker :: LXC = Ubuntu :: Linux Kernel.

One enables the other. They are not replacements, they are not the same function. The big difference between how proxmox uses containers and how docker uses containers is the focus on programmatic access- lxc on proxmox is designed as lightweight virtualization; docker is meant to operate as git for executable applications. docker without an orchestrator isnt very useful but controlled centrally becomes a very powerful tool for ci/cd lifecycle at the system level.

Possible and good are two things, running Docker in LXC in production is possible, but not good (just look at the forums).
It really depends on your use case. I use lxc containers for various applications all the time with great success; it cannot replace a VM for all purposes but then thats not the scope of containers... Much of the internet runs microservices on container these days ;)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!