Running Docker on the Proxmox host, not in VM / CT

[slavik-f]

I'm new Proxmox user. Installed 3 nodes cluster with CephFS.

Looking at docs here: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pct

It is not recommended to run docker directly on your Proxmox VE host.

Running Docker in CT is not recommended, too.
Recommendation is to run Docker in VM.

Does anyone have any explanation: why it's not recommended to run Docker on the Proxmox host?

The reason I want to try to run Docker on the host:
1. Easy access to CephFS on the host: direct bind to /mnt/pve/cephfs/mountX. Versus hassle (example) of mounting CephFS from VM
2. Easy access to storage on USB and node's hard drives (not CephFS). Easy bind if done on the host. Hassle with passthrough, if done from VM.
3. Efficiency. VM is really unnecessary. Less maintenance. Lower attack surface.

From my limited research I saw these reasons, why I should NOT run Docker on host:

1. Potential Conflicts and Interference: Proxmox VE relies on specific kernel modules and configurations to manage VMs and containers effectively. Installing Docker directly on the host can introduce conflicts that disrupt Proxmox VE's functionality.

Is it true? Seems to be speculative, as Docker runs on millions of other systems without the interference. Is it Proxmox really the edge case here?

2. Security Risks.

It's true that Docker containers doesn't have the level of isolation, which VM has. But for some environments that's acceptable risk.

Testing:
I'm going to install Docker on the Proxmox host and test it. What should I watch for? Where should I expect to have issues?

 

[alexskysilk]

Does anyone have any explanation: why it's not recommended to run Docker on the Proxmox host?

the answer is in the docs you linked.

If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox QEMU VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.

 

[slavik-f]

Thank you for the reference. But that answers another question.
My question is not about benefits of running Docker in VM. I don't need these benefits. I have different goals.

My question: what prevents Docker from running on the Proxmox host? Is there any technical reasons or limitations?

 

[alexskysilk]

My question: what prevents Docker from running on the Proxmox host? Is there any technical reasons?

the proxmox host is just a debian box. if you want to run you containers on it directly, go nuts. the reason its not recommended on PVE is because you are running a sizable stack to provide virtualization services; the implication here is that you dont really want to run services outside its management layer. If you intend to use the server as a docker host, why bother with proxmox in the first place?

 

[slavik-f]

why bother with proxmox in the first place?

Because of CephFS

 

[alexskysilk]

you dont need proxmox to attach cephfs. its available for every modern linux distro.

 

[LnxBil]

My question: what prevents Docker from running on the Proxmox host? Is there any technical reasons or limitations?

At least in PVE7, docker injects iptables rules that will interfere with the PVE firewall and will - per default - render networking useless. It will change how the forwarding chain works and therefore all VMs/containers will have no working network. You need to run iptables commands to fix this after each docker service restart or do your port redirection outside of Docker and disable the iptables integration.

 

[slavik-f]

Thank you! That's exactly what I was asking about.

You need to run iptables commands to fix this after each docker service restart or do your port redirection outside of Docker and disable the iptables integration.

Can you tell more about iptables commands? Do you have examples? references?

As I'm researching this, here are few ideas I'm considering:
1. use podman in rootless mode. That way podman will not interfere with iptables. will need to check if there are any performance implications.
2. use dedicated NIC / interface for Docker. Not sure, how complicated that to configure.

 

[LnxBil]

Can you tell more about iptables commands? Do you have examples? references?

Per default, Docker will change the FORWARD chain setting from ACCEPT to DROP or REJECT and you need to run

 iptables -P FORWARD ACCEPT

after each docker service restart in order to get forwarding to containers/VMs back. Unless you set this setting, no VM/CT will have a working network over your default bridge. This is really a PITA.

As I'm researching this, here are few ideas I'm considering:
use podman in rootless mode. That way podman will not interfere with iptables. will need to check if there are any performance implications.

podman will also mess with your network, it needs to if you want to expose a service. Podman is a bit better here, it has integration into at least a firewall framework, namely firewalld. We use it in non-PVE podman setups and it works well, yet there is no integration into the PVE firewall network that I know of.

use dedicated NIC / interface for Docker. Not sure, how complicated that to configure.

Yes, this works, same as having another IP on you interface, yet it will not solve the firewall problem.

You will not have any such problems if you just run it - as everybody always tells you - in a dedicated QEMU VM.

I'm also telling everyone to do this, yet I personally also run it on PVE directly and I'm not totally fine with it, yet it's the easiest setup for me with what I have at hand right now (PVE, NAS, Docker and ZFS). I can also say that it breaks from time to time (after updates) and if you don't want such a setup then don't use it. Running a seperate VM is ALWAYS easier to setup and maintain despite what a lot of people told you here. In a production setup, I exclusively run it in a VM on a PVE cluster and it works as it should and is much, much more secure than running it on PVE directly. You will not have the best network container separation unless you seperate it in disjunct stacks.

Easiest example is a simple Traefik setup in which you expose two service stacks with its own backend services. Most docker containers I ever saw are not security centric, many docker-compose.yml files aren't either, so for the best security setup, you need to

• no run processes in docker containers as root
• seperate stacks in service networks and public facing networks
• you need firewalling or at least auditing of connections from inside your stack to the outside world
• use security constraints like cpu/memory limiting and monitoring

with respect to the traefik example, each of your stacks exposes a network port ot the traefik network and traefik forwards it to the outside world. In order to do this, you need to have a network between traefik and your service exposing container in your respective docker stack and if you do not have ONE network for each service and use the default in any tutorial I saw and use traefik-web or traefik-external, you will also have connectivity from one exposing serivce docker container to any other in any stack and no control over the packet flow in between. This is a huge risk if one stack is compromised and can only be mitigated with proper firewalling, which you need to setup manually, or use ONE transfer traefik network from traefik to ONE stack in order to seperate it directly. I really hate that docker does not provice a better way for container security (If you know a way, let me know).

 

[keeka]

Running Docker in CT is not recommended, too.
Recommendation is to run Docker in VM.

I avoided running docker on the pve host. and went with a VM, despite the fact I do not have the PVE firewall enabled. Though you can have pre/post exec systemd overrides to handle the mangling of the firewall.
Then I wanted separate stacks, on each of 3 subnets. I could not resist the temptation to do this with CT's, despite the advice not to do so.
It has worked well for me, other than guest summary stats becoming increasingly non-functional with each PVE upgrade: First was disk writes, more recently network traffic graphs. Whilst I note the recommendation to use VMs, I'd like to understand better the issues doing it with CTs.
On my resource limited homelab, CT hosting has big benefits. During my time with proxmox, I've found no need to use Linux VMs, other than for docker, if one follows the guidelines.

 

[randy3k]

Alternatively, we could prevent docker manipulating `iptables` by modifying `/etc/docker/daemon.json`

{
    "iptables": false
}

Docker will fallback to use userland proxy to perform port forwarding when iptables is disabled.
Though note that the userland proxy is not as efficient as iptables and uses more memory. And it doesn't work in some network settings, for example `--network host`.

In my case, I used docker to run proxmox backup server which stores the backups to an external drive (raid 1). The PBS container data, config and the docker-compose.yaml are also stored in the external drive. In the case that the proxmox machine is down, I could easily launch the backup server in a new machine and start the restoring process.

 

[esi_y]

Alternatively, we could prevent docker manipulating `iptables` by modifying `/etc/docker/daemon.json`

This thread is becoming ridiculous. Think of it for a moment, you are editing a config file to politely ask service running as root not to touch what only root can touch, then you keep it running, as root.

But then, who am I to judge, the whole thing has everything under one user, root.

Yeah, do not install stuff on the host, it's brittle as it is already.

 

[esi_y]

Most docker containers I ever saw are not security centric, many docker-compose.yml files aren't either

That's tip of the iceberg when you consider how people treat docker pull as if it was apt install.