Proxmox (7.1) and Docker: LXC vs VM

[twl]

The official PVE FAQ recommends to place docker containers in VM's. (LXC vs LXD vs Proxmox Containers vs Docker)
There are also numerous posts here in the forum that recommend to use VM's.

However, there are tons of tutorials on using CT (LXC) to run Docker containers on Proxmox.


What exactly is the reason VM's are recommended?

Here some pros and cons I can come up with, and I like the help of the community to put together a post that gives an informed recommendation for different use cases.

---

Docker in a CT (LXC)
pro

• less resource hungry
• PBS Backups: single file recovery functionality

con

• less performance? (some claim it has poor performance)
• Security: less isolated than a VM (but still more than a native docker installation)
• some containers might not work and need the option “privileged container” (security issue)
• PBS Backups take longer

---

Docker in a VM
pro

• highest possible isolation.
• Docker works like on a native installation.
• PBS Backups are faster

con

• resource heavy, in particular if one wants one VM for each docker container.
• Backups with PBS: only the entire VM can be restored not single files.

---

Do you agree with my list? What did I forget to mention? Why do you recommend or do not recommend to use CT's to run Docker.
I like to hear your opinion and will update the post accordingly.

---

My personal scenario

I have rather limited home lab PVE hardware (i5 CPU and 20GB RAM) and a second machine setup as PBS.
For this reason, I like to keep the numbers of VM's at a minimum and use CT's whenever possible.

There are a couple of Docker images I like to use, all of them comparably lightweight applications. (Wireguard, duckdns, nginx-proxy-manager, Heimdall, and probably some more)
Currently, I have all of them inside an Ubuntu 20.04 VM. This means if something goes wrong with my Ubuntu VM, all docker containers will be down.

Ideally, I would like to have a separate CT for each Docker image.

What do you think, does this make sense? Is it idiotic? What do you recommend?

 

[Dunuin]

Docker in a CT (LXC)
pro

• backups to PBS faster and single file recovery

---

Docker in a VM

• Backups with PBS take longer and only the entire VM can be restored not single files.

Actually PBS backups of LXCs should be slower because only VMs can use dirty bitmapping. Lets say you got 20GB data and 200MB will change per day. Even if you store those 20GB of data on a 32GB virtual disk of a VM, with dirty bitmapping PVE needs only to read the changed 200MB and send it to the PBS. A LXC would need to read the complete 20GB to find out that only 200MB had changed and will then send die 200MB. Whats takes so long isn't the sending of the 200MB but the reading/hasing of the full 20GB. So a VM would only be slower on the first run (where it would need to read the full 32GB) or after shutting down the VM or rebooting the server (as a shutdown will reset the dirty bitmapping).

I personally switched from docker in an unprivileged LXC to a rootless docker in a VM. I really prefer the additional isolation/security especially when running 3rd party docker containers where you run code on your machine created by other people you can't really trust. Wouldn't be the first time that a LXC security hole is discovery allowing an attacker to get control of the host. Or that a docker account gets hacked and someone uploads a cryptominer that you then will run on your server.

 

[twl]

thanks @Dunuin I was not aware of that and will update the post accoringly.

 

[LnxBil]

- You can use ZFS for Docker in your VM, you (still) cannot do that in LX(C) containers
- As @Dunuin already said using a specialized docker-centric linux distribution or even directly k8s. VMs can be better provisioned, they are real VMs

 

[Helmut101]

- You can use ZFS for Docker in your VM, you (still) cannot do that in LX(C) containers

I can confirm that ZFS+Unprivileged LXC + Docker works flawlessly. See here. I have 7 Unprivileged LXC with Docker nested inside each, with a total of about 25 docker containers, running on ZFS Volumes formatted as XFS. No problems since 2 years. Even heavier containers (Gitlab, Nextcloud etc.) work without problems. I do not see any performance issues either, my CPU utilization is 1-2% on average since every process is run on the host, and most processes are idle most of the time. I would expect that this is different with full VMs.

 

[LnxBil]

running on ZFS Volumes formatted as XFS

So, you're not using the awesome ZFS-Backend for Docker then which is MUCH slower than using ZFS directly and you will not have the cool features like docker image layers in snapshots and automatic datasets for docker volumes.

 

[Helmut101]

> So, you're not using the awesome ZFS-Backend for Docker then which is MUCH slower than using ZFS directly and you will not have the cool features like docker image layers in snapshots and automatic datasets for docker volumes.

AFAIK it is only slower for building Docker images - which only applies to e.g. Gitlab Runners etc. - I usually build an image once a month (when updates appear). If you're developing software, I agree and suggest to use a VM instead.

The reading performance difference is negligible based on my observations. I have never observed I/O wait times.

Regarding snapshots: You can use ZFS Snapshots. Again, if you're developing Docker images with lots of layers, VM is the way to go.

My reply was only to your statement:

You can use ZFS [...] you (still) cannot do that in LX(C) containers

which is not true.

 

[LnxBil]

My reply was only to your statement:

> You can use ZFS [...] you (still) cannot do that in LX(C) containers

which is not true.

Sorry, but using XFS ontop of ZFS is not using ZFS. You also did not go to another continent by bicycle if you're sitting on it while flying on a plane.
The way you described it, you're just using a black box (that is ZFS here) that has no inherent features besides being a block storage.

Maybe I miscommunicated my points, so another try:

If you're running Docker on a non-ZFS (or BTRFS) based system, you always use some kind of overlay filesystem to store changes of layers. This is per se not immutable and needs work on every access to a file. Docker volumes are also directly files, which cannot be snapshotted on a per volume basis. Whereas with ZFS, you have immutable layers (even volumes if you want) of your containers that are already snapshots, direct accessible without any overlays and space efficient (before using compression or deduplication). Volumes are ZFS datasets and therefore snapshotable, which works great for e.g. migration tests. Just snapshot, clone and start your stack with the new clone without any file copy, downtime or whatever.
You have also other features like quota and refreservation directly available for docker volumes.

That is what I meant with "cannot do that in LX(C) containers". The feature sets are disjoint. I also run your black-box-approach, but it is on lower-end-machines significantly slower than using ZFS directly on the host.

 

[Helmut101]

> Sorry, but using XFS ontop of ZFS is not using ZFS. You also did not go to another continent by bicycle if you're sitting on it while flying on a plane.

I quote myself:

Even though this ZFS volume is formatted as XFS, all ZFS features are still available, such as snapshots, encryption, compression, or sparse provisioning (meaning that the volume will only occupy the space that is actually used).

> but it is on lower-end-machines significantly slower than using ZFS directly on the host.

I have done extensive performance benchmarks with pveperf and fio and there was almost zero difference to e.g. ext4 on a standard HW Raid1.

But you seem to be arguing for something different, and I won't engage further since we seem to be talking alongside each other.

 

[martin_j]

I can confirm that ZFS+Unprivileged LXC + Docker works flawlessly. See here. I have 7 Unprivileged LXC with Docker nested inside each, with a total of about 25 docker containers, running on ZFS Volumes formatted as XFS. No problems since 2 years. Even heavier containers (Gitlab, Nextcloud etc.) work without problems. I do not see any performance issues either, my CPU utilization is 1-2% on average since every process is run on the host, and most processes are idle most of the time. I would expect that this is different with full VMs.

Don't suppose you've had any luck with shared bind mounts. Can't seem to get it to work in LXC (priviledged or unprivileged) but does work in a VM. Also have the same issue when using docker from snap, so suspect it's some namespace or similar issue (https://forums.docker.com/t/docker-from-snap-on-ubuntu-cant-create-a-shared-bind-mount/119981).

Easy to check

Not working
# docker run -it --rm -v /tmp:/tmp:shared alpine ash docker: Error response from daemon: path /tmp is mounted on / but it is not a shared mount. ERRO[0000] error waiting for container: context canceled


Working (well, obviously also need to check that the /tmp directory is shared between host and docker container but it didn't fail at first hurdle
# docker run -it --rm -v /tmp:/tmp:shared alpine ash / #

 

[Helmut101]

I don't - I haven't had the need for shared bind mounts. Have you enabled fuse in the Container Proxmox options (under Features)? The link you gave refers to virtual fuse mounts produced inside the container, linked back to the host. If it is an issue, I think this may not have something to do with ZFS, since you have the same problem with privileged LXC.

 

[martin_j]

I don't - I haven't had the need for shared bind mounts. Have you enabled fuse in the Container Proxmox options (under Features)? The link you gave refers to virtual fuse mounts produced inside the container, linked back to the host. If it is an issue, I think this may not have something to do with ZFS, since you have the same problem with privileged LXC.

Ah yes, sorry should have said, I had fuse option selected.
No, pretty certain it's nothing to do with ZFS as have the same issue on a pve which isn't using ZFS.
Oh well, I'll try a few other things and if I can't get it working I'll just use a VM (which is a shame).
Cheers

 

[jaskohl]

I vote for VM with BurmillaOS on it. With the death of RancherOS there is a replacement called BurmillaOS. basically a fork of rancheros with continued development.