LXC and Docker compatibility now

[Thendeo]

I'm new to Proxmox, started to play a bit with, and currently loving it !

I'm using LXCs on a dedicated server to support various services for myself. Still, I have one that is using Docker (Affine.pro self-hosted setup is simpler with it). I saw some threads about using Docker under LXC not being that good (here for example), but other saying it is fine (here for example). On the official doc the only reference to docker is about the keyctl flag.

Currently I've got no trouble, but it seems that it is during updates that it can raise quite some headache.

I'm struggling to get up to date information :
- Are LXC and Docker fine during updates now ?
- What is the current state of the known issues regarding this topic ?

Best,
Thendeo

 

[Impact]

Here's my current thoughts/opinions about running docker/podman in CTs

- It currently works fine (at least in a debian CT)
- It might break on docker (related) or kernel updates
- Breakage is likely to be fixed soon but you'll have to be patient and find workarounds until then or hold off on upgrades (Test in a virtual PVE first)
- It is often recommended against by proxmox staff and you will have to live with hearing this whenever this is discussed
- I find the combined benefits of LXC CTs and docker (I use podman) too good not to use so I use it myself anyways

It's a compromise. If you want a stable/reliable environment to run docker/podman on you should use a VM. If you're comfortable with fixing potential breakage I think it's fine to do in a homelab.

 

[Johannes S]

Are LXC and Docker fine during updates now ?

No and propably never will be since they use the same features of the kernel ( namespaces, cgroups etc) and host system ( apparmor etc ) so any update might introduce a breaking change. This is more likely with major upgrades ( like PVE8 to PVE9 ) but can happen at any other update too since a bug fix might also have unintended side effects.
Please note that most of the time nothing bad happens. But at some point it will break like due to a change in AppArmor and lxcs between PVE8 and PVE9:
https://forum.proxmox.com/threads/d...bian-13-host-worked-fine-on-proxmox-8.169508/

https://forum.proxmox.com/threads/d...rivileged_port_start-error.175437/post-814307

If you have good knowledge of Linux and containers don't mind potential breakage and troubleshooting nobody can force you not to put docker/podman in lxcs. It might be even useful for playing around and trying new things without spinning up a vm. But if you mostly want host a service with as small frictions as possible you are better off with vm. If you want to save resources it makes sense to put all docker/podman container in the same vm.

Only you can decide what's acceptable for you or not.

 

[Johannes S]

Something I forgot: ProxmoxVE9 introduced a new, still in tech-preview-feature which allows converting oci ( aka docker ) container images to lxcs. To be honest I'm not a big fan since this procedure destroys the immutability which is the biggest benefit of docker/podman compared to lxcs. It also still doesn't work for everything and is a setup most developers of the applications propably won't test or support. But if you are resource-constrained you might give it a shot

 

[proxuser77]

The thing with setups that aren't officially recommended or supported by the developers of a product is that they're unsupported for a reason: they simply aren't something the developers explicitly test or validate.

For example, when the Proxmox developers roll out kernel or other system updates, they aren't going to verify that every possible Docker-in-LXC setup continues to work flawlessly. That means you often only find out whether something breaks after installing an update. Alternatively, you can delay updates and wait for someone else to run into the issue first and report it on the forums.

So, to answer your questions:

Are LXC and Docker fine during updates now?

See above. Nobody can guarantee that future updates won't introduce temporary issues. In most cases, there are at least workarounds and eventually (more or less) proper fixes. However, if you're not familiar enough with the underlying technologies to troubleshoot these issues yourself and instead rely on solutions posted by others, then it is possible that one or more of your Docker containers could be down for a few days until you find a fix or workaround.

Does this happen every other week? No. But it can happen.

What is the current state of the known issues regarding this topic?

If your setup is working today, then there are currently no issues with your setup.

But again, that doesn't mean there won't be any issues in the future, or that every possible edge case will work flawlessly. There's also no official or complete documentation covering Docker inside LXC, precisely because it isn't an officially supported configuration. Most of the time it works just fine, but if you happen to run into a corner case, you'll often have to piece together a solution from forum posts and other community resources.

So my recommendation would be:

Run all of your Docker containers in one or two VMs. In practice, this doesn't require significantly more resources than running them inside LXCs, as long as you group related containers together. Of course, if you create a separate VM for every single Docker application, resource usage scales much worse.

 

[alexskysilk]

The thing with setups that aren't officially recommended or supported by the developers of a product is that they're unsupported for a reason: they simply aren't something the developers explicitly test or validate.

This is true, but also irrelevant. The devs made a hypervisor; they neither test nor care about any specific software that runs within it. A debian or alpine lxc container is fully "supported" as far as that goes.

I've been running docker inside LXC for years anywhere I need to pass a hardware resource (eg, gpu.) While this does introduce some complexity in terms of correctly mapping resources to the container and coordinate driver and utility versions between guest and host, it doesn't pose any real challenge even for updates (presuming the drivers for the devices are supported by the pve kernel; even if they're not you're allowed to pin boot kernel until thats resolved.)

Whether or not this solution is appropriate for your use case I couldnt say, as I dont know what you are using it for. Suffice to say that home use doesn't suffer from business loss on outage.

 

[proxuser77]

This is true, but also irrelevant. The devs made a hypervisor; they neither test nor care about any specific software that runs within it. A debian or alpine lxc container is fully "supported" as far as that goes.

Why is it irrelevant? I actually think it's highly relevant, arguably the most relevant point with respect to the OP's questions.

What the OP is essentially asking—or at least that's how I understand it—is whether the kinds of issues that occurred in the past are now unlikely to happen again ("Are LXC and Docker fine during updates now?") and whether there are any currently known issues ("What is the current state of the known issues regarding this topic?").

The underlying hope seems to be that they can use Docker inside LXCs without worrying about future updates and, ideally, without ever running into the kinds of issues described in the threads they linked.

And no, nobody can give a definitive answer to that as long as the Proxmox developers don't officially support Docker inside LXC containers.

Sure, you can do it, and plenty of people do. But the status quo regarding official support hasn't changed. Therefore, nobody can honestly say that it definitely won't cause problems in the future.

In other words, there really isn't much more to say than what has already been said hundreds of times in previous discussions on this topic.

Whether or not this solution is appropriate for your use case I couldn't say, as I don't know what you are using it for. Suffice it to say that home use doesn't suffer from business loss on outage.

Exactly. But even if the OP tells us what they want to use it for, we still can't decide for them whether occasional downtime would be acceptable. Only they can make that judgment.

And it doesn't have to be a business use case, either. Even in a home environment, the acceptable level of downtime depends entirely on the service. Some services can be unavailable for a few days without anyone caring, while for others, even a few hours of downtime can be a major inconvenience.

 

[6equj5]

What is the current state of the known issues regarding this topic ?

Here the biggest problem isn't the LXC but Docker itself which breaks often things on updates.

 

[proxuser77]

Here the biggest problem isn't the LXC but Docker itself which breaks often things on updates.

Yes, but that is less likely to happen when Docker is running on bare metal or inside a VM. The reason is that nobody involved is specifically developing and testing for this exact use case (Docker inside LXC on Proxmox). As long as that remains the case, nobody can guarantee that there won't be issues that only occur in this specific combination of technologies.

Of course, Docker itself can also have bugs, that's a different topic. No software is completely bug-free. The difference is that a very specific issue that only occurs when running Docker inside an LXC container on Proxmox may not have the same priority, or might not even be considered a bug by the Docker project.

The same applies from the Proxmox side. They are not going to delay a kernel patch that benefits their enterprise customers just because it might potentially cause issues for some home users that installed everything via community Docker/LXC scripts.

 

[6equj5]

... but that is less likely to happen when Docker is running on bare metal or inside a VM.

Believe it or not. I do not.
Just upgraded PVE to 9.2.5. upgraded Trixie-LXC, everything is fine.

 

[alexskysilk]

Why is it irrelevant?

Because PVE "supports" the hypervisor and base OS. they DONT SUPPORT USERSPACE APPLICATIONS. You might as well suggest you shouldnt use MySQL since the PVE devs dont support that either.

The only times in recent memory where PVE updates "broke" docker were when they migrated from cgroupv1 to v2, and when there was a syntax change in lxc.conf configurations for passthrough. These issue are not docker specific and would have impacted any container with affected base OS or passthrough configuration. Since docker is certainly subject to the latter, it would have required configuration file changes to resume functionality.

Here the biggest problem isn't the LXC but Docker itself which breaks often things on updates.

This has more to do with good production habits then docker specifically. As long as you stick to "stable," both OS and docker version, I dont see any real potential for breakage. Dont mess with it, it wont break.

 

[Johannes S]

I might be wrong but the breakage at the PVE8 to PVE9 upgrade was with docker in lxcs and was AppArmor related if I recall correctly. And it's not the same as with other applications since docker is a orchestrator for running them so a breakage there will hurt more than some other random application. Except of course if you follow the recommendations and host docker/podman et all in a vm

 

[alexskysilk]

I might be wrong

You're memory is spot on- but, again, nothing specific to docker. The PVE philosophy seems to be move fast and break things- thats just the cost of doing business on a rolling platform- there will be changes.

 

[proxuser77]

You're memory is spot on- but, again, nothing specific to docker.

Ah, come on. Docker is a container platform that relies on many of the same Linux kernel features as LXC. You don't even have to be an expert on the underlying technologies to realize that this is fundamentally different from running just any application inside a container.

The PVE philosophy seems to be move fast and break things- thats just the cost of doing business on a rolling platform- there will be changes.

Not really. Proxmox is a hypervisor and a container manager that officially supports LXC, but it isn't a Docker manager.

And, to be honest, why is everyone so keen to run Docker inside LXC? Is it due to memory constraints? This isn't really an issue if you run all your Docker containers in a single VM. Or is it because you want separate backups for each Docker application via individual LXC backups? Okay, I can at least understand that argument. My suspicion is that, in many cases, the popularity of the community helper scripts has a lot to do with it.

Oh, and by the way, many applications can also be installed natively inside an LXC. Plex provides a Debian repository, Jellyfin does as well, and Nextcloud can be installed completely natively, although I'd probably run Nextcloud in a VM anyway if it's exposed to the internet. And many more...

But yeah, I get it. Docker is pretty much the standard these days for self-hosted applications in a homelab, and I use it too for many things.

If you're going to use it, though, I'd recommend learning Docker properly and running it in the environment it was designed for. Or, if you prefer to run it inside an LXC, then just accept that you may occasionally run into quirks that are specific to that setup.

Or, put more simply:

Inside a VM, you mainly need to understand Docker and its quirks. Inside an LXC, you also need to understand Proxmox, LXC, and how all of those layers interact.

 

[alexskysilk]

You don't even have to be an expert on the underlying technologies to realize that this is fundamentally different from running just any application inside a container.

Is that so? I am not an expert. perhaps you can explain. there is nothing "fundamental" seperating docker from other userspace software.

And, to be honest, why is everyone so keen to run Docker inside LXC?

Because in a non production environment it is the sweet spot of application deployability with namespace seperation with minimal resource overhead. PLUS, it is the only rational way to get resource sharing in a manner impossible with virtual machines (eg, you can offer CUDA services to multiple containers with a single GPU, but only a single VM exclusively.)

Oh, and by the way, many applications can also be installed natively inside an LXC. Plex provides a Debian repository, Jellyfin does as well, and Nextcloud can be installed completely natively, although I'd probably run Nextcloud in a VM anyway if it's exposed to the internet. And many more...

All true. but all of them are easier to install, easier to control, and easier to manage the lifecycle by using docker. The PVE devs settled on LXC as a general purpose replacement to virtuozzo, but that was many years ago. LXC is designed for lightweight GP VM use; docker is an application lifecycle mechanism.

Inside a VM, you mainly need to understand Docker and its quirks. Inside an LXC, you also need to understand Proxmox, LXC, and how all of those layers interact.

thats true.... so what? I suggest that if the complexity is undesirable, you're using the wrong tool in the first place. I will agree that in the age of docker the hypervisor ITSELF is less relevant, and I have made the argument in the past that if your applications are all dockerized you really have no need of PVE at all.

 

[proxuser77]

ITSELF is less relevant, and I have made the argument in the past that if your applications are all dockerized you really have no need of PVE at all.

Yeah, you actually got a point with this. And backups are straightforward as well with Docker because, in principle, you only need to back up your Compose files, your .env files, and your volumes. After restoring them on a new server, you can usually just start the containers again.

I use Proxmox because I primarily run VMs. My setup is a mix of different workloads, some of them in Docker containers some of them natively installed, with network segmentation between services that I want to isolate. Those services live in different VMs, which in turn are attached to different network segments. That said, I don't run every single application in its own VM.

Anyway, I'm not saying that people shouldn't use Docker inside LXC. My point is simply that there's no guarantee it will always work flawlessly. And no, I'm not an expert on the low-level details of these technologies either. That's actaually one of the reasons why I don't run Docker inside LXC, besides the fact that VMs provide better isolation, of course. I don't want to spend my time after an apt update && apt full-upgrade searching the Proxmox forums because, after a reboot, my containers suddenly refuse to start.

 

[6equj5]

And, to be honest, why is everyone so keen to run Docker inside LXC?

Only some are. Because everyone else says it's awkward, dangerous, whatever.

I like LXC because in ZFS their volumes are datasets and not ZVOLs. So I can manage files from a single pane of glass. The host.

I don't want to spend my time after an apt update && apt full-upgrade searching the Proxmox forums because, after a reboot, my containers suddenly refuse to start.

Actually we have lots of posts here "help, my VMs crash after PVE updates" ...

I have made the argument in the past that if your applications are all dockerized you really have no need of PVE at all.

And Docker will be superseded by K8S eventually.

 

[proxuser77]

Only some are. Because everyone else says it's awkward, dangerous, whatever.

Yeah, I actually find it kind of awkward. Dangerous, not so much, if you know what you're doing.

Actually we have lots of posts here "help, my VMs crash after PVE updates" ...

I've never had a VM crash after a Proxmox update, and I've been using it for about 10 years.

And Docker will be superseded by K8S eventually.

Not on my lawn.

But yeah, I guess in enterprise environments Kubernetes has probably already superseded Docker in many cases, or rather, Docker was never really much of a thing there to begin with. That said, I doubt many enterprises are running Kubernetes inside LXCs.

By the way, Proxmox appears to be developing its own Docker alternative these days: https://pve.proxmox.com/wiki/Roadmap#Virtualization_&_Guest_Management

 

[alexskysilk]

Docker=the mechanism for shipping a container
Kubernetes=Orchestration tool for containers

They're not in conflict or "better" then each other.

 

[proxuser77]

Docker=the mechanism for shipping a container
Kubernetes=Orchestration tool for containers

They're not in conflict or "better" then each other.

Yeah, it's not quite that simple. But LXC, was definitely never designed as an orchestration tool for Docker containers.

Anyway, do whatever works for you. You're going to do that anyway.