LXC and Docker compatibility now

[Johannes S]

But yeah, I guess in enterprise environments Kubernetes has probably already superseded Docker in many cases, or rather, Docker was never really much of a thing there to begin with. That said, I doubt many enterprises are running Kubernetes inside LXCs.

It not only supersedes docker but also the hypervisor. You don't necessarily need a hypervisor if you have a way to deploy Kubernetes clusters baremetal. A hypervisor might still make sense if you want to distribute resources between different tenants or teams (like have one cluster for your developers and another one for production etc). And yes, nobody runs Kubernetes inside lxcs in enterprise because it would only add another layers complexity without any benefit. In other words: Kubernetes in lxcs is pointless.

By the way, Proxmox is kind of developing its own Docker alternative these days: https://pve.proxmox.com/wiki/Roadmap#Virtualization_&_Guest_Management

Yes and I really would wish they wouldn't waste resources on this endeavour, to quote the roadmap

Mature OCI image support and provide declarative multi-guest application support based on the Compose Specification, allowing groups of LXC application containers (and, longer-term, VMs) to be defined, deployed, and lifecycle-managed as a single application unit directly from Proxmox VE.
https://pve.proxmox.com/wiki/Roadmap#Virtualization_&_Guest_Management

I might be wrong but for me this sounds like extending the existing tech-preview feature of converting oci images to lxcs. Despite usage in resource-constrained homelabs I don't see any point in this. The biggest advantage of oci images in podman/docker/kubernetes is their immutability. Meaning if something bad (Tm) happens you just shut down the container, restart it and it will be in the same state as the image. This benefit gets lost, if they are converted to lxcs. While I can see a (although limited) usecase in homelabs are there actually any serious IT environments where admins like to run oci containers in a way which is neither intended not supported by the original creaters of said oci image ? For me this sounds like a potential neverending source of frustations of debugging why some config error in a oci-converted-to-lxc stays and the upstream can't help with it.

Now deploying Vms as InfrastructureASCode is indeed useful but wouldn't it make more sense to have an official terraform/opentofu provider?

 

[proxuser77]

Yes and I really would wish they wouldn't waste resources on this endeavour, to quote the roadmap

Maybe the idea is to make it a competitor to microVMs, LXD and/or Incus, or something like that. I don't know.

Now deploying Vms as InfrastructureASCode is indeed useful but wouldn't it make more sense to have an official terraform/opentofu provider?

That would probably be useful for larger customers.

That said, to me it looks like they're trying to carve out a niche in between: something for small and medium-sized businesses that want more automation than just clicking through the "Create VM/CT" dialogs, but don't necessarily want to bring in the heavy artillery like Terraform or OpenTofu. So perhaps the idea isn't to replace or integrate with those tools, but to offer a simpler alternative for a different audience. Whether that niche actually exists, I have no idea. And... I could be completely wrong.

 

[proxuser77]

Actually, looking back at what I wrote, a lot of it doesn't really make sense.

Proxmox LXC is already an alternative to LXD/Incus, and microVMs are, simply put, more like application VMs, a bit like Docker containers, but with their own kernel. So, if you look at it that way, it is actually closer to being an alternative to Kubernetes.

A more accurate way to put it would probably be:

To me it looks like they're trying to carve out a niche in between: something for small and medium-sized businesses that want high availability, but don't necessarily want to bring in the heavy artillery like Kubernetes. So perhaps the idea is to offer a simpler alternative to Kubernetes for a different audience. Whether that niche actually exists, I have no idea. And... I could be completely wrong.

 

[Johannes S]

Incus/lxd is for managing lxcs and qemu- vms so basically the same usecase as ProxmoxVE with the same tech stack beneath it. Support for qemus MicroVMs would be nice indeed, kind of sweet spot in terms of isolation and resources usage compared to lxcs and full blown vms

 

[alexskysilk]

Yeah, it's not quite that simple

You might want to expound on that to make a cogent point.

 

[proxuser77]

Docker is a container engine together with a set of tools for building, distributing, and managing application containers. Docker includes image build tools, but so do Podman, Buildah, BuildKit, and others.

Kubernetes doesn't care how a container image was built. It simply schedules and manages containers using a compatible container runtime. So Kubernetes is an orchestration platform, not a container engine or an image build tool.

Proxmox is also an orchestration platform, but primarily for virtual machines and LXC system containers. More recently, it has also started adding support for OCI application containers, so we'll see where that goes.

When you run Docker inside an LXC, you're layering two container technologies. You're effectively running application containers inside a system container, which adds another layer of complexity. It can work perfectly well, but it wasn't the original design goal of either technology.

More importantly, though, Proxmox doesn't officially support this setup. That doesn't necessarily mean it won't work, but it does mean the Proxmox developers are unlikely to perform regression testing specifically for this use case. As a result, things may unexpectedly break after updates.

That said, you can absolutely run it this way, it's not forbidden. But neither you nor anyone else here can honestly give the OP a definitive "yes" or "no" as to whether it will always work reliably. That's the whole point I was trying to make.

 

[alexskysilk]

Docker is a container engine together with a set of tools for building, distributing, and managing application containers. Docker includes image build tools, but so do Podman, Buildah, BuildKit, and others.

Good. so for the purpose of conversation, Docker is promarily the container topology- since the manner by which you run them has options.

Kubernetes doesn't care how a container image was built. It simply schedules and manages containers using a compatible container runtime. So Kubernetes is an orchestration platform, not a container engine or an image build tool.

With you so far. still dont quite get whats "not that simple."

When you run Docker inside an LXC, you're layering two container technologies.

True. but this isnt actually a problem, since a container is simply a path to the kernel with guardrails. both lxc and docker reach for the same resources, so as long as the guardrails encompass the needs of both there's no issue. As for added complexity... To put things in perspective, the complexity amounts to a couple of lines in ctid.conf that are not exposed in the gui. If the operator chooses to run docker in LXC thats just the cost of doing business. As mentioned prior, if the desire is for a baked docker appliance there are better options (eg, truenas, etc.)

It can work perfectly well, but it wasn't the original design goal of either technology.

LXC has provisions to expand the guardrails within the design. I dont know how that means it wasnt "the original design" since the mechanism is built in.

And again, Proxmox doesn't officially support this setup. That doesn't necessarily mean it won't work, but it does mean the Proxmox developers are unlikely to perform regression testing specifically for this use case. As a result, things may unexpectedly break after updates.

At the risk of repeating myself- Proxmox doesnt "officially" support any software beyond theirs. If the intention is to have commercial support, this is a non starter; while technically possible I guess, docker on lxc isnt an enterprise ready method.

But neither you nor anyone else here can honestly give the OP a definitive "yes" or "no" answer about whether it will always work reliably. That was the point I was trying to make.

And we get to the bottom of it.

No one can give you a definitive "yes" or "no" about whether ANY SOFTWARE will always work reliably. The best commercial support is still reactive- you find an edge case, they work through it. fix the code or make a workaround, submit to devs for inclusion in a patch/next release if applicable, document, and move on. The more mature a software package is (years in production without major changes) the less edge cases you'll run into.

PVE has release lifetime of ~3 years. that is REALLY short and aggressive. If you require long term stability, might want to reconsider PVE as your substrate.

 

[Johannes S]

More recently, it has also started adding support for OCI application containers, so we'll see where that goes.

I wouldn't call "converting oci-images to lxc" support though.

 

[proxuser77]

At the risk of repeating myself- Proxmox doesnt "officially" support any software beyond theirs. If the intention is to have commercial support, this is a non starter; while technically possible I guess, docker on lxc isnt an enterprise ready method.

However, they explicitly recommend against running Docker inside an LXC and instead suggest using a VM.

I really don't understand why we have to go around in circles here. I couldn't care less whether you, the OP, or anyone else runs Docker inside an LXC. You could install Proxmox inside a VM on Proxmox, create another VM inside that, install LXD via Snap in that VM, and finally run Docker inside that container. I genuinely don't care.

It's just not something I'd recommend as a general-purpose setup. That's all I'm saying.

And when someone explicitly brings up examples of problems that have occurred in the past and essentially asks, "Have these issues been solved? Can I assume they won't happen again in the future?", you simply can't answer "yes" in good conscience.

Clear enough now?

 

[alexskysilk]

crystal

so @proxuser77 does not recommend using docker in lx, because the PVE devs are on record recommending against it.

And when someone explicitly brings up examples of problems that have occurred in the past and essentially asks, "Have these issues been solved? Can I assume they won't happen again in the future?", you simply can't answer "yes" in good conscience.

actually that one is easier. You can. known problems are addressed, either in code or in workflow.

 

[proxuser77]

crystal

so @proxuser77 does not recommend using docker in lx, because the PVE devs are on record recommending against it.


actually that one is easier. You can. known problems are addressed, either in code or in workflow.

Yeah, if that’s what they wanted, they probably would have asked for instructions on what to do if it happens again, rather than whether it could happen again, don’t you think?

And if you think Proxmox should take care of this, feel free to open a ticket on their Bugzilla.

Oh, and you don't have to convince me either. I'm certainly not opposed to running Docker inside LXCs, even though I don't do it myself. But when someone asks a question like the author of the original post did, I definitely won't endorse it as a generally recommended setup.

 

[6equj5]

Not on my lawn.

Not today, not tomorrow. But in two or three years I won't bet against it.

I doubt many enterprises are running Kubernetes inside LXCs.

Neither would I. Talos seems a good way. For testing / learning as VM, but in the long run bare metal on some small devices for high availability.

Right now I have decreasingly VMs (only one running always) and increasingly services/apps (container), 17 running and a lot more projects.
I don't see me "returning" to VMs in the near future. And I bet most people / companies are on the same path. Because developers are. At least those with the solutions I want to run.