OCI Images in LXC (Release 9.1)

P

PigLover

Renowned Member
Apr 8, 2013
142
50
93
Having the ability to import OCI images and run them in an LXC is fabulous! Thank you Proxmox team. This is a useful and necessary first step to a container runtime environment that may one day reduce the desire to run Docker/Podman VMs (or, though not advised, LXCs).

Unfortunately, this initial capability provides only a very static model of importing and running an OCI container. As valuable as that is, the OCI model really shines in a much more dynamic container orchestration environment. In order to get to that with PVE several more things needed:
  • The abiliity to update/replace and the image from the OCI repo
  • The ability to "mount" volumes into the container (PVE already provides wonder virtual disk integrations that could help develop this)
  • A solid integration of PVEs SDN model with LXC containers
  • The ability to specify Environment Variables for the container
  • A good way for containers to read the state of the Proxmox environment state and subscribe to updates in that state so that tools like Traefik, etc., could automate exposing ports
  • Likely some extensions to the existing PVE "labels" concepts.
Some of these may be major - but many of them are reasonably achievable.

Thoughts? Am I missing something? Is it silly to go this direction (i.e., does the world really need another container orchestration platform when you can just run Docker/Podman/K8s inside Proxmox today)?
 
  • Like
Reactions: Fosco and daanw
Mounting host folders is already possible using pct set or directly editing the LXC config file after creation. (i.e. mp0: /test,mp=/test)
It would be more straightforward to expose this in the GUI, but there are security concerns:
B

Thread 'Bind mounts and API'

Hello,

It this possible to create a mount between the host and a guest by editing the lxc configuration file.
cat "mp0: /srv/directory,mp=/srv/directory" >> /etc/pve/lxc/<id>.conf​
it seems to be called a bind mount reading the page https://pve.proxmox.com/wiki/LXC_Bind_Mounts

Since 4.2 it is possible to create a mount in the graphical assistant or with the API but actually it does something different by creating a separate disk. (By the way this is not clear to me what the expected JSON structure to pass.)

So I like to know if it possible to create a bind mount through...

Environment variables are available in the Options tab once the LXC is created, but currently not during initialization.

What you see here is that you want to be able to define these during creation in order to successfully deploy the OCI container at once. Pretty much like you currently do in for example docker compose or Portainer.

I also wonder how this could would work with stacked containers like for example Immich. Interesting use case for Proxmox OCI.

Non-destructive updating of OCI containers is obviously the most important and will probably be the most requested missing feature. A roadmap would be great!
 
Last edited:
  • Like
Reactions: smalltrex, Phyrene, vesalius and 1 other person
regarding the bindmounts like in docker. just use the gui and the additional mountpoints. you are creating a seperated "docker volume" like an image and mount it inside. it does not matter where it is on the host. just setup the storage in the datacenter.

sure, for shared data between lxc/oci (how does it called now? lxc docker - im confused already) you need bindmounts via host. but then you need to check the permissions between the container.
PigLover said:
The ability to specify Environment Variables for the container
Click to expand...
it is under the options tab, there you can enter your key-value-pair
PigLover said:
  • A good way for containers to read the state of the Proxmox environment state and subscribe to updates in that state so that tools like Traefik, etc., could automate exposing ports
  • Likely some extensions to the existing PVE "labels" concepts.
Click to expand...
had this in mind, too. i love traefik for docker stuff. maybe i'll fiddle arround with, shoot myself into my foot or someone else's.
maybe you need to use the file provider/dynamic router for this. im not sure.
 
Last edited:
Reading this post, I would like to understand how to add these LXC OCI... for example, like Immich, Jellyfin, etc... Sorry, I am new and I would like to understand and learn more.
 
der_baer said:
keep in mind, this is not a fully fledged alternative for a vm with docker etc. for self hosting. you will have far better expierence sticking to the old stuff.
Click to expand...
This. There is a reason, why it's still technology preview. If you want to host oci-image-based applications who just work, stick with docker-vms. If however you want to experiment, help to test things and improving the new oci-to-lxc feature, you can try it out
 
Last edited:
  • Like
Reactions: smalltrex, der_baer, Kingneutron and 2 others
I have now tested numerous OCI images (mostly from docker hub, but many also from ghcr) & I am absolutely astounded at how easy they are to spin up & get running. I love the easy environment-inject feature in the GUI.

I find it somewhat underwhelming that there is not more forum responses/queries/requests on this feature (presently tech-preview) - because I really believe this is the way to move forward for Proxmox - which inevitably will make for a more robust system, rather than all those hacks/workarounds users do to their running systems in order not to use docker in its' own VM (that is still perfectly reasonable, but usually a waste of resources).

I think the most single needed additional feature to attract a wider user-base to this new awesome OCI containerization system, would be the ability to run multiple containers in tandem (single lxc, with parallel access) - as most docker users would normally do.

I'd like to thank, once again, the whole Proxmox team for this absolutely awesome new feature, & also for all their general stunning work!

Kudos!
 
  • Like
Reactions: templar
gfngfn256 said:
which inevitably will make for a more robust system, rather than all those hacks/workarounds users do to their running systems in order not to use docker in its' own VM (that is still perfectly reasonable, but usually a waste of resources).
Click to expand...
That’s probably the reason they’re doing it: because there is demand. ;)

However, the issue of resource usage with VMs is somewhat exaggerated. The overhead is not nearly as significant as many people believe, and if you run multiple containers/applications in the same VM, this “issue” becomes even less relevant.

gfngfn256 said:
I find it somewhat underwhelming that there is not more forum responses/queries/requests on this feature (presently tech-preview)
Click to expand...
Because, believe it or not, home servers can also be production servers, and most applications, especially those typically used in home labs and self-hosting environments, are primarily built with Docker in mind, and the documentation almost always refers to Docker. In other words, Once you've migrated your family to Nextcloud, Immich, etc., this stuff has to work, and then you stick to the documentation of the applications you're using to minimize problems. I'll test it at some point, but it's not a high priority for me.

gfngfn256 said:
I think the most single needed additional feature to attract a wider user-base to this new awesome OCI containerization system, would be the ability to run multiple containers in tandem (single lxc, with parallel access) - as most docker users would normally do.
Click to expand...
I don't want to dampen your excitement, but Docker has been around for many years now, and you can't catch up in terms of features in just a few weeks or months. And I also doubt their goal is to recreate Docker 1:1. Which may not even be possible for technical reasons. But I could of course be wrong here.
 
Last edited:
  • Like
Reactions: PigLover and Johannes S
As I've played with this more I am even more excited about the possibilities for this. At the same time, more convinced this current capability is just a pre-requisite to the desired solution rather than the solution itself (thus, I am sure, why the Proxmox team is only listing this as a tech preview at that time).

The most pressing issue, for me, compared to Docker/Podman/K8s/etc use of OCI images, is the lack of "idempotent restart". As currently constructed, the LXC container created from the OCI image can be modified, stopped, and then restarted with those changes still in place. This breaks the expected operating model for these types of images. I think this could be "fixed" with something like the existing "linked clone" model Proxmox uses for VMs. Something along the lines of launching the container as a "linked clone" to the image and - upon termination - having that linked clone automatically destroyed. If the container is restarted then a new linked clone is created, etc.

This isn't fully thought out - just first thoughts.

This might also allow for a method to do in-place upgrades/downgrades as is done in Docker - create a new download of the OCI image (read only - idempotent), stop the running container, and then create a new container "linked clone" from the new image.

There are still some more UI/API updates needed to do something akin to "bind mounts" and network config - but this could go a long way to allowing Proxmox to be a full player in the OCI-based container image game.
 
I find the inclusion of OCI containers a lazy first attempt. More to the point, it offers very little beyond running docker in a container (which is fairly simple to do) but unlike that approach you dont have any granular control. Also, the need to predownload the OCI image is itself a departure of the docker model, which only downloads what you need when you need it, and has integrated image lifecycle controls.

From a design standpoint, I would probably not bother with OCI at all. Instead, build a special deployable lxc "template" that has docker and portainer included, and integrate hooks that can spin additional docker templates directly through portainer. This is already how I manager docker in my deployments (less the integrated hooks... would be very nice to have callable through the pve api.)
 
alexskysilk said:
I find the inclusion of OCI containers a lazy first attempt. More to the point, it offers very little beyond running docker in a container (which is fairly simple to do) but unlike that approach you dont have any granular control. Also, the need to predownload the OCI image is itself a departure of the docker model, which only downloads what you need when you need it, and has integrated image lifecycle controls.
Click to expand...

It offers one big advantage though: It's supported by the developers and expected not to break after updates. While docker inside lxcs is known for breakage and thus not recommended in the documentation: https://pve.proxmox.com/wiki/Linux_Container

Now we both know this but I think it's worth to point this out to any novices stumbling on this thread.

I agree with you that the new lxc feature breaks the imututable nature of docker containers and is also a reason I'm actually not that much interested in this feature. But I can see a potential benefit: Such a limited docker support might still be enough for homeusers, to run their typical services directly in PVE so we don't need to deal with people who run into issues due to running docker inside lxcs, using a helper script which does non-supported things etc anymore. Whether this feature might actually reduce support issues or raise new ones (maybe even more) can't be said at the moment though, so I guess we just have to wait and see ;)


alexskysilk said:
From a design standpoint, I would probably not bother with OCI at all. Instead, build a special deployable lxc "template" that has docker and portainer included, and integrate hooks that can spin additional docker templates directly through portainer. This is already how I manager docker in my deployments (less the integrated hooks... would be very nice to have callable through the pve api.)
Click to expand...

LXC template is problematic due to the regular breakage of docker inside lxcs due to upstream changes (e.g. the kernel, apparmor, whatever). A VM template might be the better idea. But imho it's not even needed since it's actually quite simple to spin up a new vm with docker and portainer. And somebody else might even prefer podman or some other container runtime for oci.
 
Last edited:
  • Like
Reactions: alexskysilk
Johannes S said:
LXC template is problematic due to the regular breakage in due to upstream changes
Click to expand...
You hit the nail on the head of why LXC is itself not a dependable method of containerization. As an operator, you have a choice- lightweight but far more in need of maintenance (and not appropriate for high security/multitenant application,) or heavier weight and dependable. in EITHER case, PVE was designed before Docker and has never had a good approach for integration with it. seems like this dovetails onto that other thread discussing LXC live migration- which is to say, if you need HA containers use something else.
 
  • Like
Reactions: Johannes S
Johannes S said:
Yes and I would say that a proper integration isn't even needed since in the end ProxmoxVE is about virtualization which imho is a completly different layer than application containers. But this was already discussed several times to dead before, see this thread, which up to now reached an amount of ten (mostly redundant) pages: https://forum.proxmox.com/threads/docker-support-in-proxmox.27474/
Click to expand...
I think that direct support of Docker (the subject of that older referenced discussion) and the support of Docker-like features using OCI Containers are two totally different subjects - clearly related, but also clearly seperate topics.

There are solid reasons for Proxmox not to support Docker, mostly having to do with unresolvable conflicts regarding how system packages and methods are used (apparmor and iptables, among others).

It is a completely seperate topic to consider how Proxmox might support well standardized objects (OCI Containers) in a way doesn't result in these conflicts and is consistant with how Proxmox is designed overall.

There are significant problems with the current methods of using Docker on Proxmox, especially WRT exposing devices, the "weight" of VMs to do "container things", and the extent problems with Docker on LXC. A coherent OCI runtime model on Proxmox makes a lot of sense. And the inclusion of the the OCI-on-LXC tech preview suggests that Proxmox team may at least be interested in exploring this option.

Hopefully this can remain a productive discussion about what it may take to do that in a rational way rather than a dismissive "VMs are good enough, why bother with anything new" rant.
 
You must log in or register to reply here.
Top Bottom