Docker to PVE-LXC conversion steps/tool?

[dlasher]

Wondering if anyone has a script/directions/etc to convert docker images into Proxmox Compatible LXC containers? Several projects I use are only available as docker containers, and I'd rather run it in an LXC than run docker JUST for them.

I've found a few resources online that deal with individual pieces, but haven't found someone doing the whole chain.

Anyone?


RESOURCES:
* https://www.reddit.com/r/Proxmox/comments/hagkvu/more_lxc_templates/
* https://www.buzzwrd.me/index.php/2021/03/10/creating-lxc-containers-from-docker-and-oci-images/
* https://github.com/chriswayg/proxmox-templates

 

[BobhWasatch]

Why not set up a VM and run all of your docker images there?

 

[dlasher]

Why not set up a VM and run all of your docker images there?

doing that already.

KVM has:
more disk space overhead
more cpu overhead
more ram overhead
doesn't directly offer peripheral reuse (shared gpus)

especially on smaller boxes (4-8 cores) running a thin LXC is much preferred to a fat KVM.

That being said, that's how I'm running Frigate right now, in it's own KVM. I'd like to get away from that model.

 

[Dunuin]

Running Frigate as a docker container inside an LXC is also an option. Not recommended, but works.

 

[BobhWasatch]

KVM has:
more disk space overhead
more cpu overhead
more ram overhead
doesn't directly offer peripheral reuse (shared gpus)

But does it really?

• More disk space? You can run Alpine Linux as a VM in a couple hundred MB.
• More CPU overhead? A few 10ths of a percent. Assuming your docker actually does something this is very small.
• More RAM? Not really. Again, Alpine uses just a couple of MB for itself, much less than your app will use.

Peripheral reuse is the only one that is actually significant on any but the most absurdly small machines. If you run multiple dockers in the same VM then the overhead gets amortized over all of them and is really quite trivial. I know people here obsess over chasing "efficiency" but it is almost never worth the effort expended.

Just my opinion.

 

[dlasher]

B

Running Frigate as a docker container inside an LXC is also an option. Not recommended, but works.

Been down that road. Got docker-swarm working in LXC-priv containers. Preferred it to KVM, but you can't snapshot them to back them up (long story) so moving away from that as well.

99.9% of what is running in my lab & home clusters are LXC containers. (even moved plex/jellyfin into containers) - frigate is my holdout, so trying to find a way to move it as well.

 

[dlasher]

But does it really?

• More disk space? You can run Alpine Linux as a VM in a couple hundred MB.
• More CPU overhead? A few 10ths of a percent. Assuming your docker actually does something this is very small.
• More RAM? Not really. Again, Alpine uses just a couple of MB for itself, much less than your app will use.

Peripheral reuse is the only one that is actually significant on any but the most absurdly small machines. If you run multiple dockers in the same VM then the overhead gets amortized over all of them and is really quite trivial. I know people here obsess over chasing "efficiency" but it is almost never worth the effort expended.

Just my opinion.

I appreciate your viewpoint, and when it's other people's network, money, and datacenter space, I agree, it's not as big of a deal. When you have cores to burn, there's not as much difference between the two.

In my case, I'm moving my home lab from 5 machines, 160 cores, 1280G of ram, and 40 OSD's on CEPH to ... something a bit smaller. So now the difference between KVM containers that don't share cpu or ram very well, and don't share hardware at all... to LXC containers that do, is a bigger deal than when I was throwing CPU & RAM at everything.

( as an aside, there's a 5-node proxmox cluster likely to be for sale at some point.. Is there a for-sale forum? )

 

[orwadira]

I think that what @dhalsher is attempting to do is indeed valid.

I was personally interested in doing the same thing and would have liked it if people addressed what I was inquiring about.

Below is one way I discovered to do what @dhalsher wanted to do which is to convert a Docker container to an LXC container template in Proxmox.

In the case of Docker images representing distributions and that have a shell entrypoint, this can be done in the following way:
1. Using Docker, pull the distribution image which you want to convert to an LXC (.tar.gz) template using the docker pull command
2. Run the docker image using docker run -id <image>
3. Once the image is running, keep note of the container ID you see as the result of the previous command, call it <container>
4. Export the root filesystem within the running container to a tarball using docker export --output dockerexport.tar <container>
5. Compress the tarball using gzip < dockerexport.tar > dockerexport.tar.gz
6. Rename the now-LXC-template and move it to the templates directory on the root Proxmox node Debian system (usually at /var/lib/vz/template/cache)

These steps can also be performed using the following one liner (using the bash Docker image as example):

docker export $(docker run -id bash) | gzip > bash.tar.gz

To perform these steps, you will need a VM that has Docker installed and enough disk space for converting big images.

To move the resulting template (.tar.gz) file to the templates directory on the node's root Debian system: you can either mount the filesystem of the VM using pct mount and then copy the file while on the root Debian system, or use more conventional methods like scp (which can be called from the VM or the root system).

An example SCP invocation from the VM:

scp bash.tar.gz root@<ip-address-of-node>:/var/lib/vz/template/cache

I followed this method to successfully convert the universal devcontainer Docker image from Microsoft to an LXC container template, which worked nicely. Note that for a big image like this one (10GB), it makes more sense to run the steps separately rather than using a one liner.

After the conversion and transferring the resulting template, do not forget to:
1. Kill the Docker container using docker kill <container>
2. Delete the docker image using docker image remove <image>
3. Delete the tar and tar.gz files

 

[orwadira]

I have taken a look into the frigate Docker container and it requires more work but I think I can convert it for you if you are keen to do so.

 

[dlasher]

10 months after my initial post, I have everything moved to LXC containers, and I'm 99% happy.. EXCEPT...

Frigate is still running in docker in an LXC container, and while it's "thinner" than a KVM instance + Docker, I'd still love to be able to drop it back to native LXC, lose docker, at some point.

I appreciate the direction @orwadira is taking, and I still think docker-to-lxc is worth the time & effort. I'm going to give his approach a try when I have a few minutes free.. I'd welcome & potentially support any scripts/project that could do a "1 line" conversion.. Something like

#docker2lxc.sh blakeblackshear/frigate templates/frigate.template.gz

Where the output would be a template you could place in the folder, and create any new VM's off the template. That would ... <chefs kiss> sweet..

LIGHT READING:
* https://tteck.github.io/Proxmox/#frigate-lxc
* https://github.com/blakeblackshear/frigate/discussions/5773
* https://www.reddit.com/r/frigate_nvr/comments/1ewn3m6/does_frigate_exist_outside_of_docker_in_2024/

 

[dlasher]

As a side note, because it needs attention, ttech, who created the fantastic helper scripts I've used repeatedly, and reference above, passed away Nov 2024.

https://github.com/community-scripts/ProxmoxVE/discussions/237

If you liked his scripts, used them, consider buying a coffee, dropping a donation to his memorial, which his wife gets directly:

* https://ko-fi.com/proxmoxhelperscripts

 

[orwadira]

As a side note, because it needs attention, ttech, who created the fantastic helper scripts I've used repeatedly, and reference above, passed away Nov 2024.

https://github.com/community-scripts/ProxmoxVE/discussions/237

If you liked his scripts, used them, consider buying a coffee, dropping a donation to his memorial, which his wife gets directly:

* https://ko-fi.com/proxmoxhelperscripts

Certainly did not expect to hear about the death of one of the community members as part of my first interaction within these forums. RIP tteckster. His contributions graph on GitHub speaks loudly of the weight of his abrupt loss:



Did someone know him personally? I wish that his family is not left without financial support.

Speaking of the helper script he wrote, for deploying Frigate within of a privileged LXC container. I went through the code quickly and there does not seem to be any mention of Docker.

Is it true that he managed to run it without Docker?

And if so, then why did you say that Frigate:

... is still running in docker in an LXC container,

I am just trying to understand where the community is on this point.

Disclaimer: I have never used Frigate before. I am just interested in the technical aspects of running Docker containers as LXC containers in Proxmox

 

[orwadira]

Before this exchange, I got as far as this user https://forum.proxmox.com/threads/running-as-pid-1.144068/ trying to get Frigate to run as an LXC container. Namely, I converted the root filesystem, injected the configurations, mounted the media storage, but not able to run the /init script yet.

 

[dlasher]

Certainly did not expect to hear about the death of one of the community members as part of my first interaction within these forums. RIP tteckster. His contributions graph on GitHub speaks loudly of the weight of his abrupt loss:

View attachment 80241

Beautiful representation of his contributions. Would make a good framed picture for his wall.

Speaking of the helper script he wrote, for deploying Frigate within of a privileged LXC container. I went through the code quickly and there does not seem to be any mention of Docker.

Is it true that he managed to run it without Docker?

Not sure, I didn't use his script for Frigate. I just stood up a new LXC container, jumped through the hoops to pass iGPU / Coral through, then stood up docker inside and ran Frigate.

Important pieces look like this.

arch: amd64
cores: 4
cpulimit: 6
features: mount=nfs,nesting=1
hostname: frigate-lxc
onboot: 1
swap: 128
tags: nvr
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 120:0 rwm
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/apex_0 dev/apex_0 none bind,optional,create=file

 

[orwadira]

Beautiful representation of his contributions. Would make a good framed picture for his wall.

Maybe we should create a digital poster that shifts it with time and mail it over to his wife. This would be a good place to get volunteers to make such a poster.

Not sure, I didn't use his script for Frigate.

Honestly, you should. I just ran it on the node's root Debian system. It is interactive, and works beautifully without any hiccups. It takes a while, but at the end, it is able to create an LXC container with Frigate running on it:


I used the following command to be able to connect to the Frigate's UI from my local computer:

ssh -N -L 5000:192.168.1.59:5000 proxmox

(Note that I configured the alias proxmox in my SSH config file to connect me to the root Debian system on the proxmox node)

From this point on, I was able to access Frigate without login at http://localhost:5000/:


It is clear that the person who wrote the script ensured an excellent user experience, for example:
1. The UI interface did not complain about not using https as it did when I ran it using the starter configuration using docker
2. I did not have to sieve through the logs to find the randomly-generated admin password
3. A camera feed was already configured for testing and evaluation purposes

So, it is a really good time to give this script a try and remember the man behind it.

Finally: the resulting LXC container does not use Docker and is built on your chosen base. Both Ubuntu and Debian are offered as options. I chose Debian 11 which matches the base of the official Docker image.

Since I do not use this software and since it seems that someone else (may mercy be upon his soul) has put the time and energy to convert this application to an LXC container. I think am going to stop hackathoning to convert the official Docker container to an LXC image.

 

[dlasher]

Amazing, thank you, I'll give it a shot.

 

[orwadira]

The lesson I learnt from this exercise is that docker2lxc, if ever built, will likely have to process the Dockerfile directly, rather than pull and convert the Docker image (like I did above).

My original approach, however, is quite nice and performant for simpler images:

function docker2lxc {
  echo -e "\e[33m-> Pulling Docker container '$1'...\e[0m"
  docker pull $1
  if [[ $? -ne 0 ]]; then echo "\e[31m  Container '$1' not found, aborting\e[0m" && exit 1; fi
  output_file=${2:-lxc_container_template}
  output_file=${output_file%.tar.gz}.tar.gz
  echo -e "\e[33m-> Exporting root filesystem to '$output_file'...\e[0m"
  docker export $(docker run --entrypoint sh -id $1) | gzip > $output_file
  echo -e "\e[33m-> Killing running container...\e[0m"
  docker kill $(docker ps -q -n 1)
  echo -e "\e[32;1mDone\e[0m"
}

Example invocation:


PS: Please do not use the script above as it has some bugs. A better documented version can be found here: https://github.com/diraneyya/docker2lxc