Docker inside LXC (net.ipv4.ip_unprivileged_port_start error)

Jul 21, 2022
8
4
8
Be careful when updating Docker on Proxmox LXC machines (as far as I know): when you run apt update&&apt upgrade, among other packages, containerd.io is updated to 1.7.28-2~debian.13~trixie, so if you try to launch your containers, you will get the following error:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied: unknown

So your containers aren't starting up. The solution that has worked for me in my LXC with Debian 13 is as follows:

Bash:
apt install -y --allow-downgrades containerd.io=1.7.28-1~debian.13~trixie

So we switched from version 1.7.28-2 to 1.7.28-1. After that, everything works.

I understand that this is a temporary solution until Docker fixes the issue or until there is an update to LXC.

EDIT:
Proxmox bug report submitted:
https://bugzilla.proxmox.com/show_bug.cgi?id=7006

Workaround:
https://github.com/opencontainers/runc/issues/4968#issue-3593655843

Fix that has been implemented upstream in Incus as well:
https://github.com/lxc/incus/pull/2624
 
Last edited:
Be careful when updating Docker on Proxmox LXC machines (as far as I know): when you run apt update&&apt upgrade, among other packages, containerd.io is updated to 1.7.28-2~debian.13~trixie, so if you try to launch your containers, you will get the following error:



So your containers aren't starting up. The solution that has worked for me in my LXC with Debian 13 is as follows:

Bash:
apt install -y --allow-downgrades containerd.io=1.7.28-1~debian.13~trixie

So we switched from version 1.7.28-2 to 1.7.28-1. After that, everything works.

I understand that this is a temporary solution until Docker fixes the issue or until there is an update to LXC.
Yes, debian 12 has the same issue and 1.7.29-1 still not working, i did apt-mark hold containerd.io .
 
It depends, some containers start, some don't. Plex starts fine, but Immich bombs out. They are actively working on it. I'm assuming the fix will come upstream of Proxmox and in the containerd.io package. I can confirm that the 1.7.29-1 package works with some images, but not others.
 
Obligatory reminder: Don't run docker/podman containers inside lxcs, except you think troubleshooting after updates is fun.
From ProxmoxVE Dokumentation zu lxcs:
If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox QEMU VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.
https://pve.proxmox.com/wiki/Linux_Container


It is a well-known problem that docker containers in LXC containers occasionally break during major updates. Both use the same parts of the kernel, so this is not surprising. Some examples:



In most cases, a bug fix or a (more or less ugly) workaround (usually involving the deactivation of security mechanisms, which is not a “solution” in my book) was eventually provided, allowing the system to function again. If (like me) you don't want to take that risc or prefer boredom to troubleshooting, install docker/podman containers in a VM, you will have far less issues. If, for whatever reason, you prefer to use a Linux container, check whether the instructions for the software you want to run describe how to get it working without Docker. For example running pihole or jellyfin inside an lxc without docker works quite well.
 
The problem I have with spinning up a VM to run a docker container is RAM consumption. Having one LXC container per docker container is very nice in terms of system resources.

Anyway, I will get a VM going.

Historically how long does Docker take before they fix the problem?
 
  • Like
Reactions: lynze
Yes, it's true that LXCs only consume what they need, while VMs take up all the resources allocated to them from the moment they start up.

It's a shame that LXCs have these problems. I prefer them for containers.
 
That problem of "resource allocation" isn't one really. Of course if you want to have one VM for each docker container you want to run you will end up in using more RAM (but not neccesarily way more see https://pve.proxmox.com/wiki/Dynamic_Memory_Management#KSM ). But normally you wouldn't do this but run all your docker containers in one lightweight VM. My main docker Debian Trixie vm is configured with 4 GB RAM, right now it uses 1.5 GB. And this can propably reduced even more without changing anything, since Linux always uses part of the memory as cache. By changing the VM os to alpine an even more lightweight VM should be possible. Another benefit of fitting all docker containers in one vm is, that you need the system maintenance (like updates etc) only once instead of doing housekeeping for every lxc instance.
I prefer to save on my time budget instead of saving RAM for the sake of saving RAM.
But if for the sake of "saving resources" you prefer to waste your private time by trouble shooting after breaking changes be my guest.
 
Last edited:
  • Like
Reactions: lynze
That problem of "resource allocation" isn't one really. Of course if you want to have one VM for each docker container you want to run you will end up in using more RAM (but not neccesarily way more see https://pve.proxmox.com/wiki/Dynamic_Memory_Management#KSM ). But normally you wouldn't do this but run all your docker containers in one lightweight VM. My main docker Debian Trixie vm is configured with 4 GB RAM, right now it uses 1.5 GB. And this can propably reduced even more without changing anything, since Linux always uses part of the memory as cache. By changing the VM os to alpine an even more lightweight VM should be possible. Another benefit of fitting all docker containers in one vm is, that you need the system maintenance (like updates etc) only once instead of doing housekeeping for every lxc instance.
I prefer to save on my time budget instead of saving RAM for the sake of saving RAM.
But if for the sake of "saving resources" you prefer to waste your private time by trouble shooting after breaking changes be my guest.
You're right, I currently use three LXCs, and each one has about 30 Docker containers. Perhaps it would be a good idea to create a couple of VMs to perform the same functions.

I can see that it might be better to use LXC to run single or small services without using containers.
 
  • Like
Reactions: Johannes S
But if for the sake of "saving resources" you prefer to waste your private time by trouble shooting after breaking changes be my guest.
No matter what the reasons are. I have my own, and it's not resources.

There is a clear statement on the LXC containers wiki page why VMs are recommended for docker, but not a word that docker containers are not supported in LXC containers.
So in my mind the current issue is a 100% valid support case.
 
  • Like
Reactions: tcesar
I can see that it might be better to use LXC to run single or small services without using containers.
Exactly.. Something like pi-hole which is self-containing (so doesn't need network shares and fiddling around with bind mounts) or jellyfin (which can be installed with apt install under Debian) where I want to use the host hardware (for jellyfin iGPU) but also share it with other lxcs/the host (I could passtrough the iGPU to a VM but then I wouldn't have a console anymore for trouble shooting if the WebInterface gets broken) are great usecases for LXC.
My monitoring software (icinga2 + graphite at the moment, I'm considering switching to Prometheus+grafana though) is also installed in a lxc, since everything was installed with adding the icinga+graphite repos followed by an apt install $package.

Another usecase is if your install of ProxmoxVE is on a vserver or inside a vm without support for nested virtualization so you can still have several environments for testing/playground/seperating applications. Maybe you want to have several instances of the same application (like a webserver for wordpress, one instance for testing/developing, another one for production).
I remember that one of the Proxmox developers mentioned at some point, that one of their customers is running multiple hundreds of lxcs on the same host, which all contain the same software stack to host the website of the schools in their region:

In fact, I know production setups handling school websites which has >1500 CT on two PVE nodes (plus one quorum node).


Please note that the quote is from 2018, so right now even higher numbers might be possible (I don't know whether the mentioned technical limit of maximal 3000 vm or lxcs in one cluster are still relevant) Interesting enough in the same topic fellow community member sprit (who also developed the lvm snapshot support introduced in PVE 9.0) mentioned that at that time he run 200 VMs on a 2x12 core node (he didn't mention the available memory tohough). Anyhow I think it's save to assume that even on a Mini-PC-"Homeserver" the available resources should be enough, given you plan accordingly how to utilice them ;)

Another example:
On a technical conference I met a guy who is using lxcs as generic tool to provide his customers with managed webservers. I asked him why he don't use VMs (since they allow downtime-free migrations and provide better isolation). He pointed out that this allows him to fit more customers on one host, and lxc downtimes are small to beginn with. For any customer who can't tolerate even some seconds of downtime in case of an migration he would use a loadbalancer like caddy or haproxy. What managed webhosting usecases all have in common, that they are not using docker containers but are basically multiple instances of the same basic system (a Linux distribution with an installed webserver, scripting languages (php5/python/ruby etc) and a sql database). So the lxc containers are to isolate the customers from each other but still allow dynamical allocation of the resources of the host system. Since the system is nearly identical mentioned webhosting admin also automated the maintenance with ansible to avoid the overhead of manual administration of every container.

Obviouvsly you could also use this approach for VMs, it really depends which tradeoffs in terms of resource usage/security considerations/maintenance etc you are willing to take. German webhoster uberspace run their plattform on CentOS VMs in a ProxmoxVE cluster managed with ansible. They write a lot about their setup in https://blog.uberspace.de/ (if you understand German). Basically every vm has the same setup of CentOS (they plan to migrate to Arch but they are not there yet) with a webserver, several scripting languages and mariadb/postgres. The webserver is configured for shared webhosting, so on each VM they have a couple hundred users with their websites, several of these vms are on one node and all nodes together build their ProxmoxVE cluster. The same setup could propably also be done with LXCs to fit even more customers , but the VMs provide stricter isolation which is a good thing from a security point of view.

Any of the mentioned corporate scenarios (with lxcs or vms) are however not typical for homelabs imho ;)
 
The problem I have with spinning up a VM to run a docker container is RAM consumption. Having one LXC container per docker container is very nice in terms of system resources.

Anyway, I will get a VM going.

Historically how long does Docker take before they fix the problem?
This is the reason I mainly use them, less resource usage.
 
Exactly.. Something like pi-hole which is self-containing (so doesn't need network shares and fiddling around with bind mounts) or jellyfin (which can be installed with apt install under Debian) where I want to use the host hardware (for jellyfin iGPU) but also share it with other lxcs/the host (I could passtrough the iGPU to a VM but then I wouldn't have a console anymore for trouble shooting if the WebInterface gets broken) are great usecases for LXC.
My monitoring software (icinga2 + graphite at the moment, I'm considering switching to Prometheus+grafana though) is also installed in a lxc, since everything was installed with adding the icinga+graphite repos followed by an apt install $package.

Another usecase is if your install of ProxmoxVE is on a vserver or inside a vm without support for nested virtualization so you can still have several environments for testing/playground/seperating applications. Maybe you want to have several instances of the same application (like a webserver for wordpress, one instance for testing/developing, another one for production).
I remember that one of the Proxmox developers mentioned at some point, that one of their customers is running multiple hundreds of lxcs on the same host, which all contain the same software stack to host the website of the schools in their region:



Please note that the quote is from 2018, so right now even higher numbers might be possible (I don't know whether the mentioned technical limit of maximal 3000 vm or lxcs in one cluster are still relevant) Interesting enough in the same topic fellow community member sprit (who also developed the lvm snapshot support introduced in PVE 9.0) mentioned that at that time he run 200 VMs on a 2x12 core node (he didn't mention the available memory tohough). Anyhow I think it's save to assume that even on a Mini-PC-"Homeserver" the available resources should be enough, given you plan accordingly how to utilice them ;)

Another example:
On a technical conference I met a guy who is using lxcs as generic tool to provide his customers with managed webservers. I asked him why he don't use VMs (since they allow downtime-free migrations and provide better isolation). He pointed out that this allows him to fit more customers on one host, and lxc downtimes are small to beginn with. For any customer who can't tolerate even some seconds of downtime in case of an migration he would use a loadbalancer like caddy or haproxy. What managed webhosting usecases all have in common, that they are not using docker containers but are basically multiple instances of the same basic system (a Linux distribution with an installed webserver, scripting languages (php5/python/ruby etc) and a sql database). So the lxc containers are to isolate the customers from each other but still allow dynamical allocation of the resources of the host system. Since the system is nearly identical mentioned webhosting admin also automated the maintenance with ansible to avoid the overhead of manual administration of every container.

Obviouvsly you could also use this approach for VMs, it really depends which tradeoffs in terms of resource usage/security considerations/maintenance etc you are willing to take. German webhoster uberspace run their plattform on CentOS VMs in a ProxmoxVE cluster managed with ansible. They write a lot about their setup in https://blog.uberspace.de/ (if you understand German). Basically every vm has the same setup of CentOS (they plan to migrate to Arch but they are not there yet) with a webserver, several scripting languages and mariadb/postgres. The webserver is configured for shared webhosting, so on each VM they have a couple hundred users with their websites, several of these vms are on one node and all nodes together build their ProxmoxVE cluster. The same setup could propably also be done with LXCs to fit even more customers , but the VMs provide stricter isolation which is a good thing from a security point of view.

Any of the mentioned corporate scenarios (with lxcs or vms) are however not typical for homelabs imho ;)
I do use LXC for Docker and individual services like Technitium and avahi (because Unifi's implementation isn't that good).
 
Last edited:
Tutorals Docker Compose Fix Debian oder Ubuntu Bug

apt install docker.io

fehler
/tmp/apt-dpkg-install-gYJjDA/08-docker-buildx_0.13.1+ds1-3_amd64.deb

Sub-process /usr/bin/dpkg returned an error code(1)

sudo dpkg --configure -a


apt update && apt upgrade

apt dist-upgrade
 
Last edited: