[SOLVED] LXC + docker, containerd cannot find /lib/modules/4.15.18-9-pve


Dec 3, 2018
Hi all,

Been stuck on this for quite some time and looking for some feedback. I'm running Proxmox 5.2 and I currently have 6 LXC Ubuntu 18.10 containers. 1 hosts rancher, 5 others host k8s worker/etcd/controller machines. I've been using the unsecure mode outlined here

http-s://ww-w.solaris-cookbook.eu/virtualisation/proxmox/proxmox-lxc-running-docker-inside-container/ (had to break the link due to being a new forum user)

to change apparmor etc. settings. This has worked for a month, including after updating the container's packages. I am now trying to add a 6th container and I run into an issue running docker:

docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

This led me to seeing containerd failing to startup:

Dec 03 20:53:00 docker-test2 systemd[1]: Starting containerd container runtime...
Dec 03 20:53:00 docker-test2 modprobe[4636]: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.15.18
Dec 03 20:53:00 docker-test2 modprobe[4636]: modprobe: FATAL: Module overlay not found in directory /lib/modules/4.15.18-9-pve
Dec 03 20:53:00 docker-test2 systemd[1]: containerd.service: Control process exited, code=exited status=1
Dec 03 20:53:00 docker-test2 systemd[1]: containerd.service: Failed with result 'exit-code'.
Dec 03 20:53:00 docker-test2 systemd[1]: Failed to start containerd container runtime.

I am installing docker-ce from docker's repository. In my /lib/modules for proxmox I of course have 4.15.18-9-pve, but in the LXC container I only have 4.18.0-12-lowlatency. I saw that this was actually installed during the installation of docker-ce, dependency "linux-image-unsigned-4.18.0-12-lowlatency" and "linux-modules-4.18.0-12-lowlatency". In the containers that are working I have "4.18.0-1003-kvm".

So I tried installing the different package:

apt install linux-image-4.18.0-1003-kvm linux-modules-4.18.0-1003-kvm

and the "4.18.0-1003-kvm" folder is available in /lib/modules, but containerd is still looking for "4.15.18-9-pve".

pveversion: pve-manager/5.2-12/ba196e4b (running kernel: 4.15.18-9-pve)
docker-ce: :18.09.0~3-0~ubuntu-bionic
LXC template: ubuntu-18.10-standard_18.10-1_amd64.tar.gz
LXC's uname: Linux docker-test2 4.15.18-9-pve #1 SMP PVE 4.15.18-30 (Thu, 15 Nov 2018 13:32:46 +0100) x86_64 x86_64 x86_64 GNU/Linux

Proxmox /etc/pve/lxc/110.conf:

arch: amd64
cores: 2
hostname: docker-test.example.com
memory: 1024
net0: name=eth0,bridge=vmbr0,gw=,hwaddr=02:6E:BD:2A:1F:25,ip=,type=veth
ostype: ubuntu
rootfs: local-lvm:vm-100-disk-0,size=8G
swap: 0
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a

Any insight would be great; I am not seeing how my process is any different from a month ago and I took pretty good notes. I am still splunking around.

Magical, thank you! I had created two test containers both experiencing the same problem, and doing this on one ALSO fixed it in the second. I am not sure what to make of that. This seems to stick over container reboots too. Thanks!
Basically your containers are running the host kernel, therefore we're just bind-mounting the host kernel modules into the container (readonly). You'll need to repeat this step after an pve host kernel update.

i run into the same problem. But i find a different solution which not depends on the host kernel.

There is a service called containerd which starts before docker
this service tries to modprobe the overlay Modul

cat /lib/systemd/system/containerd.service
Description=containerd container runtime

ExecStartPre=/sbin/modprobe overlay
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.

so in the LXC container you can create a file
nano /etc/systemd/system/containerd.service.d/override.conf
with content:
systemctl daemon-reload
service docker restart
and the docker service will be running
  • Like
Reactions: wasp

I have another Problem with the actual Docker Version:

The new Version of Docker CE in Debian Stretch


which has the new containerd Service, keeps the LXC from doing a complete shutdown.

When I shutdown the LXC Container it ends up in an undefined state. Not really stopped and not really running.
It keeps the filesystems mounted.
I had to completley shutdown the host to get rid of this.

But it is not only affecting LXC Containers, also KVM-VM won't shutdown completely.

I tested the previous Version of Docker CE


in a KVM-VM and it did a complete shutdown


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!