[SOLVED] LXC + docker, containerd cannot find /lib/modules/4.15.18-9-pve

[Klox]

Hi all,

Been stuck on this for quite some time and looking for some feedback. I'm running Proxmox 5.2 and I currently have 6 LXC Ubuntu 18.10 containers. 1 hosts rancher, 5 others host k8s worker/etcd/controller machines. I've been using the unsecure mode outlined here

http-s://ww-w.solaris-cookbook.eu/virtualisation/proxmox/proxmox-lxc-running-docker-inside-container/ (had to break the link due to being a new forum user)

to change apparmor etc. settings. This has worked for a month, including after updating the container's packages. I am now trying to add a 6th container and I run into an issue running docker:

docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

This led me to seeing containerd failing to startup:

Dec 03 20:53:00 docker-test2 systemd[1]: Starting containerd container runtime...
Dec 03 20:53:00 docker-test2 modprobe[4636]: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.15.18
Dec 03 20:53:00 docker-test2 modprobe[4636]: modprobe: FATAL: Module overlay not found in directory /lib/modules/4.15.18-9-pve
Dec 03 20:53:00 docker-test2 systemd[1]: containerd.service: Control process exited, code=exited status=1
Dec 03 20:53:00 docker-test2 systemd[1]: containerd.service: Failed with result 'exit-code'.
Dec 03 20:53:00 docker-test2 systemd[1]: Failed to start containerd container runtime.

I am installing docker-ce from docker's repository. In my /lib/modules for proxmox I of course have 4.15.18-9-pve, but in the LXC container I only have 4.18.0-12-lowlatency. I saw that this was actually installed during the installation of docker-ce, dependency "linux-image-unsigned-4.18.0-12-lowlatency" and "linux-modules-4.18.0-12-lowlatency". In the containers that are working I have "4.18.0-1003-kvm".

So I tried installing the different package:

apt install linux-image-4.18.0-1003-kvm linux-modules-4.18.0-1003-kvm

and the "4.18.0-1003-kvm" folder is available in /lib/modules, but containerd is still looking for "4.15.18-9-pve".

pveversion: pve-manager/5.2-12/ba196e4b (running kernel: 4.15.18-9-pve)
docker-ce: :18.09.0~3-0~ubuntu-bionic
LXC template: ubuntu-18.10-standard_18.10-1_amd64.tar.gz
LXC's uname: Linux docker-test2 4.15.18-9-pve #1 SMP PVE 4.15.18-30 (Thu, 15 Nov 2018 13:32:46 +0100) x86_64 x86_64 x86_64 GNU/Linux

Proxmox /etc/pve/lxc/110.conf:

arch: amd64
cores: 2
hostname: docker-test.example.com
memory: 1024
net0: name=eth0,bridge=vmbr0,gw=10.0.0.1,hwaddr=02:6E:BD:2A:1F:25,ip=10.0.0.60/24,type=veth
ostype: ubuntu
rootfs: local-lvm:vm-100-disk-0,size=8G
swap: 0
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:

Any insight would be great; I am not seeing how my process is any different from a month ago and I took pretty good notes. I am still splunking around.

Thanks!

 

[morph027]

Try;

pct set xxx --mp0 /lib/modules/$(uname -r),mp=/lib/modules/$(uname -r),ro=1

 

[Klox]

Magical, thank you! I had created two test containers both experiencing the same problem, and doing this on one ALSO fixed it in the second. I am not sure what to make of that. This seems to stick over container reboots too. Thanks!

 

[morph027]

Basically your containers are running the host kernel, therefore we're just bind-mounting the host kernel modules into the container (readonly). You'll need to repeat this step after an pve host kernel update.

 

[der-brumm-baer]

Hello,

i run into the same problem. But i find a different solution which not depends on the host kernel.

There is a service called containerd which starts before docker
this service tries to modprobe the overlay Modul

cat /lib/systemd/system/containerd.service

[Unit]
Description=containerd container runtime
Documentation=htxxps://containerd.io
After=network.target

[Service]
ExecStartPre=/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
KillMode=process
Delegate=yes
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity

[Install]
WantedBy=multi-user.target

so in the LXC container you can create a file

nano /etc/systemd/system/containerd.service.d/override.conf

with content:

[Service]
ExecStartPre=
ExecStartPre=

do

systemctl daemon-reload
service docker restart

and the docker service will be running

 

[der-brumm-baer]

Hello,

I have another Problem with the actual Docker Version:

The new Version of Docker CE in Debian Stretch

18.09.0~3-0~debian-stretch

which has the new containerd Service, keeps the LXC from doing a complete shutdown.

When I shutdown the LXC Container it ends up in an undefined state. Not really stopped and not really running.
It keeps the filesystems mounted.
I had to completley shutdown the host to get rid of this.

But it is not only affecting LXC Containers, also KVM-VM won't shutdown completely.

I tested the previous Version of Docker CE

18.06.1~ce~3-0~debian

in a KVM-VM and it did a complete shutdown