Proxmox 7 Docker stoped working

[Vasilij Lebedinskij]

Hello! Recently I've updated one of my servers to PVE 7 and encountered a problem with on of LXC containers on it. There was a docker service UNMS (management software for network equipment) and it stoped working. In syslog I've found error

failed to start daemon: error initializing graphdriver: driver not supported

LXC config:

arch: amd64
cores: 4
features: nesting=1
hostname: unms-srv
memory: 4096
net0: name=eth0,bridge=vmbr0,firewall=1,gw=172.16.100.1,hwaddr=86:A7:81:E8:E0:5D,ip=172.16.100.35/24,tag=4000,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-zfs:subvol-116-disk-0,size=16G
swap: 4096
lxc.cgroup.devices.allow: a
lxc.cap.drop:

Any help would be appreciated.

 

[Vasilij Lebedinskij]

I think I've found general error in syslog

Dec 24 03:40:52 unms-srv modprobe[153]: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.11.22-5-pve/modules.dep.bin'
Dec 24 03:40:52 unms-srv modprobe[153]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.11.22-5-pve

However

root@pve:~# lsmod | grep overlay
overlay               126976  0

 

[LnxBil]

Any help would be appreciated.

Also the help that says: Don't run Docker on your hypervisor? Every PVE update can break it, because it is not supported and will never be supported. Also, your lxc.cgroup.devices.allow: a opens up your system to be hacked from inside the LX(C) container, which is a big no-go.

 

[Vasilij Lebedinskij]

Also the help that says: Don't run Docker on your hypervisor? Every PVE update can break it, because it is not supported and will never be supported. Also, your lxc.cgroup.devices.allow: a opens up your system to be hacked from inside the LX(C) container, which is a big no-go.

I don't like docker either but this particular service is available in docker only... I've figured out that problem is in aufs module which is no longer available in PVE kernel.

 

[LnxBil]

I don't like docker either but this particular service is available in docker only... I've figured out that problem is in aufs module which is no longer available in PVE kernel.

aufs is no longer available in the kernel and it is deprecated in Docker since 2019.

Docker perfectly works inside of an VM (KVM/QEMU) without any problems and with maximum security (compared to running it on the hypervisor)

 

[joshfindit]

Here's an experience I found so interesting that I registered a forum account to share it:

I had the exact same error ("Error starting daemon: error initializing graphdriver: driver not supported"), but it was also preceded by

[graphdriver] prior storage driver overlay2 failed: driver not supported

.

But in my case this happened on a normal reboot with zero updates applied.

Thankfully, I remembered I did something out of the ordinary while testing in the previous 12 hours: I had created a named pipe (using `mkfifo`) inside an Ubuntu LXC container.


I deleted it and the docker instances inside all the containers started working again.


Yes I know: don't run Docker inside LXC. But I'm comfortable with the tradeoff of it breaking at some point if it means I can have all the dockerfiles exposed on the local filesystem and don't have to have the overhead of 10+ VMs.

 

[RolandK]

i would not recommend running docker inside pve host OS or inside LXC. Better use a VM for that.

 

[LnxBil]

i would not recommend running docker inside pve host OS or inside LXC. Better use a VM for that.

welcome to the choir

 

[peter70]

i would not recommend running docker inside pve host OS or inside LXC. Better use a VM for that.

why VM is better for that?
I'm using CT for that and I feel it's ok, I wonder what I don't know about it,I'm no expert in that field

 

[RolandK]

docker in lxc is more likely to break because of updates

 

[LnxBil]

docker in lxc is more likely to break because of updates

... and the speed.

I also use it inside LX(C) containers and on the hypervisor itself and it makes a huge difference if your are e.g. on ZFS. On the hypervisor itself, Docker can use ZFS as its underlying data store, which creates each container layer as snapshotted dataset and adding stacks is very simple an just uses the COW architecture of ZFS, whereas without ZFS inside the LX(C) container, you currently cannot store directly inside of ZFS due to the fact that ZFS is currently not capable of delegating nested features to containers without breaking security. You have to create overlayfs layers, which are not so efficient and fast so that your Docker inside of LXC works a little bit slower. I'm looking forward to having this capability inside of LXC which will be a game changer for performance.

At home, I run Home Assistant inside of LCX Docker because I wanted a separated network stack for HA and just pulling the image is magnitudes slower than pulling the image on the hypervisor. In bigger environments I just run VMs for docker with ZFS inside.