Proxmox 7 Docker stoped working

Vasilij Lebedinskij

Active Member
Jan 30, 2016
64
2
28
36
Hello! Recently I've updated one of my servers to PVE 7 and encountered a problem with on of LXC containers on it. There was a docker service UNMS (management software for network equipment) and it stoped working. In syslog I've found error

Code:
failed to start daemon: error initializing graphdriver: driver not supported

LXC config:

Code:
arch: amd64
cores: 4
features: nesting=1
hostname: unms-srv
memory: 4096
net0: name=eth0,bridge=vmbr0,firewall=1,gw=172.16.100.1,hwaddr=86:A7:81:E8:E0:5D,ip=172.16.100.35/24,tag=4000,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-zfs:subvol-116-disk-0,size=16G
swap: 4096
lxc.cgroup.devices.allow: a
lxc.cap.drop:

Any help would be appreciated.
 

Vasilij Lebedinskij

Active Member
Jan 30, 2016
64
2
28
36
I think I've found general error in syslog

Code:
Dec 24 03:40:52 unms-srv modprobe[153]: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.11.22-5-pve/modules.dep.bin'
Dec 24 03:40:52 unms-srv modprobe[153]: modprobe: FATAL: Module overlay not found in directory /lib/modules/5.11.22-5-pve

However
Code:
root@pve:~# lsmod | grep overlay
overlay               126976  0
 

LnxBil

Famous Member
Feb 21, 2015
5,621
651
133
Germany
Any help would be appreciated.
Also the help that says: Don't run Docker on your hypervisor? Every PVE update can break it, because it is not supported and will never be supported. Also, your lxc.cgroup.devices.allow: a opens up your system to be hacked from inside the LX(C) container, which is a big no-go.
 

Vasilij Lebedinskij

Active Member
Jan 30, 2016
64
2
28
36
Also the help that says: Don't run Docker on your hypervisor? Every PVE update can break it, because it is not supported and will never be supported. Also, your lxc.cgroup.devices.allow: a opens up your system to be hacked from inside the LX(C) container, which is a big no-go.
I don't like docker either but this particular service is available in docker only... I've figured out that problem is in aufs module which is no longer available in PVE kernel.
 

LnxBil

Famous Member
Feb 21, 2015
5,621
651
133
Germany
I don't like docker either but this particular service is available in docker only... I've figured out that problem is in aufs module which is no longer available in PVE kernel.
aufs is no longer available in the kernel and it is deprecated in Docker since 2019.

Docker perfectly works inside of an VM (KVM/QEMU) without any problems and with maximum security (compared to running it on the hypervisor)
 

joshfindit

New Member
Jan 6, 2022
2
0
1
43
Here's an experience I found so interesting that I registered a forum account to share it:

I had the exact same error ("Error starting daemon: error initializing graphdriver: driver not supported"), but it was also preceded by
Code:
[graphdriver] prior storage driver overlay2 failed: driver not supported
.

But in my case this happened on a normal reboot with zero updates applied.

Thankfully, I remembered I did something out of the ordinary while testing in the previous 12 hours: I had created a named pipe (using `mkfifo`) inside an Ubuntu LXC container.


I deleted it and the docker instances inside all the containers started working again.


Yes I know: don't run Docker inside LXC. But I'm comfortable with the tradeoff of it breaking at some point if it means I can have all the dockerfiles exposed on the local filesystem and don't have to have the overhead of 10+ VMs.
 

RolandK

Member
Mar 5, 2019
100
10
23
48
i would not recommend running docker inside pve host OS or inside LXC. Better use a VM for that.
 

peter70

New Member
Oct 31, 2021
8
1
3
51
i would not recommend running docker inside pve host OS or inside LXC. Better use a VM for that.
why VM is better for that?
I'm using CT for that and I feel it's ok, I wonder what I don't know about it,I'm no expert in that field :)
 

LnxBil

Famous Member
Feb 21, 2015
5,621
651
133
Germany
docker in lxc is more likely to break because of updates
... and the speed.

I also use it inside LX(C) containers and on the hypervisor itself and it makes a huge difference if your are e.g. on ZFS. On the hypervisor itself, Docker can use ZFS as its underlying data store, which creates each container layer as snapshotted dataset and adding stacks is very simple an just uses the COW architecture of ZFS, whereas without ZFS inside the LX(C) container, you currently cannot store directly inside of ZFS due to the fact that ZFS is currently not capable of delegating nested features to containers without breaking security. You have to create overlayfs layers, which are not so efficient and fast so that your Docker inside of LXC works a little bit slower. I'm looking forward to having this capability inside of LXC which will be a game changer for performance.

At home, I run Home Assistant inside of LCX Docker because I wanted a separated network stack for HA and just pulling the image is magnitudes slower than pulling the image on the hypervisor. In bigger environments I just run VMs for docker with ZFS inside.
 
  • Like
Reactions: joshfindit

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!