docker overlay2 error with LXC container

[rcd]

So I have this LXC container running docker with unprivileged,nesting=1 running fine, but just realize I created it on a raw image, which means snapshots take too long. I restored a snapshot to ZFS storage, which completed fine, but now docker no longer starts.

I get this error:

ERRO[2021-07-25T06:55:03.130281512Z] failed to mount overlay: invalid argument     storage-driver=overlay2
ERRO[2021-07-25T06:55:03.130305791Z] [graphdriver] prior storage driver overlay2 failed: driver not supported

I thought perhaps it had something to do with the CT being unpriv, but restoring it priv changed nothing.

Can this be fixed?

 

[oguz]

hi,

maybe you can try just moving the rootfs volume of the initial container to a different storage?

something like this:

pct move_volume <CTID> rootfs <OTHER_STORAGE>

you could move it to a ZFS pool for example.

which completed fine, but now docker no longer starts.

I get this error:

ERRO[2021-07-25T06:55:03.130281512Z] failed to mount overlay: invalid argument storage-driver=overlay2
ERRO[2021-07-25T06:55:03.130305791Z] [graphdriver] prior storage driver overlay2 failed: driver not supported

I thought perhaps it had something to do with the CT being unpriv, but restoring it priv changed nothing.

can you post your container configuration? cat /etc/pve/lxc/CTID.conf

 

[rcd]

Will moving the volume to another storage not result in the same as a snapshot restore to that same storage (zfs storage in both cases)?

arch: amd64
cores: 4
features: nesting=1
hostname: mail.xxx.net
memory: 6144
net0: name=eth0,bridge=vmbr0,firewall=1,gw=xx.yy.zz.108,hwaddr=02:00:00:xx:yy:zz,ip=xx.yy.zz.73/32,type=veth
onboot: 1
ostype: debian
rootfs: local:4008/vm-4008-disk-0.raw,size=50G
swap: 4096
unprivileged: 1

Edit: I tried anyway, and as expected, the result is the same.

 

[rcd]

So .... only way is to start over again?

Or just not possible to run Docker on ZFS storage?
Created a new CT with ZFS storage, installed docker, and it runs fine, so that's not the problem.

 

[oguz]

Created a new CT with ZFS storage, installed docker, and it runs fine, so that's not the problem.

then it must be something you did in that container.

normally unprivileged container with nesting should be okay for docker to work

 

[rcd]

It's running mailcow-dockerized. It's a huge docker stack.

 

[damarg]

I have the same problem. I installed Proxmox 7 with zfs on one of my servers. Before I had Proxmox 6.4 with lvm. This server has only one disk for the OS and containers. When I restore one of the containers running docker inside I get the same error. Before it ran on this server with lvm and I had no issues.
@rcd did you find a solution for this problem?

 

[damarg]

When I create a new lxc container and install docker it works. Nevertheless I see some error in the docker service.

Aug 26 18:05:18 test dockerd[16680]: time="2021-08-26T18:05:18.487786303Z" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay
Aug 26 18:05:18 test dockerd[16680]: time="2021-08-26T18:05:18.509975950Z" level=info msg="Loading containers: start."
Aug 26 18:05:18 test dockerd[16680]: time="2021-08-26T18:05:18.558933921Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16.>
Aug 26 18:05:18 test dockerd[16680]: time="2021-08-26T18:05:18.592357772Z" level=info msg="Loading containers: done."
Aug 26 18:05:18 test dockerd[16680]: time="2021-08-26T18:05:18.614170610Z" level=info msg="Docker daemon" commit=75249d8 graphdriver(s)=vfs version=20.10.8
Aug 26 18:05:18 test dockerd[16680]: time="2021-08-26T18:05:18.614495124Z" level=info msg="Daemon has completed initialization"
Aug 26 18:05:18 test systemd[1]: Started Docker Application Container Engine.
Aug 26 18:05:18 test dockerd[16680]: time="2021-08-26T18:05:18.637020205Z" level=info msg="API listen on /run/docker.sock"
Aug 26 18:05:35 test dockerd[16680]: time="2021-08-26T18:05:35.590274350Z" level=info msg="ignoring event" container=f258b63cd0205b86e2598694ceb0663401b5107c8dec>

When I remove docker on the restored image and install it again, I'm still not able to start the daemon. I tried to run dockerd with storage zfs but I get an error: wrong filesystem?

NFO[2021-08-27T04:05:26.844465173Z] Starting up                                 
DEBU[2021-08-27T04:05:26.845067974Z] Listener created for HTTP on unix (/var/run/docker.sock)
DEBU[2021-08-27T04:05:26.845485640Z] Golang's threads limit set to 112680         
INFO[2021-08-27T04:05:26.845739498Z] parsed scheme: "unix"                         module=grpc
INFO[2021-08-27T04:05:26.845750349Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-08-27T04:05:26.845774823Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-08-27T04:05:26.845786184Z] ClientConn switching balancer to "pick_first"  module=grpc
DEBU[2021-08-27T04:05:26.846160079Z] metrics API listening on /var/run/docker/metrics.sock
INFO[2021-08-27T04:05:26.846727828Z] parsed scheme: "unix"                         module=grpc
INFO[2021-08-27T04:05:26.846739684Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-08-27T04:05:26.846755889Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-08-27T04:05:26.846762837Z] ClientConn switching balancer to "pick_first"  module=grpc
DEBU[2021-08-27T04:05:26.847403141Z] Using default logging driver json-file       
DEBU[2021-08-27T04:05:26.847414570Z] [graphdriver] trying provided driver: zfs   
DEBU[2021-08-27T04:05:26.847449563Z] zfs command is not available: exec: "zfs": executable file not found in $PATH  storage-driver=zfs
DEBU[2021-08-27T04:05:26.847769384Z] Cleaning up old mountid : start.             
failed to start daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?)

 

[rcd]

@damarg no, I'm still stuck with the raw filesystem. I know if I start over with a ZFS filesystem it will work, but I can't convert this.

 

[damarg]

I copied the docker volumes over to a new lxc container with docker on zfs. I didn't have to move many so it was done quickly.

 

[joshfindit]

I had a possibly related issue that I solved and documented here: https://forum.proxmox.com/threads/proxmox-7-docker-stoped-working.101926/post-441959

Short version is that I created a named pipe inside an LXC and that broke docker for all other containers. After I deleted it everything was fine again.
If this is related, it's likely that copying the files to another device/filesystem would drop anything that's not a 'valid' file (such as a pipe).