Docker issue after moving disk of container to second ZFS storage

Z

Zoker

Active Member
Oct 31, 2020
36
10
28
Hi there,

I added a second SSD to my installation and now I want to move container disk to this second storage.
In proxmox I added the second storage as a ZFS and set the content to be "Disk images" and "Container".
Now my issue is, that when I move the disk to this second storage device, my docker within my alpine container does not work anymore.
It gives me all kind of weird errors and when I want to run `docker ps` I get the following issue:
Code: 
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

When I move the disk back to the main storage, docker works again without any issue.
When I move a VM disk to the second storage, everything works without issue.

Maybe someone has experienced similar issues and can give me a hint here?
Thanks!
 
In this case it would be smart to check the logs of the docker daemon to see what went wrong.
 
Thanks for the hin!
Here are the logs:

Code: 
time="2022-12-12T13:20:45.221818056Z" level=info msg="Starting up"
time="2022-12-12T13:20:45.222560643Z" level=info msg="libcontainerd: started new containerd process" pid=702
time="2022-12-12T13:20:45.222586457Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-12-12T13:20:45.222597322Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-12-12T13:20:45.222613909Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2022-12-12T13:20:45.222624963Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-12-12T13:20:45Z" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
time="2022-12-12T13:20:45.238145639Z" level=info msg="starting containerd" revision=770bd0108c32f3fb5c73ae1264f7e503fe7b2661 version=v1.6.10
time="2022-12-12T13:20:45.248047073Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2022-12-12T13:20:45.248092098Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2022-12-12T13:20:45.248559682Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2022-12-12T13:20:45.248584965Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2022-12-12T13:20:45.248699190Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (zfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-12-12T13:20:45.248717333Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2022-12-12T13:20:45.248730821Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2022-12-12T13:20:45.248742223Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2022-12-12T13:20:45.248766079Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2022-12-12T13:20:45.248922974Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2022-12-12T13:20:45.249055587Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="exec: \"zfs\": executable file not found in $PATH: \"zfs fs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset Storage/subvol-151-disk-0\" => : skip plugin" type=io.containerd.snapshotter.v1
time="2022-12-12T13:20:45.249075362Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2022-12-12T13:20:45.249095250Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2022-12-12T13:20:45.249108184Z" level=info msg="metadata content store policy set" policy=shared
time="2022-12-12T13:20:45.249175524Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2022-12-12T13:20:45.249193762Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2022-12-12T13:20:45.249211724Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2022-12-12T13:20:45.249234050Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2022-12-12T13:20:45.249249040Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2022-12-12T13:20:45.249262913Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2022-12-12T13:20:45.249275658Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2022-12-12T13:20:45.249289773Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2022-12-12T13:20:45.249303854Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2022-12-12T13:20:45.249317439Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2022-12-12T13:20:45.249330804Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2022-12-12T13:20:45.249344484Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2022-12-12T13:20:45.249401213Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2022-12-12T13:20:45.249443402Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2022-12-12T13:20:45.249811892Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2022-12-12T13:20:45.249858773Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.249880685Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2022-12-12T13:20:45.249928700Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.249948924Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.249966981Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.249986301Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.250003755Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.250021063Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.250037978Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.250054071Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.250083773Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2022-12-12T13:20:45.250128956Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.250148404Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.250164976Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2022-12-12T13:20:45.250184272Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2022-12-12T13:20:45.250203615Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
time="2022-12-12T13:20:45.250219847Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2022-12-12T13:20:45.250243612Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
time="2022-12-12T13:20:45.250507585Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2022-12-12T13:20:45.250579108Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2022-12-12T13:20:45.250632822Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2022-12-12T13:20:45.250654901Z" level=info msg="containerd successfully booted in 0.012951s"
time="2022-12-12T13:20:45.255078152Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-12-12T13:20:45.255098986Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-12-12T13:20:45.255114800Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2022-12-12T13:20:45.255129107Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-12-12T13:20:45.255599132Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-12-12T13:20:45.255618656Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-12-12T13:20:45.255637068Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2022-12-12T13:20:45.255648937Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-12-12T13:20:45.256849564Z" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
time="2022-12-12T13:20:45.256866578Z" level=error msg="[graphdriver] prior storage driver overlay2 failed: driver not supported"
time="2022-12-12T13:20:45.257114315Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2022-12-12T13:20:45.257114505Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
failed to start daemon: error initializing graphdriver: driver not supported

I tried to fix this, by creating a `/etc/docker/daemon.json` file as suggested here and put the following content in the file:
Code: 
{
  "storage-driver": "zfs"
}

But unfortunately that did not solve the issue:
Code: 
time="2022-12-12T13:49:39.036028455Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\". Reconnecting..." module=grpc
failed to start daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?)
 
That's a weird error, not sure if using zfs as storage-driver would work.

What is your container config as well as storage.cfg on the host? What does df -h say on the guest?

According to this GitHub issue, resetting docker might work although that options seems a bit drastic:
https://github.com/docker/for-linux/issues/1400
 
This is the container config:
Code: 
arch: amd64
cores: 2
features: keyctl=1,nesting=1
hostname: DroneRunner
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=26:99:40:66:B8:15,ip=dhcp,type=veth
onboot: 1
ostype: alpine
protection: 1
rootfs: Storage:subvol-151-disk-0,size=16G
swap: 512
unprivileged: 1

Storage config:
Code: 
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

nfs: NAS
        export /volume1/proxmox
        path /mnt/pve/NAS
        server 192.168.180.150
        content backup
        prune-backups keep-last=2

zfspool: Storage
        pool Storage
        content images,rootdir
        mountpoint /Storage
        nodes saturn,jupiter
Some context here: I have two nodes, each have a NVME as main storage and a sata SSD as second storage and is called "Storage" on both, but is not shared between the nodes.

And the df command:
Code: 
Filesystem                        Size  Used Avail Use% Mounted on
udev                               16G     0   16G   0% /dev
tmpfs                             3.2G  1.7M  3.2G   1% /run
/dev/mapper/pve-root               94G   32G   58G  36% /
tmpfs                              16G   76M   16G   1% /dev/shm
tmpfs                             5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p2                    511M   27M  485M   6% /boot/efi
Storage                            87G  128K   87G   1% /Storage
/dev/fuse                         128M   48K  128M   1% /etc/pve
192.168.180.150:/volume1/proxmox   13T  7.0T  5.3T  57% /mnt/pve/NAS
Storage/subvol-151-disk-0          16G  439M   16G   3% /Storage/subvol-151-disk-0
tmpfs                             3.2G     0  3.2G   0% /run/user/0

I tried the docker reset commands, but that did not work. Same issue as before
 
You must log in or register to reply here.
Top Bottom