Debian LXC Container with Docker restore failed "Cannot write: No space left on device"

[r00tusr]

Hello there,

I just wanted to restore a unprivileged Debian LXC Container and get the following error in the logs:

recovering backed-up configuration from 'PVE-Backup-NAS:backup/vzdump-lxc-104-2021_05_23-14_25_36.tar.zst'
  Logical volume "vm-104-disk-0" created.
Creating filesystem with 15728640 4k blocks and 3932160 inodes
Filesystem UUID: 44a30080-f22f-4ac3-a06d-8b94687efd63
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424
  Logical volume "vm-104-disk-1" created.
Creating filesystem with 13107200 4k blocks and 3276800 inodes
Filesystem UUID: b0588b3f-511b-4870-aadb-32e5b02cb8da
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424
restoring 'PVE-Backup-NAS:backup/vzdump-lxc-104-2021_05_23-14_25_36.tar.zst' now..
extracting archive '/mnt/pve/PVE-Backup-NAS/dump/vzdump-lxc-104-2021_05_23-14_25_36.tar.zst'
tar: ./var/lib/docker/vfs/dir/55ab3cf4b1d68663033e892a856433e12bcf7b3703ed88ff80cc5dc0df858005/usr/local/bin/phpdbg: Cannot write: No space left on device
tar: ./var/lib/docker/vfs/dir/55ab3cf4b1d68663033e892a856433e12bcf7b3703ed88ff80cc5dc0df858005/usr/local/bin/docker-php-ext-enable: Cannot write: No space left on device
tar: ./var/lib/docker/vfs/dir/55ab3cf4b1d68663033e892a856433e12bcf7b3703ed88ff80cc5dc0df858005/usr/local/bin/php: Cannot write: No space left on device
tar: ./var/lib/docker/vfs/dir/55ab3cf4b1d68663033e892a856433e12bcf7b3703ed88ff80cc5dc0df858005/usr/local/bin/pecl: Cannot write: No space left on device
tar: ./var/lib/docker/vfs/dir/55ab3cf4b1d68663033e892a856433e12bcf7b3703ed88ff80cc5dc0df858005/usr/local/bin/phar.phar: Cannot write: No space left on device
tar: ./var/lib/docker/vfs/dir/55ab3cf4b1d68663033e892a856433e12bcf7b3703ed88ff80cc5dc0df858005/usr/local/bin/peardev: Cannot write: No space left on device

There are tons of these "Cannot write: No space left on device" error lines. This is a Server that I just installed today. I wanted to restore all my LXC Containers from my old PVE Server. All my other Containers were restored successfully - including another one with Docker. Just this one is not working. I have enough space left on my local-lvm, where this should be restored to. How can I successfully restore this Container?

Best Regards

 

[r00tusr]

I tried to restore it as unprivileged container:

TASK ERROR: unable to restore CT 104 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - --zstd --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/104/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2

And as privileged container:

TASK ERROR: unable to restore CT 104 - command 'tar xpf - --zstd --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/104/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2

Both methods are not working as you can see.

 

[r00tusr]

Enough disk space is available as you can see. 00:10 - 00:25 is where I tried on of the restores of my Docker Server.

 

[dcsapak]

did you make the backups on zfs storage originally? if yes, it may be that the files were compressed and could thus be stored in a smaller image than normally. try restoring onto a zfs again in this case (with compression on)

 

[r00tusr]

yes, I initially had a proxmox server running on zfs and I wanted to restore the backup on a new server with ext4, because I replaced the zfs raid 1 with a hardware raid controller. I have already set up a test-server with zfs and tried restoring the backup - and voila it worked. After this I just synced my files over to my new productivity server. But shouldn't we be able to restore a backup thats coming from a host with a zfs file system on a new host with a ext4 file system? I have a second lxc container that runs docker which restored without any problems - that's what confused me a lot. Why can one docker server be restored successfully and one fails completely? As you can see in the logs above, only files from the vfs from docker in the path /var/lib/docker/vfs seem to cause this problem, so not only the initial zfs file system caused the problem - I would say, it's zfs in combination with a strange docker constellation...

Best regards