Restoring Backup fails - "no space left on device"

H

HomerJ.S.

New Member
Aug 7, 2024
13
1
3
Hi community,

I need a little help, again. A few month ago I had a problem with the backup of my containers. The community helped me in this thread:
H

Post in thread 'Backup Job Error - No space left on device'

Hi,
finally I could fix the problem. As Fiona mentioned the backup job generates a temporary "file" and on my system the local ssd is to small. So it run full and the backup job failed. At first I moved the temp directory to one of my data hdd and so there was enough space for the temp file. As in the post
https://forum.proxmox.com/threads/lxc-backup-fails-with-rsync-exit-code-11.120667/
suggested, I edited the

Code: 
/etc/vzdump.conf

file and changed the line:

Code: 
tmpdir:

entry to a local path on the hdd.

And finally I installed an old ssd additionally to the system, just...

The solution was to define a tmpdir in the vzdump.conf.

Now I tried to restore one of my containers and I run into the same problem. Obviously the backup is restored in a backup file, and there is not enough space.

Code: 
Creating filesystem with 13107200 4k blocks and 3276800 inodes
Filesystem UUID: 7d37948f-8a75-45a5-a70a-491d92bc1b03
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424
restoring 'Daten2:backup/vzdump-lxc-101-2025_08_01-03_00_04.tar.zst' now..
extracting archive '/mnt/pve/Daten2/dump/vzdump-lxc-101-2025_08_01-03_00_04.tar.zst'
tar: ./srv/openhab-userdata/backups/openhab-backup-24_10_02-22_25_49.zip: Wrote only 2560 of 10240 bytes
tar: ./srv/openhab-userdata/tmpalt: Cannot mkdir: No space left on device
tar: ./srv/openhab-userdata/tmpalt: Cannot mkdir: No space left on device
tar: ./srv/openhab-userdata/tmpalt/.paxweb11889480466940345254: Cannot mkdir: No such file or directory
tar: ./srv/openhab-userdata/tmpalt: Cannot mkdir: No space left on device
tar: ./srv/openhab-userdata/tmpalt/fileinstall-318041911140237937: Cannot mkdir: No such file or directory
tar: ./srv/openhab-userdata/tmpalt: Cannot mkdir: No space left on device
tar: ./srv/openhab-userdata/tmpalt/fileinstall-2754830848287325891: Cannot mkdir: No such file or directory
tar: ./srv/openhab-userdata/tmpalt: Cannot mkdir: No space left on device

My short question, is there a file like the vzdump.conf for the restoring settings? I didn't find the option.

Thanks
 
HomerJ.S. said:
My short question, is there a file like the vzdump.conf for the restoring settings?
Click to expand...
That file was IMHO for creating a temporary file from which the actual backup was written out to.

In the case of a restore, the file is directly unpacked to the destination, so it seems you don't have enough space available there.
Can you check if that is the case?
 
this is likely caused by your previous container storage being ZFS with compression, and your new one being something else. you either need to increase the rootfs/mp size and redo the backup, or do a custom restore using `pct restore ..` where you can override the size to accommodate for this difference.
 
Thanks for the fast replies.

My storage:

Code: 
root@proxmox:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   16G     0   16G   0% /dev
tmpfs                 3.1G  6.1M  3.1G   1% /run
/dev/mapper/pve-root   94G   31G   59G  35% /
tmpfs                  16G   46M   16G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
efivarfs              128K   16K  108K  13% /sys/firmware/efi/efivars
/dev/nvme1n1p2       1022M   12M 1011M   2% /boot/efi
/dev/nvme0n1p1        234G   49G  174G  22% /mnt/pve/BackupCache
/dev/sdb1             7.3T  4.7T  2.2T  69% /mnt/pve/Daten2
/dev/sda1              11T  1.4T  8.9T  14% /mnt/pve/Daten3-12TB
/dev/fuse             128M   28K  128M   1% /etc/pve
tmpfs                 3.1G     0  3.1G   0% /run/user/0

@LnxBil: The destination storage is my local-lvm, there is more than 300 GB left.

@fabian: no I don't not use ZFS, all my storage is ext4.

I think the problem is, that the backup mode was suspend, so the backup is restored in a temp file. The temp file is (so far I understand) on the pve-root which only has 30 GB free space left. That was the problem with the backup from my initial post and I could fix it with redirect the file into a tempdir on an other drive.
 
backup mode and restore have no relation at all. maybe the size part of the config stored in the backup is wrong? do you still have access to the original system?
 
then please post the container config ("pct config XXX") and the space information from inside the container ("pct exec XXX -- df -h")
 
NOTE: I made a additional backup, in Stop-Mode. And this backup i can restore without problem.

Here the info you asked for from the original container:

Code: 
root@proxmox:~# pct config 101
arch: amd64
cores: 2
features: nesting=1
hostname: openHABian
memory: 3072
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=BC:24:11:AA:47:64,ip=192.168.1.101/24,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-101-disk-0,size=50G
swap: 512
unprivileged: 1

Code: 
root@proxmox:~# pct exec 101 -- df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--101--disk--0   49G   33G   14G  71% /
none                              492K  4.0K  488K   1% /dev
udev                               16G     0   16G   0% /dev/tty
tmpfs                              16G     0   16G   0% /dev/shm
tmpfs                             6.2G  1.6M  6.2G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
 
so that looks okay, what about (warning, will take long and cause some I/O load) "pct exec 100 -- du -sm /"
 
Did you mean ...101...

Anyway here the prints:

Code: 
root@proxmox:~# pct exec 100 -- du -sm /
Configuration file 'nodes/proxmox/lxc/100.conf' does not exist
root@proxmox:~# pct exec 101 -- du -sm /
du: cannot read directory '/lost+found': Permission denied
du: cannot read directory '/dev/.lxc/sys/kernel': Permission denied
du: cannot read directory '/dev/.lxc/sys/power': Permission denied
du: cannot read directory '/dev/.lxc/sys/class': Permission denied
du: cannot read directory '/dev/.lxc/sys/devices': Permission denied
du: cannot read directory '/dev/.lxc/sys/dev': Permission denied
du: cannot read directory '/dev/.lxc/sys/hypervisor': Permission denied
du: cannot read directory '/dev/.lxc/sys/fs': Permission denied
du: cannot read directory '/dev/.lxc/sys/bus': Permission denied
du: cannot read directory '/dev/.lxc/sys/firmware': Permission denied
du: cannot read directory '/dev/.lxc/sys/block': Permission denied
du: cannot read directory '/dev/.lxc/sys/module': Permission denied
du: cannot read directory '/dev/.lxc/proc/fs': Permission denied
du: cannot read directory '/dev/.lxc/proc/bus': Permission denied
du: cannot read directory '/dev/.lxc/proc/irq': Permission denied
du: cannot read directory '/dev/.lxc/proc/spl': Permission denied
du: cannot read directory '/dev/.lxc/proc/sys': Permission denied
du: cannot read directory '/dev/.lxc/proc/tty': Permission denied
du: cannot read directory '/dev/.lxc/proc/acpi': Permission denied
du: cannot read directory '/dev/.lxc/proc/scsi': Permission denied
du: cannot read directory '/dev/.lxc/proc/asound': Permission denied
du: cannot read directory '/dev/.lxc/proc/driver': Permission denied
du: cannot read directory '/dev/.lxc/proc/sysvipc': Permission denied
du: cannot read directory '/dev/.lxc/proc/pressure': Permission denied
du: cannot read directory '/dev/.lxc/proc/dynamic_debug': Permission denied
du: cannot read directory '/dev/.lxc/proc/1': Permission denied
du: cannot read directory '/dev/.lxc/proc/47': Permission denied
du: cannot read directory '/dev/.lxc/proc/63': Permission denied
du: cannot read directory '/dev/.lxc/proc/104': Permission denied
du: cannot read directory '/dev/.lxc/proc/106': Permission denied
du: cannot read directory '/dev/.lxc/proc/107': Permission denied
du: cannot read directory '/dev/.lxc/proc/108': Permission denied
du: cannot read directory '/dev/.lxc/proc/110': Permission denied
du: cannot read directory '/dev/.lxc/proc/111': Permission denied
du: cannot read directory '/dev/.lxc/proc/112': Permission denied
du: cannot read directory '/dev/.lxc/proc/124': Permission denied
du: cannot read directory '/dev/.lxc/proc/127': Permission denied
du: cannot read directory '/dev/.lxc/proc/128': Permission denied
du: cannot read directory '/dev/.lxc/proc/129': Permission denied
du: cannot read directory '/dev/.lxc/proc/131': Permission denied
du: cannot read directory '/dev/.lxc/proc/134': Permission denied
du: cannot read directory '/dev/.lxc/proc/138': Permission denied
du: cannot read directory '/dev/.lxc/proc/139': Permission denied
du: cannot read directory '/dev/.lxc/proc/266': Permission denied
du: cannot read directory '/dev/.lxc/proc/316': Permission denied
du: cannot read directory '/dev/.lxc/proc/323': Permission denied
du: cannot read directory '/dev/.lxc/proc/341': Permission denied
du: cannot read directory '/dev/.lxc/proc/462': Permission denied
du: cannot read directory '/dev/.lxc/proc/463': Permission denied
du: cannot read directory '/dev/.lxc/proc/464': Permission denied
du: cannot read directory '/dev/.lxc/proc/468': Permission denied
du: cannot read directory '/dev/.lxc/proc/7342': Permission denied
du: cannot read directory '/sys/kernel/debug': Permission denied
du: cannot read directory '/sys/fs/fuse/connections/46': Permission denied
du: cannot read directory '/sys/fs/fuse/connections/43': Permission denied
du: cannot read directory '/proc/tty/driver': Permission denied
du: cannot access '/proc/138/task/1428/fdinfo/207': No such file or directory
du: cannot access '/proc/138/task/7397': No such file or directory
du: cannot access '/proc/7342/task/7342/fd/3': No such file or directory
du: cannot access '/proc/7342/task/7342/fdinfo/3': No such file or directory
du: cannot access '/proc/7342/fd/4': No such file or directory
du: cannot access '/proc/7342/fdinfo/4': No such file or directory
57226    /
 
yes, 101, sorry. so it seems you have more "logical" data stored on your disk than it's physical size.. that likely means some sparse files somewhere that are not re-created as sparse when restoring. the same advice still applies - you can make the original container's rootfs bigger and do another backup, or restore it using a custom --rootfs parameter to give it enough space..
 
Yes your right, my system is a bit of a playground and I run a lot of containers and vms which do reserve more storage than physical is available. But the real use of storage is approx. 60 % of the physical storage.
The second option sounds like a plan, even if I don't know how to do this.
Maybe I switch to snapshots for the containers which should not have any downtime and stop-mode for the other ones. Both mode work, only the suspend mode does not work.

Thanks, I'll report what worked and what's my solution will be.
 
you can also find out the cause, maybe it's possible to remove some of the offending files or make them logically smaller ;)
 
So finally I managed the problem by increasing the rootfs of the container. The minimum working size was 60 GB, 32 GB are used within the container.

It works, but I don't understand it. Why does the system need free space within the container for restoring it?

Anyway, thanks for your help fabian, really incredible how fast you answered.
 
You must log in or register to reply here.
Top Bottom