[SOLVED] [HELP] Unable to restore LXC via PBS

[lthornl]

Hi ProxMox Forums,

I am trying to restore an LXC container to a new host and encountering an error. Similarly sized VMs have restored fine but this LXC backup keeps encountering the following error (Names adjusted):

Virtual Environment 8.1.4
Storage 'RESTORE_ONLY-PBSServer' on node 'HOST2'
recovering backed-up configuration from 'RESTORE_ONLY-PBSServer:backup/ct/110/2024-03-27T16:00:35Z'
  Logical volume "vm-111-disk-0" created.
Creating filesystem with 41943040 4k blocks and 10485760 inodes
Filesystem UUID: 5b09824b-5d64-4aca-b2cb-7e4a753d5535
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872
restoring 'RESTORE_ONLY-PBSServer:backup/ct/110/2024-03-27T16:00:35Z' now..
Error: error extracting archive - encountered unexpected error during extraction: error at entry "libzstd.so.1.5.4": failed to extract file: failed to copy file contents: No space left on device (os error 28)
  Logical volume "vm-111-disk-0" successfully removed.
TASK ERROR: unable to restore CT 111 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client restore '--crypt-mode=none' ct/110/2024-03-27T16:00:35Z root.pxar /var/lib/lxc/111/rootfs --allow-existing-dirs --repository root@pam@10.10.0.224:backups' failed: exit code 255

The final line seems to be somewhat of a nothing error, but the error before it mentions "No space left on device (os error 28)" which I suspect is the root cause
I am restoring from a 500GB Restore Volume (RESTORE_ONLY-PBSServer) to a LVM-Thin 8TB volume
The container is 161.67GB in size

The PBS itself is just installed on a 40GB drive, could this be the source of the issue?
I am fairly new to PBS but I would think if it needed to extract the compressed back up it would do so on either the destination or restore volume not the OS Disk?

I'm running the latest version of each, I have tried with a smaller container and it worked without issue, just seems odd to me that the small PBS OS Disk would be the cause.
I will be resizing the OS Disk here shortly and will return with the results.

In the interim, any assistance or advice would be most appreciated!

Thank you,
lThornl

 

[lthornl]

UPDATE: Expanded the PBS OS partition to 200GB and encountered the exact same error

 

[lthornl]

UPDATE: I was able to restore to an entirely different host, not sure what was wrong with the one I was using.

 

[fitful]

I encountered a similar issue while restoring an LXC container and wanted to share what I found to be the root cause and solution.

The error "No space left on device (os error 28)" during the restore process is typically caused by insufficient disk space allocated to the container. When restoring an LXC container, Proxmox assigns the disk size based on the original configuration. If the allocated disk size for the container is smaller than the size of the backup on the Proxmox Backup Server (PBS), the restore process will fail due to a lack of space.

In my case, the allocated disk size for the container was 12GB, but the backup size on the PBS was slightly larger, around 12.5GB. During restoration, a loop device (e.g., loop1) is mounted with the allocated disk size of 12GB. Since the backup size exceeded the available disk space, the process couldn't complete successfully.

To resolve this, I extended the disk size at the container (CT) level, created a new backup, and was then able to restore the container from PBS on another server without any issues.

The takeaway is to ensure that the allocated disk space is large enough to accommodate the size of the backup during restoration, which will prevent running into this error.

I hope this helps anyone facing a similar problem!

root@pve-hpe:~# cat /etc/pve/lxc/134.conf  | grep rootf
rootfs: local-zfs:subvol-134-disk-0,size=12G

root@pve-hpe:~# df -h | grep 134
rpool/data/subvol-134-disk-0   12G  6.8G  8.3G  46% /rpool/data/subvol-134-disk-0

root@pve-hpe:~# pvesm list pbs | grep ct/134 | head -1
pbs:backup/ct/134/2024-08-24T23:21:26Z pbs-ct  backup     13440908440 134

root@pve-hpe:~# echo "scale=2; 13440908440 / 1073741824" | bc
12.51

 

[zakmaf]

root@pve-hpe:~# cat /etc/pve/lxc/134.conf  | grep rootf
rootfs: local-zfs:subvol-134-disk-0,size=12G

root@pve-hpe:~# df -h | grep 134
rpool/data/subvol-134-disk-0   12G  6.8G  8.3G  46% /rpool/data/subvol-134-disk-0

root@pve-hpe:~# pvesm list pbs | grep ct/134 | head -1
pbs:backup/ct/134/2024-08-24T23:21:26Z pbs-ct  backup     13440908440 134

root@pve-hpe:~# echo "scale=2; 13440908440 / 1073741824" | bc
12.51

Thanks a lot.

Had a similar isssue so I ran the same code as you to get the "real" space used :

oot@optiblack:~# cat /etc/pve/lxc/102.conf | grep rootf
rootfs: nvzfs:subvol-102-disk-0,size=4G
root@optiblack:~# df -h | grep 102
/dev/sdb2               1022M   11M 1012M   2% /boot/efi
nvzfs/subvol-102-disk-0  4.0G  1.6G  2.5G  39% /nvzfs/subvol-102-disk-0
root@optiblack:~# pvesm list pbs | grep ct/102 | head -1
pbs:backup/ct/102/2024-12-16T04:46:17Z pbs-ct  backup     5589514908 102
root@optiblack:~# echo "scale=2; 5589514908 / 1073741824" | bc
5.20

 

[hebocon]

The error "No space left on device (os error 28)" during the restore process is typically caused by insufficient disk space allocated to the container. When restoring an LXC container, Proxmox assigns the disk size based on the original configuration. If the allocated disk size for the container is smaller than the size of the backup on the Proxmox Backup Server (PBS), the restore process will fail due to a lack of space.

In my case, the allocated disk size for the container was 12GB, but the backup size on the PBS was slightly larger, around 12.5GB. During restoration, a loop device (e.g., loop1) is mounted with the allocated disk size of 12GB. Since the backup size exceeded the available disk space, the process couldn't complete successfully.

To resolve this, I extended the disk size at the container (CT) level, created a new backup, and was then able to restore the container from PBS on another server without any issues.

The takeaway is to ensure that the allocated disk space is large enough to accommodate the size of the backup during restoration, which will prevent running into this error.

Thank you! I was relieved to see this thread show up when I searched for the full error. In my case it was an LXC (Audiobookshelf) with a 16GB disk using 12 in the container... but the backup was 19GB! I wonder if symlinks are being "flattened" in some way which might cause the ballooning of data rather than its compression.