[SOLVED] Unable to Restore Specific Container Backup on New Proxmox Cluster - "No Space Left on Device" Error

[w453y]

Hello,

I am facing an issue while migrating a container from my old Proxmox node to a new cluster. To facilitate the migration, I set up a Proxmox Backup Server (PBS), successfully backed up the containers and VMs from the old node, and restored most of them on the new cluster without any issues. However, when I attempt to restore one specific container backup, I encounter the following error:

recovering backed-up configuration from 'pbs2:backup/ct/116/2024-12-18T11:41:11Z'
/dev/rbd5
/dev/rbd6
rbd: warning: image already mapped as /dev/rbd5
Creating filesystem with 83886080 4k blocks and 20971520 inodes
Filesystem UUID: ca76a0a6-a19d-4228-bc42-2bddbb8fe5e9
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968
restoring 'pbs2:backup/ct/116/2024-12-18T11:41:11Z' now..
Error: error extracting archive - encountered unexpected error during extraction: error at entry "activity_logs.ibd": failed to extract file: failed to copy file contents: No space left on device (os error 28)
Removing image: 1% complete...
Removing image: 2% complete...
Removing image: 3% complete...
Removing image: 4% complete...
Removing image: 5% complete...
Removing image: 6% complete...
Removing image: 7% complete...
Removing image: 8% complete...
Removing image: 9% complete...
Removing image: 10% complete...
Removing image: 11% complete...
Removing image: 12% complete...
Removing image: 13% complete...
Removing image: 14% complete...
Removing image: 15% complete...
Removing image: 16% complete...
Removing image: 17% complete...
Removing image: 18% complete...
Removing image: 19% complete...
Removing image: 20% complete...
Removing image: 21% complete...
Removing image: 22% complete...
Removing image: 23% complete...
Removing image: 24% complete...
Removing image: 25% complete...
Removing image: 26% complete...
Removing image: 27% complete...
Removing image: 28% complete...
Removing image: 29% complete...
Removing image: 30% complete...
Removing image: 31% complete...
Removing image: 32% complete...
Removing image: 33% complete...
Removing image: 34% complete...
Removing image: 35% complete...
Removing image: 36% complete...
Removing image: 37% complete...
Removing image: 38% complete...
Removing image: 39% complete...
Removing image: 40% complete...
Removing image: 41% complete...
Removing image: 42% complete...
Removing image: 43% complete...
Removing image: 44% complete...
Removing image: 45% complete...
Removing image: 46% complete...
Removing image: 47% complete...
Removing image: 48% complete...
Removing image: 49% complete...
Removing image: 50% complete...
Removing image: 51% complete...
Removing image: 52% complete...
Removing image: 53% complete...
Removing image: 54% complete...
Removing image: 55% complete...
Removing image: 56% complete...
Removing image: 57% complete...
Removing image: 58% complete...
Removing image: 59% complete...
Removing image: 60% complete...
Removing image: 61% complete...
Removing image: 62% complete...
Removing image: 63% complete...
Removing image: 64% complete...
Removing image: 65% complete...
Removing image: 66% complete...
Removing image: 67% complete...
Removing image: 68% complete...
Removing image: 69% complete...
Removing image: 70% complete...
Removing image: 71% complete...
Removing image: 72% complete...
Removing image: 73% complete...
Removing image: 74% complete...
Removing image: 75% complete...
Removing image: 76% complete...
Removing image: 77% complete...
Removing image: 78% complete...
Removing image: 79% complete...
Removing image: 80% complete...
Removing image: 81% complete...
Removing image: 82% complete...
Removing image: 83% complete...
Removing image: 84% complete...
Removing image: 85% complete...
Removing image: 86% complete...
Removing image: 87% complete...
Removing image: 88% complete...
Removing image: 89% complete...
Removing image: 90% complete...
Removing image: 91% complete...
Removing image: 92% complete...
Removing image: 93% complete...
Removing image: 94% complete...
Removing image: 95% complete...
Removing image: 96% complete...
Removing image: 97% complete...
Removing image: 98% complete...
Removing image: 99% complete...
Removing image: 100% complete...done.
TASK ERROR: unable to restore CT 124 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client restore '--crypt-mode=none' ct/116/2024-12-18T11:41:11Z root.pxar /var/lib/lxc/124/rootfs --allow-existing-dirs --repository irisbackups@pbs@10.15.64.250:irisvmbackups2' failed: exit code 255

Initially, I thought the backup might be corrupted. I deleted the backup, created a fresh one, and tried restoring it again. The error persisted, also verified that there is ample storage available on my new cluster, which is configured with a Ceph storage backend. However I have successfully created and restored other VM and container backups, so this indicats the issue is specific to this particular container.

What could be causing this error for this specific container backup? Could it be related to the Ceph configuration, a possible issue with the container's filesystem, or something else? Any guidance or suggestions for troubleshooting would be greatly appreciated.

Thank you in advance for your help!

 

[tcabernoch]

I'll readily admit that this is a guess. Maybe that stuff won't fit into the space allocated?
I'd try two things.

Clone it. Can you backup and migrate the clone? You're done.

If that didn't work, add some space to the disk of the clone. Can you backup and migrate? Done.

... And if those things don't work ... um ... idk.

Ooo, I got another one.
Don't use PBS.
Try backing it up to NFS or other storage.
You could even backup to local storage and then scp the vzdump file to the other server if you don't have a NAS.

 

[w453y]

I'll readily admit that this is a guess. Maybe that stuff won't fit into the space allocated?
I'd try two things.

Clone it. Can you backup and migrate the clone? You're done.

If that didn't work, add some space to the disk of the clone. Can you backup and migrate? Done.

... And if those things don't work ... um ... idk.

Ooo, I got another one.
Don't use PBS.
Try backing it up to NFS or other storage.
You could even backup to local storage and then scp the vzdump file to the other server if you don't have a NAS.

I added more disk space to the clone's disk, and it worked. Thanks!!!!

 

[tcabernoch]

:]