Backup cannot be restored, allegedly not enough memory

M

moxer_0815

New Member
Oct 23, 2024
9
0
1
(i have moved the german topic here because of the greater reach, origin: https://forum.proxmox.com/threads/b...len-angeblich-nicht-genügend-speicher.156384/ )

Hello everyone,

I would like to restore a backup.
Setup:
Dell Optiplex Micro, Proxmox Linux 6.8.12-2-pve (2024-09-05T10:03Z).

The following drives have been created:
sda: ISO image & container templates, 17 GB/100 GB used.
sdb: Disk image, Container, Snippets, 340 GB/983 GB used.
NAS1 via smb/cifs: VZDump backupfile
NAS2 via smb/cifs (on and connected only once a month): VZDump backupfile

Size of the backup (excerpt from the log):


Code:

2024-10-22 03:04:26 INFO: Total bytes written: 69654425600 (65GiB, 262MiB/s)
2024-10-22 03:04:30 INFO: archive file size: 6.01GB


When restoring, it aborts at some point with the error that there is no more space for unpacking.

If I monitor the memory in parallel in the shell, I see the following:

Before backup restore:


Code:

Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 781M 2.9M 779M 1% /run
/dev/mapper/pve-root 94G 16G 74G 18% /
tmpfs 3.9G 46M 3.8G 2% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
efivarfs 256K 182K 70K 73% /sys/firmware/efi/efivars
/dev/sdb 916G 318G 553G 37% /mnt/sdb
/dev/sda2 1022M 12M 1011M 2% /boot/efi
//192.168.1.99/proxmox_backup 28T 17T 11T 61% /mnt/pve/NAS
tmpfs 781M 0 781M 0% /run/user/0
/dev/fuse 128M 24K 128M 1% /etc/pve



Abort:


Code:

Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 781M 2.8M 779M 1% /run
/dev/mapper/pve-root 94G 16G 74G 18% /
tmpfs 3.9G 52M 3.8G 2% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
efivarfs 256K 182K 70K 73% /sys/firmware/efi/efivars
/dev/sdb 916G 383G 487G 45% /mnt/sdb
/dev/sda2 1022M 12M 1011M 2% /boot/efi
//192.168.1.99/proxmox_backup 28T 17T 11T 61% /mnt/pve/NAS
tmpfs 781M 0 781M 0% /run/user/0
/dev/fuse 128M 24K 128M 1% /etc/pve
/dev/loop4 69G 65G 0 100% /var/lib/lxc/102/rootfs


I can recognize the following:
1. Unpacking takes place in /dev/loop4.
2. The named directory(?, or is that a "virtual drive"?) is full, which is why the error occurs.
3. The data is actually written to sdb, where there is still enough space even if it is canceled.

I don't know why I can't manage with the 69 GB there. Opened in Winrar, the program confirms the size of the backup: unpacked size 69,229,176,417 bytes

Memory config:

View attachment 76713

How do I fix my problem now? I would be very happy to receive help. If information is missing just let me know :)
 
Code: 
recovering backed-up configuration from 'NAS:backup/vzdump-lxc-102-2024_10_22-03_00_09.tar.zst'
Formatting '/mnt/sdb//images/102/vm-102-disk-0.raw', fmt=raw size=75161927680 preallocation=off
Creating filesystem with 18350080 4k blocks and 4587520 inodes
Filesystem UUID: fa163fae-9184-4b3a-a1ee-ae524efbf3c8
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424
restoring 'NAS:backup/vzdump-lxc-102-2024_10_22-03_00_09.tar.zst' now..
extracting archive '/mnt/pve/NAS/dump/vzdump-lxc-102-2024_10_22-03_00_09.tar.zst'
tar: ./var/cache/apt/archives/docker-buildx-plugin_0.13.0-1~debian.12~bookworm_amd64.deb: Wrote only 5120 of 10240 bytes
tar: ./var/cache/apt/archives/libsystemd-shared_252.19-1~deb12u1_amd64.deb: Cannot write: No space left on device
tar: ./var/cache/apt/archives/pinentry-curses_1.2.1-1_amd64.deb: Cannot write: No space left on device
tar: ./var/cache/apt/archives/slirp4netns_1.2.0-1_amd64.deb: Cannot write: No space left on device
tar: ./var/cache/apt/archives/libgnutls30_3.7.9-2+deb12u2_amd64.deb: Cannot write: No space left on device
tar: ./var/cache/apt/archives/curl_7.88.1-10+deb12u5_amd64.deb: Cannot write: No space left on device
[2827 lines of no space errors]
tar: ./var/lib/docker/overlay2/l/XBMIVSF6LDU4M7TI3JFUXEXJ3X: Cannot create symlink to '../2fdc0c753d567ac0c5fab5dc197fbe2bdf5a48e8dd9d1c65c46a7c1500a115db/diff': No space left on device
tar: ./var/lib/docker/overlay2/l/O6LY3LIEDJZ26WC7SRG4FHVGDZ: Cannot create symlink to '../33b7fd9ad545d6d184f6321ca3242b1c96dfb5e0a52a5d3c98bad28084280c61/diff': No space left on device
Total bytes read: 69654425600 (65GiB, 275MiB/s)
tar: Exiting with failure status due to previous errors
TASK ERROR: unable to restore CT 102 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - --zstd --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/102/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2
 
Please also post the config of the backup, you can use the Show Configuration button from the top-bar of the Backup content tab of a storage or guest for that.

Because it might be that the inner/target filesystem runs out of space or inodes when extracting the backup, not the root filesystem from your PVE node itself.
Which could, e.g., stem from the default 5% reservation for root that ext4 makes, or a to small disk size in the config (however that could happen).
 
alexskysilk said:
No it doesnt ;)

grep sdb /etc/mtab
df -i /dev/sdb
Click to expand...
sorry, linux noob :p


Code: 
root@server000:/var/log# grep sdb /etc/mtab
/dev/sdb /mnt/sdb ext4 rw,relatime,errors=remount-ro 0 0
root@server000:/var/log# df -i /dev/sdb
Filesystem       Inodes IUsed    IFree IUse% Mounted on
/dev/sdb       61054976    41 61054935    1% /mnt/sdb


t.lamprecht said:
Please also post the config of the backup, you can use the Show Configuration button from the top-bar of the Backup content tab of a storage or guest for that.

Because it might be that the inner/target filesystem runs out of space or inodes when extracting the backup, not the root filesystem from your PVE node itself.
Which could, e.g., stem from the default 5% reservation for root that ext4 makes, or a to small disk size in the config (however that could happen).
Click to expand...

Code: 
#comment
arch: amd64
cores: 4
features: nesting=1
hostname: gitlab
memory: 5120
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:21:C1:6F,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: sdb:102/vm-102-disk-0.raw,size=70G
swap: 8192
unprivileged: 1
 
moxer_0815 said:
can you give me a hint how i can achive this?
Click to expand...
I never tried. it looks like vzdump doesnt alllow resize on restore, at least not anywhere I can see. you would have to unpack the config from the tarball, edit it, and put it back- not worth the effort. there's gotta be some other way.

@t.lamprecht would this be an issue with the destination store being a filestore on ext4? I have to admit I never had this configuration...
 
I have tested a little further: installed Proxmox on a second similar PC. Then connected an external hard disk there and formatted the whole thing with xfs. The backup can then be restored to this disk without any problems. This is good for someone who has lost the data from the backup (you can still access the NAS via the backup archive if necessary) but unfortunately it is not a solution to my problem, as xfs is out of the question in the productive system.


@t.lamprecht Do you have any ideas on how this could work with ext4?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back