Backup failes all the time

G

Gamienator

Member
Mar 16, 2021
46
6
13
33
Hello everyone,

I got an issue here that maybe you're able to help me.

On my root Server on my hoster Hetzner is the latest Proxmox VE running. For backup reasons I ordered a "Storage Box", which is a backup place that can be reached via SMB/CIFS, WebDAV etc. After connecting the box via SMB/CIFS I run my backups. Sadly every backup fails for my LXC Containers. The VMs are running totally fine:

Code: 
vzdump 105 106 108 --prune-backups 'keep-last=2' --storage storagebox --compress zstd --quiet 1 --mailnotification always --node proxmox --mailto me@mail.com --mode suspend --notes-template '{{guestname}}'


105: 2023-01-02 04:00:01 INFO: Starting Backup of VM 105 (lxc)
105: 2023-01-02 04:00:01 INFO: status = running
105: 2023-01-02 04:00:01 INFO: backup mode: suspend
105: 2023-01-02 04:00:01 INFO: ionice priority: 7
105: 2023-01-02 04:00:01 INFO: CT Name: stephan-seedbox
105: 2023-01-02 04:00:01 INFO: including mount point rootfs ('/') in backup
105: 2023-01-02 04:00:01 INFO: starting first sync /proc/1640275/root/ to /var/tmp/vzdumptmp1551178_105
105: 2023-01-02 11:01:08 ERROR: Backup of VM 105 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/1640275/root//./ /var/tmp/vzdumptmp1551178_105' failed: exit code 23

106: 2023-01-02 11:01:08 INFO: Starting Backup of VM 106 (lxc)
106: 2023-01-02 11:01:08 INFO: status = running
106: 2023-01-02 11:01:08 INFO: backup mode: suspend
106: 2023-01-02 11:01:08 INFO: ionice priority: 7
106: 2023-01-02 11:01:08 INFO: CT Name: stephan-nextcloud
106: 2023-01-02 11:01:08 INFO: including mount point rootfs ('/') in backup
106: 2023-01-02 11:01:08 INFO: starting first sync /proc/3468251/root/ to /var/tmp/vzdumptmp1551178_106
106: 2023-01-02 12:15:42 ERROR: Backup of VM 106 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/3468251/root//./ /var/tmp/vzdumptmp1551178_106' failed: exit code 23

108: 2023-01-02 12:15:42 INFO: Starting Backup of VM 108 (lxc)
108: 2023-01-02 12:15:42 INFO: status = running
108: 2023-01-02 12:15:42 INFO: backup mode: suspend
108: 2023-01-02 12:15:42 INFO: ionice priority: 7
108: 2023-01-02 12:15:42 INFO: CT Name: stephan-plex
108: 2023-01-02 12:15:42 INFO: including mount point rootfs ('/') in backup
108: 2023-01-02 12:15:42 INFO: starting first sync /proc/24013/root/ to /var/tmp/vzdumptmp1551178_108
108: 2023-01-02 12:52:56 INFO: first sync finished - transferred 181.80G bytes in 2234s
108: 2023-01-02 12:52:56 INFO: suspending guest
108: 2023-01-02 12:52:56 INFO: starting final sync /proc/24013/root/ to /var/tmp/vzdumptmp1551178_108
108: 2023-01-02 12:53:21 INFO: final sync finished - transferred 7.15M bytes in 25s
108: 2023-01-02 12:53:21 INFO: resuming guest
108: 2023-01-02 12:53:21 INFO: guest is online again after 25 seconds
108: 2023-01-02 12:53:21 INFO: creating vzdump archive '/mnt/pve/storagebox/dump/vzdump-lxc-108-2023_01_02-12_15_42.tar.zst'
108: 2023-01-02 13:32:30 INFO: Total bytes written: 181692794880 (170GiB, 74MiB/s)
108: 2023-01-02 13:32:33 INFO: zstd: /*stdout*\: Input/output error
108: 2023-01-02 13:33:23 ERROR: Backup of VM 108 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp1551178_108' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/var/tmp/vzdumptmp1551178_108' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . | zstd --rsyncable '--threads=1' >/mnt/pve/storagebox/dump/vzdump-lxc-108-2023_01_02-12_15_42.tar.dat' failed: exit code 1

Can somebody can give a hint How I'm able to debug it?

Cheers,
Gamie
 
Most likely there is a permission problem for some files.
Are the filesystem permissions on the storage box configured correctly?
 
Well, I can'T configure any permissions on that storage box. I get the credentials which got full access.
 
Try setting a "tmpdir" with 777 rights (for example "/tmp") in "/etc/vzdump.conf". Otherwise UID 0 and 100000 need read/write access on that SMB share to be able to backup unprivileged LXCs.
 
  • Like
Reactions: keeka
Dunuin said:
Try setting a "tmpdir" with 777 rights (for example "/tmp") in "/etc/vzdump.conf". Otherwise UID 0 and 100000 need read/write access on that SMB share to be able to backup unprivileged LXCs.
Click to expand...

Hmm, didn't worked.

I created /tmp/backup with 777 and run another backup. Thats the result:


Code: 
109: 2023-01-09 13:35:56 INFO: Starting Backup of VM 109 (lxc)
109: 2023-01-09 13:35:56 INFO: status = running
109: 2023-01-09 13:35:56 INFO: CT Name: stephan-websites
109: 2023-01-09 13:35:56 INFO: including mount point rootfs ('/') in backup
109: 2023-01-09 13:35:56 INFO: backup mode: snapshot
109: 2023-01-09 13:35:56 INFO: ionice priority: 7
109: 2023-01-09 13:35:56 INFO: create storage snapshot 'vzdump'
109: 2023-01-09 13:35:56 INFO: creating vzdump archive '/mnt/pve/storagebox/dump/vzdump-lxc-109-2023_01_09-13_35_56.tar.zst'
109: 2023-01-09 14:07:33 INFO: Total bytes written: 43113687040 (41GiB, 22MiB/s)
109: 2023-01-09 14:07:34 INFO: zstd: /*stdout*\: Input/output error
109: 2023-01-09 14:07:35 INFO: cleanup temporary 'vzdump' snapshot
109: 2023-01-09 14:07:36 ERROR: Backup of VM 109 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/tmp/backup/vzdumptmp4140504_109/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/storagebox/dump/vzdump-lxc-109-2023_01_09-13_35_56.tar.dat' failed: exit code 1
 
Hmm, I tried then with /tmp
But still no success with the Backup. Any other Suggestions or where I'm able to debug even more?
 
Please provide the output of cat /etc/pve/storage.cfg, df -h and mount.
 
mira said:
Please provide the output of cat /etc/pve/storage.cfg, df -h and mount.
Click to expand...
Code: 
dir: local
        path /var/lib/vz
        content backup,vztmpl,iso

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        sparse 1

zfspool: ssdpool
        pool ssdpool
        content rootdir,images
        mountpoint /ssdpool
        nodes proxmox

dir: BigStorage
        path /mnt/pve/bigstoragebox
        content backup
        prune-backups keep-all=1
        shared 0

Code: 
Filesystem                           Size  Used Avail Use% Mounted on
udev                                  32G     0   32G   0% /dev
tmpfs                                6.3G  1.8M  6.3G   1% /run
rpool/ROOT/pve-1                     7.2T  8.2G  7.2T   1% /
tmpfs                                 32G   49M   32G   1% /dev/shm
tmpfs                                5.0M     0  5.0M   0% /run/lock
rpool                                7.2T  128K  7.2T   1% /rpool
rpool/ROOT                           7.2T  128K  7.2T   1% /rpool/ROOT
rpool/data                           7.2T  256K  7.2T   1% /rpool/data
/dev/fuse                            128M   36K  128M   1% /etc/pve
ssdpool                              545G  128K  545G   1% /ssdpool
rpool/data/subvol-100-disk-0          25G  1.2G   24G   5% /rpool/data/subvol-100-disk-0
rpool/data/subvol-102-disk-0          25G  1.5G   24G   6% /rpool/data/subvol-102-disk-0
rpool/data/subvol-198-disk-0          55G  380M   55G   1% /rpool/data/subvol-198-disk-0
ssdpool/subvol-103-disk-0             25G   13G   13G  51% /ssdpool/subvol-103-disk-0
rpool/data/subvol-104-disk-0          25G   11G   15G  43% /rpool/data/subvol-104-disk-0
rpool/data/subvol-105-disk-0         2.3T  1.7T  610G  74% /rpool/data/subvol-105-disk-0
rpool/data/subvol-106-disk-0         1.2T  328G  823G  29% /rpool/data/subvol-106-disk-0
rpool/data/subvol-107-disk-0          50G  9.3G   41G  19% /rpool/data/subvol-107-disk-0
rpool/data/subvol-108-disk-0         200G  169G   32G  85% /rpool/data/subvol-108-disk-0
rpool/data/subvol-109-disk-0         100G   24G   77G  24% /rpool/data/subvol-109-disk-0
ssdpool/subvol-110-disk-0             55G  4.5G   51G   9% /ssdpool/subvol-110-disk-0
//u32211.your-storagebox.de/backup  5.0T  813G  4.3T  16% /mnt/pve/bigstoragebox
tmpfs                                6.3G     0  6.3G   0% /run/user/0

Code: 
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32854168k,nr_inodes=8213542,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6577532k,mode=755,inode64)
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21416)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
rpool on /rpool type zfs (rw,relatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,relatime,xattr,noacl)
rpool/data on /rpool/data type zfs (rw,relatime,xattr,noacl)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
ssdpool on /ssdpool type zfs (rw,xattr,noacl)
rpool/data/subvol-100-disk-0 on /rpool/data/subvol-100-disk-0 type zfs (rw,relatime,xattr,posixacl)
rpool/data/subvol-102-disk-0 on /rpool/data/subvol-102-disk-0 type zfs (rw,relatime,xattr,posixacl)
rpool/data/subvol-198-disk-0 on /rpool/data/subvol-198-disk-0 type zfs (rw,relatime,xattr,posixacl)
ssdpool/subvol-103-disk-0 on /ssdpool/subvol-103-disk-0 type zfs (rw,xattr,posixacl)
rpool/data/subvol-104-disk-0 on /rpool/data/subvol-104-disk-0 type zfs (rw,relatime,xattr,posixacl)
rpool/data/subvol-105-disk-0 on /rpool/data/subvol-105-disk-0 type zfs (rw,relatime,xattr,posixacl)
rpool/data/subvol-106-disk-0 on /rpool/data/subvol-106-disk-0 type zfs (rw,relatime,xattr,posixacl)
rpool/data/subvol-107-disk-0 on /rpool/data/subvol-107-disk-0 type zfs (rw,relatime,xattr,posixacl)
rpool/data/subvol-108-disk-0 on /rpool/data/subvol-108-disk-0 type zfs (rw,relatime,xattr,posixacl)
rpool/data/subvol-109-disk-0 on /rpool/data/subvol-109-disk-0 type zfs (rw,relatime,xattr,posixacl)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
ssdpool/subvol-110-disk-0 on /ssdpool/subvol-110-disk-0 type zfs (rw,xattr,posixacl)
//u32211.your-storagebox.de/backup on /mnt/pve/bigstoragebox type cifs (rw,relatime,vers=3.1.1,cache=strict,username=u32211,uid=0,noforceuid,gid=0,noforcegid,addr=116.202.54.30,file_mode=0755,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,noperm,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=6577528k,nr_inodes=1644382,mode=700,inode64)
 
Last edited:
Is there anything in the journal during the time the backup runs and fails?
Any issues with the network storage during that time?

Looks like we had a few reports together with Hetzner's storage boxes.
 
Last edited:
Not really, because it instant fails. Which journals do you exactly mean?

Weirdly I was now able to make sometimes backups with a weird setup.

I ordered a VM on Hetzner, installed on it Debian 11 and Proxmox Backup Server. Mounting it via CIFS and that mountoptions makes it possible to make a backup, but really slowly and successrate 90%. But that was just to test if there is any possibility to mount CIFS and backup LXC. Without the additional VM would be great :p
 
It would seem those issues appear when the backup storage is too slow, which doesn't surprise me when it comes to Hetzner.
PBS seems to be a bit more resilient in this case, but still network storages should never lose the connection since both CIFS and NFS can't handle this all to well.
 
You must log in or register to reply here.
Top Bottom