[SOLVED] Backup woes

A

alh

Member
Proxmox Subscriber
Jul 7, 2021
48
5
13
46
Unfortunately in the past weeks having reliable backups of our PVE has been impossible. There is never any run that manages to backup all VMs and LXCs without error.

We already set the tmp directory to a local one but have no idea on how to improve the situation further.

All backup storage is SMB (Synology NAS & Hetzner StorageBox) and the errors are as such:

Code: 
command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/tmp/vzdumptmp827990_100/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | cstream -t 51200000 | zstd --rsyncable '--threads=1' >/mnt/pve/storagebox/dump/vzdump-lxc-100-2022_05_05-01_11_04.tar.dat' failed: exit code 1

Code: 
command 'qm shutdown 103 --skiplock --keepActive --timeout 600' failed: exit code 255

Code: 
zstd --rsyncable --threads=1 failed - wrong exit status 1

If anyone has any ideas, they are most welcome (it seemed to have worked fine back under PVE6).
 
Please provide the output of pveversion -v as well as the complete backup task logs and the syslog around the backup time.
 
Code: 
root@srv1:~# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.11.22-4-pve)
pve-manager: 7.2-4 (running version: 7.2-4/ca9d43cc)
pve-kernel-5.15: 7.2-3
pve-kernel-helper: 7.2-3
pve-kernel-5.13: 7.1-9
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-3
pve-kernel-5.15.35-1-pve: 5.15.35-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-5-pve: 5.13.19-13
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.11.22-2-pve: 5.11.22-4
pve-kernel-5.11.22-1-pve: 5.11.22-2
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-1
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-2
libpve-storage-perl: 7.2-4
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.1-1
proxmox-backup-file-restore: 2.2.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-7
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

the logs and the syslog attached
 

Attachments

your log contains :

12 102: 2022-05-19 01:15:12 INFO: zstd: /*stdout*\: Input/output error␣
Click to expand...

this is most often a problem with the target storage, check if there is a problem on your storagebox
also check the permissions of it
 
I contacted Hetzner (actually did before) but they couldn't find any problems. I hope they can help this time. Will update as soon as I have feedback.
 
So, Hetzner moved our account to a new instance but the error remains the same. The question I have is:
- we configured vzdump to use /tmp on the local filesystem as temp dir
- to me it seems that the compression of the snapshot fails and not the transfer to a storage

Code: 
INFO: starting new backup job: vzdump 102 104 --compress zstd --quiet 1 --prune-backups 'keep-last=4' --mailto it@craze.toys --node srv1 --storage storagebox --mode snapshot --mailnotification always
INFO: HOOK: job-start
INFO: HOOK-ENV: dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;
INFO: Starting Backup of VM 102 (lxc)
INFO: Backup started at 2022-05-24 04:11:02
INFO: status = running
INFO: CT Name: licenseserver
INFO: including mount point rootfs ('/') in backup
INFO: found old vzdump snapshot (force removal)
INFO: backup mode: snapshot
INFO: bandwidth limit: 50000 KB/s
INFO: ionice priority: 7
INFO: HOOK: backup-start snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.tar.zst;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.log;
INFO: HOOK: pre-stop snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.tar.zst;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.log;
INFO: create storage snapshot 'vzdump'
INFO: HOOK: pre-restart snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.tar.zst;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.log;
INFO: HOOK: post-restart snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.tar.zst;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.log;
INFO: creating vzdump archive '/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.tar.zst'
INFO: Total bytes written: 1357322240 (1.3GiB, 49MiB/s)
INFO: zstd: /*stdout*\: Input/output error
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 102 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/tmp/vzdumptmp2357150_102/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | cstream -t 51200000 | zstd --rsyncable '--threads=1' >/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.tar.dat' failed: exit code 1
INFO: Failed at 2022-05-24 04:11:29
INFO: HOOK: backup-abort snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.tar.zst;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.log;
INFO: HOOK: log-end snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.tar.zst;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-04_11_02.log;
INFO: Starting Backup of VM 104 (qemu)
INFO: Backup started at 2022-05-24 04:11:29
INFO: status = running
INFO: VM Name: srv1-dc
INFO: include disk 'scsi0' 'local-zfs:vm-104-disk-0' 24G
INFO: backup mode: snapshot
INFO: bandwidth limit: 50000 KB/s
INFO: ionice priority: 7
INFO: HOOK: backup-start snapshot 104
INFO: HOOK-ENV: vmtype=qemu;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=srv1-dc;target=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.vma.zst;logfile=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.log;
INFO: HOOK: pre-stop snapshot 104
INFO: HOOK-ENV: vmtype=qemu;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=srv1-dc;target=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.vma.zst;logfile=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.log;
INFO: HOOK: pre-restart snapshot 104
INFO: HOOK-ENV: vmtype=qemu;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=srv1-dc;target=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.vma.zst;logfile=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.log;
INFO: HOOK: post-restart snapshot 104
INFO: HOOK-ENV: vmtype=qemu;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=srv1-dc;target=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.vma.zst;logfile=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.log;
INFO: creating vzdump archive '/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'e4a9c7df-8d2d-40f2-96c3-1d30810bbb99'
INFO: resuming VM again
INFO:   0% (155.0 MiB of 24.0 GiB) in 3s, read: 51.7 MiB/s, write: 11.2 MiB/s
INFO:   1% (300.2 MiB of 24.0 GiB) in 6s, read: 48.4 MiB/s, write: 30.7 MiB/s
INFO:   2% (495.2 MiB of 24.0 GiB) in 10s, read: 48.8 MiB/s, write: 34.0 MiB/s
INFO:   3% (785.2 MiB of 24.0 GiB) in 16s, read: 48.3 MiB/s, write: 23.5 MiB/s
INFO:   4% (1.0 GiB of 24.0 GiB) in 21s, read: 49.0 MiB/s, write: 25.4 MiB/s
INFO:   5% (1.2 GiB of 24.0 GiB) in 26s, read: 48.0 MiB/s, write: 32.3 MiB/s
INFO:   6% (1.5 GiB of 24.0 GiB) in 31s, read: 49.0 MiB/s, write: 26.6 MiB/s
INFO:   7% (1.7 GiB of 24.0 GiB) in 36s, read: 48.4 MiB/s, write: 19.7 MiB/s
INFO:   8% (2.0 GiB of 24.0 GiB) in 41s, read: 49.0 MiB/s, write: 5.2 MiB/s
INFO:   9% (2.2 GiB of 24.0 GiB) in 46s, read: 48.0 MiB/s, write: 9.9 MiB/s
INFO:  10% (2.4 GiB of 24.0 GiB) in 51s, read: 49.0 MiB/s, write: 14.6 MiB/s
INFO:  11% (2.7 GiB of 24.0 GiB) in 56s, read: 48.5 MiB/s, write: 16.6 MiB/s
INFO:  12% (2.9 GiB of 24.0 GiB) in 1m 1s, read: 48.2 MiB/s, write: 14.2 MiB/s
INFO:  13% (3.1 GiB of 24.0 GiB) in 1m 6s, read: 49.0 MiB/s, write: 3.2 KiB/s
INFO:  14% (3.4 GiB of 24.0 GiB) in 1m 11s, read: 49.0 MiB/s, write: 3.2 KiB/s
INFO:  15% (3.6 GiB of 24.0 GiB) in 1m 16s, read: 48.0 MiB/s, write: 38.4 KiB/s
INFO:  16% (3.9 GiB of 24.0 GiB) in 1m 21s, read: 49.0 MiB/s, write: 0 B/s
INFO:  17% (4.1 GiB of 24.0 GiB) in 1m 26s, read: 49.0 MiB/s, write: 1.5 MiB/s
INFO:  18% (4.3 GiB of 24.0 GiB) in 1m 31s, read: 48.0 MiB/s, write: 12.8 MiB/s
INFO:  19% (4.6 GiB of 24.0 GiB) in 1m 36s, read: 49.0 MiB/s, write: 300.8 KiB/s
INFO:  20% (4.8 GiB of 24.0 GiB) in 1m 41s, read: 49.0 MiB/s, write: 0 B/s
INFO:  21% (5.0 GiB of 24.0 GiB) in 1m 46s, read: 48.1 MiB/s, write: 102.4 KiB/s
INFO:  22% (5.3 GiB of 24.0 GiB) in 1m 51s, read: 49.0 MiB/s, write: 0 B/s
INFO:  23% (5.6 GiB of 24.0 GiB) in 1m 57s, read: 48.3 MiB/s, write: 10.7 KiB/s
INFO:  24% (5.8 GiB of 24.0 GiB) in 2m 2s, read: 49.0 MiB/s, write: 0 B/s
INFO:  25% (6.0 GiB of 24.0 GiB) in 2m 7s, read: 48.0 MiB/s, write: 1.2 MiB/s
INFO:  26% (6.3 GiB of 24.0 GiB) in 2m 12s, read: 49.0 MiB/s, write: 18.9 MiB/s
INFO:  27% (6.5 GiB of 24.0 GiB) in 2m 17s, read: 49.0 MiB/s, write: 3.9 MiB/s
INFO:  28% (6.8 GiB of 24.0 GiB) in 2m 22s, read: 48.0 MiB/s, write: 36.0 KiB/s
INFO:  29% (7.0 GiB of 24.0 GiB) in 2m 27s, read: 49.0 MiB/s, write: 0 B/s
INFO:  30% (7.2 GiB of 24.0 GiB) in 2m 32s, read: 49.0 MiB/s, write: 0 B/s
INFO:  31% (7.5 GiB of 24.0 GiB) in 2m 37s, read: 48.0 MiB/s, write: 0 B/s
INFO:  32% (7.7 GiB of 24.0 GiB) in 2m 42s, read: 49.0 MiB/s, write: 0 B/s
INFO:  33% (7.9 GiB of 24.0 GiB) in 2m 47s, read: 49.0 MiB/s, write: 0 B/s
INFO:  34% (8.2 GiB of 24.0 GiB) in 2m 52s, read: 48.5 MiB/s, write: 3.9 MiB/s
INFO:  35% (8.4 GiB of 24.0 GiB) in 2m 57s, read: 49.0 MiB/s, write: 20.6 MiB/s
INFO:  36% (8.7 GiB of 24.0 GiB) in 3m 2s, read: 48.0 MiB/s, write: 18.0 MiB/s
INFO:  37% (8.9 GiB of 24.0 GiB) in 3m 7s, read: 49.0 MiB/s, write: 28.5 MiB/s
INFO:  38% (9.1 GiB of 24.0 GiB) in 3m 12s, read: 48.0 MiB/s, write: 29.0 MiB/s
INFO:  39% (9.4 GiB of 24.0 GiB) in 3m 17s, read: 49.0 MiB/s, write: 26.0 MiB/s
INFO:  40% (9.6 GiB of 24.0 GiB) in 3m 22s, read: 49.0 MiB/s, write: 12.3 MiB/s
INFO:  41% (9.9 GiB of 24.0 GiB) in 3m 28s, read: 48.3 MiB/s, write: 20.3 MiB/s
INFO:  42% (10.1 GiB of 24.0 GiB) in 3m 33s, read: 49.0 MiB/s, write: 8.7 MiB/s
INFO:  43% (10.4 GiB of 24.0 GiB) in 3m 38s, read: 48.2 MiB/s, write: 26.2 MiB/s
INFO:  44% (10.6 GiB of 24.0 GiB) in 3m 43s, read: 49.0 MiB/s, write: 14.9 MiB/s
INFO:  45% (10.8 GiB of 24.0 GiB) in 3m 48s, read: 48.0 MiB/s, write: 37.9 MiB/s
INFO:  46% (11.1 GiB of 24.0 GiB) in 3m 53s, read: 49.0 MiB/s, write: 10.3 MiB/s
INFO:  47% (11.3 GiB of 24.0 GiB) in 3m 58s, read: 48.0 MiB/s, write: 18.7 MiB/s
INFO:  48% (11.5 GiB of 24.0 GiB) in 4m 3s, read: 49.0 MiB/s, write: 11.3 MiB/s
INFO:  49% (11.8 GiB of 24.0 GiB) in 4m 8s, read: 49.0 MiB/s, write: 32.8 MiB/s
INFO:  50% (12.0 GiB of 24.0 GiB) in 4m 13s, read: 48.0 MiB/s, write: 34.5 MiB/s
INFO:  51% (12.3 GiB of 24.0 GiB) in 4m 18s, read: 49.0 MiB/s, write: 24.4 MiB/s
INFO:  52% (12.5 GiB of 24.0 GiB) in 4m 23s, read: 49.0 MiB/s, write: 18.1 MiB/s
INFO:  53% (12.7 GiB of 24.0 GiB) in 4m 28s, read: 48.0 MiB/s, write: 18.2 MiB/s
INFO:  54% (13.0 GiB of 24.0 GiB) in 4m 33s, read: 49.0 MiB/s, write: 20.7 MiB/s
INFO:  55% (13.2 GiB of 24.0 GiB) in 4m 38s, read: 48.0 MiB/s, write: 23.0 MiB/s
INFO:  56% (13.4 GiB of 24.0 GiB) in 4m 43s, read: 49.0 MiB/s, write: 32.4 MiB/s
INFO:  57% (13.7 GiB of 24.0 GiB) in 4m 48s, read: 49.0 MiB/s, write: 23.9 MiB/s
INFO:  58% (13.9 GiB of 24.0 GiB) in 4m 53s, read: 48.0 MiB/s, write: 13.0 MiB/s
INFO:  59% (14.2 GiB of 24.0 GiB) in 4m 58s, read: 49.0 MiB/s, write: 12.2 MiB/s
INFO:  60% (14.4 GiB of 24.0 GiB) in 5m 3s, read: 49.0 MiB/s, write: 14.9 MiB/s
INFO:  61% (14.7 GiB of 24.0 GiB) in 5m 9s, read: 48.3 MiB/s, write: 0 B/s
INFO:  62% (14.9 GiB of 24.0 GiB) in 5m 14s, read: 49.0 MiB/s, write: 0 B/s
INFO:  63% (15.2 GiB of 24.0 GiB) in 5m 19s, read: 48.0 MiB/s, write: 0 B/s
INFO:  64% (15.4 GiB of 24.0 GiB) in 5m 24s, read: 49.0 MiB/s, write: 0 B/s
INFO:  65% (15.6 GiB of 24.0 GiB) in 5m 30s, read: 40.0 MiB/s, write: 2.7 KiB/s
INFO:  66% (15.9 GiB of 24.0 GiB) in 5m 35s, read: 49.0 MiB/s, write: 0 B/s
INFO:  67% (16.1 GiB of 24.0 GiB) in 5m 40s, read: 49.0 MiB/s, write: 9.6 KiB/s
INFO:  68% (16.3 GiB of 24.0 GiB) in 5m 45s, read: 48.0 MiB/s, write: 8.8 KiB/s
INFO:  69% (16.6 GiB of 24.0 GiB) in 5m 50s, read: 49.0 MiB/s, write: 819.0 B/s
INFO:  70% (16.8 GiB of 24.0 GiB) in 5m 55s, read: 49.0 MiB/s, write: 0 B/s
INFO:  71% (17.1 GiB of 24.0 GiB) in 6m, read: 48.0 MiB/s, write: 0 B/s
INFO:  72% (17.3 GiB of 24.0 GiB) in 6m 5s, read: 49.0 MiB/s, write: 0 B/s
INFO:  73% (17.5 GiB of 24.0 GiB) in 6m 10s, read: 48.0 MiB/s, write: 0 B/s
INFO:  74% (17.8 GiB of 24.0 GiB) in 6m 15s, read: 49.0 MiB/s, write: 6.4 KiB/s
INFO:  75% (18.0 GiB of 24.0 GiB) in 6m 20s, read: 48.1 MiB/s, write: 1.6 KiB/s
INFO:  76% (18.2 GiB of 24.0 GiB) in 6m 25s, read: 49.0 MiB/s, write: 51.2 KiB/s
INFO:  77% (18.5 GiB of 24.0 GiB) in 6m 30s, read: 49.0 MiB/s, write: 0 B/s
INFO:  78% (18.8 GiB of 24.0 GiB) in 6m 36s, read: 48.3 MiB/s, write: 0 B/s
INFO:  79% (19.0 GiB of 24.0 GiB) in 6m 41s, read: 48.0 MiB/s, write: 0 B/s
INFO:  80% (19.2 GiB of 24.0 GiB) in 6m 46s, read: 49.0 MiB/s, write: 0 B/s
INFO:  81% (19.5 GiB of 24.0 GiB) in 6m 51s, read: 49.0 MiB/s, write: 0 B/s
INFO:  82% (19.7 GiB of 24.0 GiB) in 6m 56s, read: 48.0 MiB/s, write: 0 B/s
INFO:  83% (20.0 GiB of 24.0 GiB) in 7m 1s, read: 49.0 MiB/s, write: 0 B/s
INFO:  84% (20.2 GiB of 24.0 GiB) in 7m 6s, read: 48.1 MiB/s, write: 10.5 MiB/s
INFO:  85% (20.4 GiB of 24.0 GiB) in 7m 11s, read: 49.0 MiB/s, write: 8.4 MiB/s
INFO:  86% (20.7 GiB of 24.0 GiB) in 7m 16s, read: 49.0 MiB/s, write: 0 B/s
INFO:  87% (20.9 GiB of 24.0 GiB) in 7m 21s, read: 48.0 MiB/s, write: 0 B/s
INFO:  88% (21.1 GiB of 24.0 GiB) in 7m 26s, read: 49.0 MiB/s, write: 0 B/s
INFO:  89% (21.4 GiB of 24.0 GiB) in 7m 31s, read: 49.0 MiB/s, write: 0 B/s
INFO:  90% (21.6 GiB of 24.0 GiB) in 7m 36s, read: 48.0 MiB/s, write: 0 B/s
INFO:  91% (21.9 GiB of 24.0 GiB) in 7m 41s, read: 49.0 MiB/s, write: 0 B/s
INFO:  92% (22.1 GiB of 24.0 GiB) in 7m 46s, read: 48.0 MiB/s, write: 1.6 KiB/s
INFO:  93% (22.3 GiB of 24.0 GiB) in 7m 51s, read: 49.0 MiB/s, write: 0 B/s
INFO:  94% (22.6 GiB of 24.0 GiB) in 7m 56s, read: 49.0 MiB/s, write: 0 B/s
INFO:  95% (22.8 GiB of 24.0 GiB) in 8m 2s, read: 48.3 MiB/s, write: 0 B/s
INFO:  96% (23.1 GiB of 24.0 GiB) in 8m 7s, read: 49.0 MiB/s, write: 8.4 MiB/s
INFO:  97% (23.3 GiB of 24.0 GiB) in 8m 12s, read: 48.0 MiB/s, write: 47.5 MiB/s
INFO:  98% (23.6 GiB of 24.0 GiB) in 8m 17s, read: 49.0 MiB/s, write: 48.9 MiB/s
INFO:  99% (23.8 GiB of 24.0 GiB) in 8m 22s, read: 48.0 MiB/s, write: 47.8 MiB/s
zstd: /*stdout*\: Input/output error
INFO: 100% (24.0 GiB of 24.0 GiB) in 8m 27s, read: 41.8 MiB/s, write: 41.3 MiB/s
INFO: backup is sparse: 18.85 GiB (78%) total zero data
INFO: transferred 24.00 GiB in 507 seconds (48.5 MiB/s)
ERROR: Backup of VM 104 failed - zstd --rsyncable --threads=1 failed - wrong exit status 1
INFO: Failed at 2022-05-24 04:19:56
INFO: HOOK: backup-abort snapshot 104
INFO: HOOK-ENV: vmtype=qemu;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=srv1-dc;target=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.vma.zst;logfile=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.log;
INFO: HOOK: log-end snapshot 104
INFO: HOOK-ENV: vmtype=qemu;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=srv1-dc;target=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.vma.zst;logfile=/mnt/pve/storagebox/dump/vzdump-qemu-104-2022_05_24-04_11_29.log;
INFO: HOOK: job-end
INFO: HOOK-ENV: dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;
 
does it work with a different compression (e.g. gzip) or without compression?
 
Same error

Code: 
INFO: creating vzdump archive '/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_02_54.tar.gz'

INFO: Total bytes written: 1357475840 (1.3GiB, 28MiB/s)

INFO: gzip: stdout: Input/output error
 
It works without compression:

Code: 
INFO: HOOK: job-init
INFO: HOOK-ENV: storeid=storagebox;
INFO: starting new backup job: vzdump 102 --node srv1 --compress 0 --all 0 --mailnotification always --storage storagebox --mailto it@craze.toys --mode snapshot --prune-backups 'keep-last=4'
INFO: HOOK: job-start
INFO: HOOK-ENV: dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;
INFO: Starting Backup of VM 102 (lxc)
INFO: Backup started at 2022-05-24 10:12:28
INFO: status = running
INFO: CT Name: licenseserver
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: bandwidth limit: 50000 KB/s
INFO: ionice priority: 7
INFO: HOOK: backup-start snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.tar;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.log;
INFO: HOOK: pre-stop snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.tar;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.log;
INFO: create storage snapshot 'vzdump'
INFO: HOOK: pre-restart snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.tar;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.log;
INFO: HOOK: post-restart snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.tar;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.log;
INFO: creating vzdump archive '/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.tar'
INFO: Total bytes written: 1357475840 (1.3GiB, 49MiB/s)
INFO: archive file size: 1.26GB
INFO: prune older backups with retention: keep-last=4
INFO: pruned 0 backup(s)
INFO: HOOK: backup-end snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.tar;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.log;
INFO: Warning: Permanently added the ECDSA host key for IP address '[162.55.10.26]:23' to the list of known hosts.
INFO: cleanup temporary 'vzdump' snapshot
INFO: Finished Backup of VM 102 (00:00:41)
INFO: Backup finished at 2022-05-24 10:13:09
INFO: HOOK: log-end snapshot 102
INFO: HOOK-ENV: vmtype=lxc;dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;hostname=licenseserver;target=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.tar;logfile=/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_24-10_12_28.log;
INFO: HOOK: job-end
INFO: HOOK-ENV: dumpdir=/mnt/pve/storagebox/dump;storeid=storagebox;
INFO: Backup job finished successfully
TASK OK
 
could you try configuring the dumpdir / tmpdir to a local directory (not on the target storage box) ?
 
the theory is that the target storage box does not handle the writes of the compression tools/pipe correctly...
so when the tools write onto local storage it works.
 
  • Like
Reactions: alh
Don't count your chickens before they are hatched... last night we go again:

Code: 
102: 2022-05-25 01:11:00 INFO: creating vzdump archive '/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_25-01_11_00.tar.zst'
102: 2022-05-25 01:11:27 INFO: Total bytes written: 1357629440 (1.3GiB, 49MiB/s)
102: 2022-05-25 01:11:29 INFO: zstd: /*stdout*\: Input/output error
102: 2022-05-25 01:11:29 INFO: cleanup temporary 'vzdump' snapshot
102: 2022-05-25 01:11:29 ERROR: Backup of VM 102 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/tmp/vzdumptmp2182845_102/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | cstream -t 51200000 | zstd --rsyncable '--threads=1' >/mnt/pve/storagebox/dump/vzdump-lxc-102-2022_05_25-01_11_00.tar.dat' failed: exit code 1

I don't understand it... why does zstd fail on /tmp?
 
i guess you must configure both 'tmpdir' and 'dumpdir' for it to work
 
mhmm... seems i misremembered the options... AFAICS it's not possible to create the archive at a different path than the real target path
one way to do it would be to let it backup to a different (local) storage, and use the vzdump hookscripts to move them to your storage box

edit: but if those options did not do what i thought, it must mean that the backup worked temporarily? i'd not rule out some weird quirks with the storage box...
 
You must log in or register to reply here.
Top Bottom