[SOLVED] vzdump: backupp failed could not activate storage 'backup': storage 'backup' is not available on node 'SRV-01'

[FloUhl]

vzdump backup status (MAIL) : backup failed: could not activate storage 'Backup': storage 'Backup' is not available on node 'SRV-01'

is the Error I get via Mail

For context I got a cluster with 3 nodes SRV-01, SRV-02, SRV-03. On 01 I got a few Container running, 02 is for repclicating 01, and 03 is for the backup of the container.
But it seems that SRV-01 can't access the storage on SRV-03.

 

[fiona]

Hi,
Proxmox VE cannot magically share storages across nodes. You need to set up a file-based shared storage yourself (e.g NFS/CIFS/CephFS). Configuring an NFS export and making it available on the other node is probably the easiest. Then you can add it to the storage configuration (Datacenter > Storage > Add in the UI, see here for descriptions and parameters).

If you want, it's also possible to setup a PBS instance alongside Proxmox VE, but it's a bit more involved and requires some resources of course.

 

[FloUhl]

Now I get this Error

unable to create temporary directory '/mnt/pve/BackupNFS/dump/vzdump-lxc-100-2022_08_24-13_35_46.tmp' at /usr/share/perl5/PVE/VZDump.pm line 919.

it shouldn't even go /mnt/pve/BackupNFS instead it should go to /mnt/backup-storage

 

[fiona]

Please ensure that the permissions for the export are configured correctly on the NFS server. Note that for unprivileged containers you also need to allow UID 100000. Alternatively, you can configure a local tmpdir by editing /etc/vzdump.conf (needs to be done on each node).

 

[FloUhl]

Thx, but it just keeps getting worse,

command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/tmp/vzdumptmp3152140_100/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_24-15_06_22.tar.dat' failed: exit code 1

I just wanna give up at this point

 

[fiona]

Thx, but it just keeps getting worse,

command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/tmp/vzdumptmp3152140_100/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_24-15_06_22.tar.dat' failed: exit code 1

I just wanna give up at this point

It's getting much further along, so it's actually getting better. In the beginning, it couldn't even access the storage
Can you share the complete backup task log and the output of pveversion -v?

 

[FloUhl]

The Error on the mail I recieve

100: 2022-08-24 21:00:04 INFO: Starting Backup of VM 100 (lxc)
100: 2022-08-24 21:00:04 INFO: status = stopped
100: 2022-08-24 21:00:04 INFO: backup mode: stop
100: 2022-08-24 21:00:04 INFO: ionice priority: 7
100: 2022-08-24 21:00:04 INFO: CT Name: ct-srv-dhcp-01
100: 2022-08-24 21:00:04 INFO: including mount point rootfs ('/') in backup
100: 2022-08-24 21:00:04 INFO: creating vzdump archive '/mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_24-21_00_04.tar.zst'
100: 2022-08-24 21:00:04 INFO: /bin/bash: line 1: /mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_24-21_00_04.tar.dat: Permission denied
100: 2022-08-24 21:00:04 ERROR: Backup of VM 100 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/tmp/vzdumptmp3416220_100/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_24-21_00_04.tar.dat' failed: exit code 1

Pveversion -v

proxmox-ve: 7.2-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.15.35-2-pve: 5.15.35-5
pve-kernel-5.15.35-1-pve: 5.15.35-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

 

[fiona]

The Error on the mail I recieve

100: 2022-08-24 21:00:04 INFO: Starting Backup of VM 100 (lxc)
100: 2022-08-24 21:00:04 INFO: status = stopped
100: 2022-08-24 21:00:04 INFO: backup mode: stop
100: 2022-08-24 21:00:04 INFO: ionice priority: 7
100: 2022-08-24 21:00:04 INFO: CT Name: ct-srv-dhcp-01
100: 2022-08-24 21:00:04 INFO: including mount point rootfs ('/') in backup
100: 2022-08-24 21:00:04 INFO: creating vzdump archive '/mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_24-21_00_04.tar.zst'
100: 2022-08-24 21:00:04 INFO: /bin/bash: line 1: /mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_24-21_00_04.tar.dat: Permission denied

So it still seems to be a permission problem. How is the export configured on the server-side? What is the output of stat /mnt/pve/Backup and mount | grep /mnt/pve/Backup?

Can you e.g. create a directory in /mnt/pve/Backup manually?

100: 2022-08-24 21:00:04 ERROR: Backup of VM 100 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/tmp/vzdumptmp3416220_100/' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_24-21_00_04.tar.dat' failed: exit code 1

 

[FloUhl]

root@SRV-03:/mnt/pve/Backup# stat /mnt/pve/Backup/
File: /mnt/pve/Backup/
Size: 4096 Blocks: 8 IO Block: 1048576 directory
Device: 2dh/45d Inode: 2 Links: 5
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2022-08-24 13:51:55.711823257 +0200
Modify: 2022-08-23 13:06:16.382101648 +0200
Change: 2022-08-23 13:06:16.382101648 +0200
Birth: -
root@SRV-03:/mnt/pve/Backup# mount | grep /mnt/pve/Backup/
root@SRV-03:/mnt/pve/Backup# ls
dump images lost+found
root@SRV-03:/mnt/pve/Backup# mkdir test
mkdir: cannot create directory ‘test’: Permission denied

I cant change the permissions, there is a error comng than.
And I dont even want it to save it in /mnt/pve/backup, it should go to /mnt/backup-storage, but if i change that it doesnt work at all.

 

[fiona]

root@SRV-03:/mnt/pve/Backup# mount | grep /mnt/pve/Backup/

Strange..what is the output of findmnt?

root@SRV-03:/mnt/pve/Backup# ls
dump images lost+found
root@SRV-03:/mnt/pve/Backup# mkdir test
mkdir: cannot create directory ‘test’: Permission denied

I cant change the permissions, there is a error comng than.

You need to configure the correct permissions on the server side. There's a lot of NFS tutorials if you search for them.

And I dont even want it to save it in /mnt/pve/backup, it should go to /mnt/backup-storage, but if i change that it doesnt work at all.

On the client side, NFS storages in Proxmox VE are mounted in /mnt/pve/<storeid>. Is /mnt/backup-storage the path on the server? Can you post the contents of /etc/pve/storage.cfg?

 

[FloUhl]

Well ok, I fixed the permission issue and got the first backup to run, but it seems to save a copy of the backup on SRV-01, can I somehow dissable that or do I need to write a script to delete it

 

[fiona]

Glad to hear
The copy is there, because of the tmpdir setting. You can try removing that again, but you might need to also allow UID/GID 100000 on the NFS share for unprivileged containers.

 

[FloUhl]

Nop, we back to errors after I changed something I cant quite remember

INFO: starting new backup job: vzdump 100 --mode snapshot --all 0 --node SRV-01 --storage Backup --notes-template '{{guestname}}' --compress zstd
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2022-08-26 15:50:25
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: ct-srv-dhcp-01
INFO: including mount point rootfs ('/') in backup
INFO: creating vzdump archive '/mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_26-15_50_25.tar.zst'
INFO: tar: /mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_26-15_50_25.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
ERROR: Backup of VM 100 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_26-15_50_25.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/Backup/dump/vzdump-lxc-100-2022_08_26-15_50_25.tar.dat' failed: exit code 2
INFO: Failed at 2022-08-26 15:50:25
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[FloUhl]

Fixed it myself, this darn permission. It need all rights to work.