tmp: Cannot open: Permission denied

[MichaelS]

The backup of a CT fails with this:

INFO: starting new backup job: vzdump 300 --mode snapshot --remove 0 --storage Backups_Proxmox --node pve --compress zstd
INFO: Starting Backup of VM 300 (lxc)
INFO: Backup started at 2021-04-16 08:07:24
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: unifi-IP241
INFO: including mount point rootfs ('/') in backup
INFO: creating vzdump archive '/mnt/pve/Backups_Proxmox/dump/vzdump-lxc-300-2021_04_16-08_07_23.tar.zst'
INFO: tar: /mnt/pve/Backups_Proxmox/dump/vzdump-lxc-300-2021_04_16-08_07_23.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
ERROR: Backup of VM 300 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/Backups_Proxmox/dump/vzdump-lxc-300-2021_04_16-08_07_23.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/Backups_Proxmox/dump/vzdump-lxc-300-2021_04_16-08_07_23.tar.dat' failed: exit code 2
INFO: Failed at 2021-04-16 08:08:08
INFO: Backup job finished with errors
TASK ERROR: job errors

Interesting: I can create a snapshot though and thought there would be nearly no differences between backup and snapshot.
But the snapshot save data internally and the backup is designed to save on a NAS.

What can I do?

 

[aaron]

Interesting: I can create a snapshot though and thought there would be nearly no differences between backup and snapshot.

Technically there are quite some differences between a snapshot and a backup. Snapshots depend highly on the storage underneath for example, backups don't.

The error shows that PVE was not able to write the backup file to the network share as it does not have permission to do so.
Check the permissions of the PVE server wherever that network share is located (your NAS?).

 

[MichaelS]

There are four parts to backup: Three VMs and the one CT with the error.
This is the complete log for the PVE backup (sorry that I presented the above excerpt only in the first step):
You will realise that the VMs´ backup is fine, the CT backup comes with an error.

INFO: starting new backup job: vzdump 100 102 200 300 --mode snapshot --mailto michael@schaaf.com --compress zstd --mailnotification always --quiet 1 --storage Backups_Proxmox
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-04-16 02:00:01
INFO: status = running
INFO: VM Name: iobroker-IP201
INFO: include disk 'scsi0' 'local-lvm:vm-100-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/Backups_Proxmox/dump/vzdump-qemu-100-2021_04_16-02_00_01.vma.zst'
INFO: started backup task '21f36b1a-78c1-4a5e-9622-bd9f33199feb'
INFO: resuming VM again
INFO: 1% (577.4 MiB of 32.0 GiB) in 3s, read: 192.5 MiB/s, write: 134.1 MiB/s
INFO: 12% (4.2 GiB of 32.0 GiB) in 6s, read: 1.2 GiB/s, write: 102.6 MiB/s
INFO: 25% (8.1 GiB of 32.0 GiB) in 9s, read: 1.3 GiB/s, write: 95.3 MiB/s
INFO: 37% (12.1 GiB of 32.0 GiB) in 12s, read: 1.3 GiB/s, write: 103.3 MiB/s
INFO: 50% (16.2 GiB of 32.0 GiB) in 15s, read: 1.4 GiB/s, write: 112.4 MiB/s
INFO: 60% (19.4 GiB of 32.0 GiB) in 18s, read: 1.1 GiB/s, write: 142.4 MiB/s
INFO: 74% (23.9 GiB of 32.0 GiB) in 21s, read: 1.5 GiB/s, write: 93.1 MiB/s
INFO: 76% (24.6 GiB of 32.0 GiB) in 24s, read: 241.1 MiB/s, write: 160.3 MiB/s
INFO: 78% (25.2 GiB of 32.0 GiB) in 27s, read: 183.0 MiB/s, write: 170.4 MiB/s
INFO: 80% (25.7 GiB of 32.0 GiB) in 30s, read: 174.6 MiB/s, write: 167.2 MiB/s
INFO: 87% (28.1 GiB of 32.0 GiB) in 33s, read: 845.8 MiB/s, write: 128.1 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 36s, read: 1.3 GiB/s, write: 54.1 MiB/s
INFO: backup is sparse: 27.71 GiB (86%) total zero data
INFO: transferred 32.00 GiB in 36 seconds (910.2 MiB/s)
INFO: archive file size: 1.52GB
INFO: Finished Backup of VM 100 (00:00:52)
INFO: Backup finished at 2021-04-16 02:00:53
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2021-04-16 02:00:53
INFO: status = running
INFO: VM Name: Zigbee-IP211
INFO: include disk 'scsi0' 'local-lvm:vm-102-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/Backups_Proxmox/dump/vzdump-qemu-102-2021_04_16-02_00_53.vma.zst'
INFO: started backup task '33c12848-506b-4ca6-9ed8-3541cebc0c52'
INFO: resuming VM again
INFO: 1% (648.9 MiB of 32.0 GiB) in 3s, read: 216.3 MiB/s, write: 144.6 MiB/s
INFO: 3% (1.2 GiB of 32.0 GiB) in 6s, read: 179.7 MiB/s, write: 169.7 MiB/s
INFO: 5% (1.7 GiB of 32.0 GiB) in 9s, read: 197.2 MiB/s, write: 181.5 MiB/s
INFO: 7% (2.4 GiB of 32.0 GiB) in 12s, read: 228.5 MiB/s, write: 145.9 MiB/s
INFO: 9% (2.9 GiB of 32.0 GiB) in 15s, read: 172.7 MiB/s, write: 156.1 MiB/s
INFO: 10% (3.3 GiB of 32.0 GiB) in 18s, read: 141.3 MiB/s, write: 123.6 MiB/s
INFO: 11% (3.8 GiB of 32.0 GiB) in 21s, read: 168.6 MiB/s, write: 143.6 MiB/s
INFO: 13% (4.4 GiB of 32.0 GiB) in 24s, read: 197.6 MiB/s, write: 137.6 MiB/s
INFO: 19% (6.2 GiB of 32.0 GiB) in 27s, read: 612.8 MiB/s, write: 129.9 MiB/s
INFO: 32% (10.5 GiB of 32.0 GiB) in 30s, read: 1.4 GiB/s, write: 102.0 MiB/s
INFO: 38% (12.5 GiB of 32.0 GiB) in 33s, read: 665.5 MiB/s, write: 127.7 MiB/s
INFO: 43% (13.8 GiB of 32.0 GiB) in 36s, read: 462.4 MiB/s, write: 168.3 MiB/s
INFO: 61% (19.7 GiB of 32.0 GiB) in 39s, read: 2.0 GiB/s, write: 67.3 MiB/s
INFO: 74% (23.7 GiB of 32.0 GiB) in 42s, read: 1.3 GiB/s, write: 96.2 MiB/s
INFO: 88% (28.2 GiB of 32.0 GiB) in 45s, read: 1.5 GiB/s, write: 80.3 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 48s, read: 1.3 GiB/s, write: 73.0 MiB/s
INFO: backup is sparse: 26.00 GiB (81%) total zero data
INFO: transferred 32.00 GiB in 48 seconds (682.7 MiB/s)
INFO: archive file size: 2.19GB
INFO: Finished Backup of VM 102 (00:00:50)
INFO: Backup finished at 2021-04-16 02:01:43
INFO: Starting Backup of VM 200 (qemu)
INFO: Backup started at 2021-04-16 02:01:43
INFO: status = running
INFO: VM Name: Windows10-iP231
INFO: include disk 'scsi0' 'local-lvm:vm-200-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: pending configuration changes found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/Backups_Proxmox/dump/vzdump-qemu-200-2021_04_16-02_01_43.vma.zst'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task '51c3bfcd-1f3b-4dea-8dc9-188e36f5f9ce'
INFO: resuming VM again
INFO: 2% (735.9 MiB of 32.0 GiB) in 3s, read: 245.3 MiB/s, write: 197.1 MiB/s
INFO: 4% (1.4 GiB of 32.0 GiB) in 6s, read: 216.5 MiB/s, write: 186.6 MiB/s
INFO: 6% (2.0 GiB of 32.0 GiB) in 9s, read: 217.3 MiB/s, write: 191.0 MiB/s
INFO: 8% (2.7 GiB of 32.0 GiB) in 12s, read: 239.2 MiB/s, write: 181.9 MiB/s
INFO: 10% (3.3 GiB of 32.0 GiB) in 15s, read: 201.8 MiB/s, write: 186.2 MiB/s
INFO: 11% (3.8 GiB of 32.0 GiB) in 18s, read: 161.9 MiB/s, write: 157.4 MiB/s
INFO: 13% (4.3 GiB of 32.0 GiB) in 21s, read: 171.6 MiB/s, write: 160.9 MiB/s
INFO: 15% (5.1 GiB of 32.0 GiB) in 24s, read: 274.3 MiB/s, write: 154.8 MiB/s
INFO: 18% (6.0 GiB of 32.0 GiB) in 27s, read: 329.9 MiB/s, write: 169.0 MiB/s
INFO: 20% (6.7 GiB of 32.0 GiB) in 30s, read: 222.3 MiB/s, write: 156.4 MiB/s
INFO: 23% (7.5 GiB of 32.0 GiB) in 33s, read: 285.2 MiB/s, write: 196.2 MiB/s
INFO: 26% (8.4 GiB of 32.0 GiB) in 36s, read: 301.9 MiB/s, write: 175.6 MiB/s
INFO: 28% (9.2 GiB of 32.0 GiB) in 39s, read: 269.7 MiB/s, write: 188.5 MiB/s
INFO: 30% (9.8 GiB of 32.0 GiB) in 42s, read: 209.0 MiB/s, write: 184.3 MiB/s
INFO: 32% (10.4 GiB of 32.0 GiB) in 45s, read: 198.2 MiB/s, write: 171.1 MiB/s
INFO: 34% (10.9 GiB of 32.0 GiB) in 48s, read: 176.4 MiB/s, write: 137.4 MiB/s
INFO: 36% (11.5 GiB of 32.0 GiB) in 51s, read: 212.7 MiB/s, write: 198.7 MiB/s
INFO: 38% (12.2 GiB of 32.0 GiB) in 54s, read: 219.9 MiB/s, write: 163.6 MiB/s
INFO: 42% (13.6 GiB of 32.0 GiB) in 57s, read: 473.7 MiB/s, write: 162.0 MiB/s
INFO: 44% (14.1 GiB of 32.0 GiB) in 1m, read: 198.7 MiB/s, write: 179.4 MiB/s
INFO: 46% (14.8 GiB of 32.0 GiB) in 1m 3s, read: 229.0 MiB/s, write: 221.9 MiB/s
INFO: 48% (15.5 GiB of 32.0 GiB) in 1m 6s, read: 253.2 MiB/s, write: 232.6 MiB/s
INFO: 50% (16.1 GiB of 32.0 GiB) in 1m 9s, read: 180.2 MiB/s, write: 164.0 MiB/s
INFO: 51% (16.5 GiB of 32.0 GiB) in 1m 12s, read: 152.6 MiB/s, write: 152.4 MiB/s
INFO: 53% (17.0 GiB of 32.0 GiB) in 1m 15s, read: 158.5 MiB/s, write: 158.4 MiB/s
INFO: 54% (17.5 GiB of 32.0 GiB) in 1m 18s, read: 168.8 MiB/s, write: 163.6 MiB/s
INFO: 56% (18.0 GiB of 32.0 GiB) in 1m 21s, read: 183.8 MiB/s, write: 175.2 MiB/s
INFO: 58% (18.6 GiB of 32.0 GiB) in 1m 24s, read: 204.2 MiB/s, write: 180.7 MiB/s
INFO: 59% (19.1 GiB of 32.0 GiB) in 1m 27s, read: 171.2 MiB/s, write: 169.5 MiB/s
INFO: 61% (19.6 GiB of 32.0 GiB) in 1m 30s, read: 174.4 MiB/s, write: 166.5 MiB/s
INFO: 63% (20.3 GiB of 32.0 GiB) in 1m 33s, read: 244.1 MiB/s, write: 206.5 MiB/s
INFO: 69% (22.3 GiB of 32.0 GiB) in 1m 36s, read: 650.1 MiB/s, write: 120.2 MiB/s
INFO: 71% (23.0 GiB of 32.0 GiB) in 1m 39s, read: 241.7 MiB/s, write: 171.9 MiB/s
INFO: 72% (23.1 GiB of 32.0 GiB) in 1m 42s, read: 54.4 MiB/s, write: 54.4 MiB/s
INFO: 74% (24.0 GiB of 32.0 GiB) in 1m 47s, read: 171.1 MiB/s, write: 47.0 MiB/s
INFO: 90% (29.0 GiB of 32.0 GiB) in 1m 50s, read: 1.7 GiB/s, write: 0 B/s
INFO: 99% (32.0 GiB of 32.0 GiB) in 1m 53s, read: 1015.6 MiB/s, write: 124.4 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 1m 54s, read: 48.8 MiB/s, write: 46.8 MiB/s
INFO: backup is sparse: 14.26 GiB (44%) total zero data
INFO: transferred 32.00 GiB in 114 seconds (287.4 MiB/s)
INFO: archive file size: 8.55GB
INFO: Finished Backup of VM 200 (00:02:35)
INFO: Backup finished at 2021-04-16 02:04:18
INFO: Starting Backup of VM 300 (lxc)
INFO: Backup started at 2021-04-16 02:04:18
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: unifi-IP241
INFO: including mount point rootfs ('/') in backup
INFO: creating vzdump archive '/mnt/pve/Backups_Proxmox/dump/vzdump-lxc-300-2021_04_16-02_04_18.tar.zst'
INFO: tar: /mnt/pve/Backups_Proxmox/dump/vzdump-lxc-300-2021_04_16-02_04_18.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now
ERROR: Backup of VM 300 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/Backups_Proxmox/dump/vzdump-lxc-300-2021_04_16-02_04_18.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd --rsyncable '--threads=1' >/mnt/pve/Backups_Proxmox/dump/vzdump-lxc-300-2021_04_16-02_04_18.tar.dat' failed: exit code 2
INFO: Failed at 2021-04-16 02:04:18
INFO: Backup job finished with errors

TASK ERROR: job errors


So why should this be a permission issue on the target NAS where three parts of the backups do not have any issue?
Please let me know.

 

[operator]

Exact same behavior here. Some CT's can run the backup; some can't.

INFO: stopping vm
INFO: creating vzdump archive '/mnt/pve/backupNAS/dump/vzdump-lxc-111-2021_08_11-16_57_45.tar.zst'
INFO: tar: /mnt/pve/backupNAS/dump/vzdump-lxc-111-2021_08_11-16_57_45.tmp: Cannot open: Permission denied
INFO: tar: Error is not recoverable: exiting now

It's confusing because the archive that is created in the first step can't be reached. The Logfile which is located in the same backup directory on the NAS can be created - So the permission to write is granted.

I updated my Synology NAS to a new Firmware (DSM 7) these days but the NFS-Share and version is still the same and has also been recreated.

 

[fabian]

if the container is unprivileged you might need unprivileged access to the tmp/dump dirs, depending on backup mode.

 

[RokModTwo]

This tip was helpful, pointed me in the direction!

I have a Synology NFS share mounted - VMs would backup OK, containers did not. Checked and found that all containers were unprivileged. On the Synology, under NFS Permissions there was a "Squash" setting where it was showing options to map the user. I had originally set map "root to admin" (Synology user admin had read/write to the folder). When I altered it to be "map all users to admin" then the unprivileged containers were able to successfully write to the NFS share. "root" of the container was not "root" of the host -- so must have a different UID as far as what the Synology was seeing. Map-all setting seems to "squash" that problem

 

[phpcode]

hier die lösung

https://www.bachmann-lan.de/proxmox-unprivileged-container-backup-failed-permission-denied/

 

[acdoussan]

If anyone else wanders their way in here and the file permissions look fine, and you don't want to use a local temp dir, take a look at this post:

https://blog.doussan.info/posts/container-backup-permission-denied-nfs/

TL;DR - you want to swap your nfs share to map all users, rather than just the root user. In TrueNAS, you can change this on the NFS share under advanced -> access -> set mapall user to your nfs user, remove maproot user (leave it empty).

 

[Johannes Weberhofer]

For Synology products, you must set the following options:
* Squash -> Map all users to admin
* Allow connections from non-privileged ports (ports higher than 1024) -> YES

 

[bullitt168]

For Synology products, you must set the following options:
* Squash -> Map all users to admin
* Allow connections from non-privileged ports (ports higher than 1024) -> YES

This is the solution which worked for me! Thanks a lot!

 

[pax0707]

This tip was helpful, pointed me in the direction!

I have a Synology NFS share mounted - VMs would backup OK, containers did not. Checked and found that all containers were unprivileged. On the Synology, under NFS Permissions there was a "Squash" setting where it was showing options to map the user. I had originally set map "root to admin" (Synology user admin had read/write to the folder). When I altered it to be "map all users to admin" then the unprivileged containers were able to successfully write to the NFS share. "root" of the container was not "root" of the host -- so must have a different UID as far as what the Synology was seeing. Map-all setting seems to "squash" that problem

THIS!