Getting Permissions issues when running a backup job

[bobloadmire]

I'm getting the following errors on a few of my LXCs when running a bulk backup job. Most of my backup jobs complete fine, but a couple give me this error:

INFO: CT Name: photoprism
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/coopnet') from backup (not a volume)
INFO: creating vzdump archive '/mnt/pve/CoopNet/dump/vzdump-lxc-111-2024_06_15-18_28_46.tar.zst'
INFO: Total bytes written: 2559375360 (2.4GiB, 136MiB/s)
INFO: zstd: /*stdout*\: Permission denied
ERROR: Backup of VM 111 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp7724_111' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd '--threads=1' >/mnt/pve/CoopNet/dump/vzdump-lxc-111-2024_06_15-18_28_46.tar.dat' failed: exit code 1
INFO: Failed at 2024-06-15 18:29:06

I asked copilot, gave me this:
Copilot
The backup process for your LXC container (VM 111) encountered an error related to permissions. Let’s troubleshoot this issue:

1. Permission Denied Error:
   • The error message zstd: /*stdout*: Permission denied indicates that the zstd command failed due to insufficient permissions.
   • The backup process runs under a specific user (UID 100000), and it needs write access to the target directory.
2. Check Permissions:
   • Verify the permissions for the following directories:
     • /mnt/pve/CoopNet/dump/
     • /mnt/vzsnap0
   • Ensure that the user running the backup process has write permissions to these directories.
3. Fix Permissions:
   • Change the permissions on the directories:
     chmod 755 /mnt/pve/CoopNet/dump
     chmod 755 /mnt/vzsnap0
     
     
   • This ensures that the backup process can write to these locations.
4. Retry Backup:
   • After adjusting permissions, try running the backup again.
   • Monitor the process to ensure it completes successfully.

Remember to replace the directory paths with the actual ones used in your setup. If you encounter any issues or need further assistance, feel free to ask!
here's another similar error out:

INFO: starting new backup job: vzdump 110 --remove 0 --mode snapshot --storage CoopNet --notes-template '{{guestname}}' --node pve --notification-mode auto --compress zstd
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp32913_110 for temporary files
INFO: Starting Backup of VM 110 (lxc)
INFO: Backup started at 2024-06-15 18:42:09
INFO: status = running
INFO: CT Name: cloudflared
INFO: including mount point rootfs ('/') in backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: cloudflared
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/1212/root/ to /var/tmp/vzdumptmp32913_110
ERROR: Backup of VM 110 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/1212/root//./ /var/tmp/vzdumptmp32913_110' failed: exit code 23
INFO: Failed at 2024-06-15 18:42:13
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

Why would only a few of my containers have this issue? Also CoopNet folder is a network mount for my CIFS share. This network share appears to be fully functional, I have no issues reading/writing to it before and after the backup attempt. and only 2 of my 12 LXCs are failing backup. Also I thought I just might have a botched install or a bad NVME drive, so I did a reinstall on a brand new NVME drive, same issue.

 

[fschauer]

It seems like the UID 100000 of the unprivileged container does not have write access to the backup storage.

To resolve this, you can grant the UID 100000 write access to the CIFS share.

Alternatively you can specify a tmpdir in /etc/vzdump.conf
https://pve.proxmox.com/wiki/Backup_and_Restore#vzdump_configuration
Or you could run the vzdump command manually through the CLI and specify the --tmpdir <path to temporary directory> argument.
This way the backup will be written to a local temporary directory and then moved to the backup storage.

 

[bobloadmire]

Thank you! This fixed it