LXC backups fail - exitcode 255

[GHMC]

I have the latest version PVE running and PBS as a LXC on that PVE. A mountpoint in the LXC connecting to a PVE dit that is connected to a SMB directory on a NAS. From the PBS LXC I can reach and write that dir. Can also make a Datastore on that and connect it as backup storage in the PVE.

When run a backup of a LXC as test I'm getting the following error:

INFO: starting new backup job: vzdump 122 --notification-mode auto --node proxmox1 --remove 0 --storage pbs --mode stop --notes-template '{{guestname}}'
INFO: Starting Backup of VM 122 (lxc)
INFO: Backup started at 2024-07-25 10:17:47
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: whoogle
INFO: including mount point rootfs ('/') in backup
INFO: stopping virtual guest
INFO: creating Proxmox Backup Server archive 'ct/122/2024-07-25T08:17:47Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp3779201_122/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 122 --backup-time 1721895467 --entries-max 1048576 --repository root@pam@192.168.1.39:pbs-synology
INFO: Starting backup: ct/122/2024-07-25T08:17:47Z
INFO: Client name: proxmox1
INFO: Starting backup protocol: Thu Jul 25 10:17:51 2024
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp3779201_122/etc/vzdump/pct.conf' to 'root@pam@192.168.1.39:8007:pbs-synology' as pct.conf.blob
INFO: Error: fchmod "/mnt/pbs/ct/122/2024-07-25T08:17:47Z/pct.conf.tmp_xzyPwV" failed: EPERM: Operation not permitted
INFO: restarting vm
INFO: guest is online again after 5 seconds
ERROR: Backup of VM 122 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp3779201_122/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 122 --backup-time 1721895467 --entries-max 1048576 --repository root@pam@192.168.1.39:pbs-synology' failed: exit code 255
INFO: Failed at 2024-07-25 10:17:52
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

In PBS the task error is:

2024-07-25T10:17:51+02:00: starting new backup on datastore 'pbs-synology' from ::ffff:192.168.1.30: "ct/122/2024-07-25T08:17:47Z"
2024-07-25T10:17:51+02:00: protocol upgrade done
2024-07-25T10:17:51+02:00: GET /previous_backup_time
2024-07-25T10:17:51+02:00: POST /blob
2024-07-25T10:17:51+02:00: POST /blob: 400 Bad Request: fchmod "/mnt/pbs/ct/122/2024-07-25T08:17:47Z/pct.conf.tmp_xzyPwV" failed: EPERM: Operation not permitted
2024-07-25T10:17:51+02:00: backup failed: connection error: not connected
2024-07-25T10:17:51+02:00: removing failed backup
2024-07-25T10:17:51+02:00: TASK ERROR: connection error: not connected

I am not clear how to proceed.

 

[Chris]

INFO: Error: fchmod "/mnt/pbs/ct/122/2024-07-25T08:17:47Z/pct.conf.tmp_xzyPwV" failed: EPERM: Operation not permitted

Seems like the network share you setup as datastore does not allow to set/alter the file permissions. Please check the permissions and ACLs on the host providing the share.

 

[GHMC]

It is a Synology NAS and that's always trouble with permissions. So for testing I set the dir on that with all read and write permissions for everyone.

After that I get the following error:

INFO: starting new backup job: vzdump 121 --node proxmox1 --notification-mode auto --remove 0 --storage pbs --mode stop --notes-template '{{guestname}}'
INFO: Starting Backup of VM 121 (lxc)
INFO: Backup started at 2024-07-25 14:09:50
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: runtipi
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/homes') from backup (not a volume)
INFO: excluding bind mount point mp1 ('/mnt/downloads') from backup (not a volume)
INFO: excluding bind mount point mp2 ('/mnt/proxmox') from backup (not a volume)
INFO: excluding bind mount point mp3 ('/mnt/code') from backup (not a volume)
INFO: excluding bind mount point mp4 ('/mnt/calibre') from backup (not a volume)
INFO: stopping virtual guest
INFO: creating Proxmox Backup Server archive 'ct/121/2024-07-25T12:09:50Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp32789_121/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 121 --backup-time 1721909390 --entries-max 1048576 --repository root@pam@192.168.1.39:pbs-synology
INFO: Starting backup: ct/121/2024-07-25T12:09:50Z
INFO: Client name: proxmox1
INFO: Starting backup protocol: Thu Jul 25 14:10:03 2024
INFO: Error: Permission denied (os error 13)
INFO: restarting vm
INFO: guest is online again after 14 seconds
ERROR: Backup of VM 121 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp32789_121/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 121 --backup-time 1721909390 --entries-max 1048576 --repository root@pam@192.168.1.39:pbs-synology' failed: exit code 255
INFO: Failed at 2024-07-25 14:10:04
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

The mount in fstab on the PVE is:

//synology10g.calmus.one/PBS/ /mnt/pbs cifs _netdev,x-systemd.automount,noatime,uid=100000,gid=110000,dir_mode=0777,file_mode=0777,user=REDACTED,pass=REDACTED 0 0

I guess that should have worked? I can make backups from the PVE without PBS to similar folders on that NAS with the same fstab mount. So whats different with PBS?

 

[GHMC]

Also tried without the uid, gid, file_mode and dir_mode setting, but that does not change the error.

Deleted and created a new datastore with these permissions also did not help.

 

[Chris]

//synology10g.calmus.one/PBS/ /mnt/pbs cifs _netdev,x-systemd.automount,noatime,uid=100000,gid=110000,dir_mode=0777,file_mode=0777,user=REDACTED,pass=REDACTED 0 0

Note that the contents of the datastore backing patch needs to be owned by the backup user and group, which is uid/gid 34. So that user must be allowed to access the share. First you have to set that correctly, the uid and gid are not mapped like for lxc backups.

Further check eventual ACL settings for the user on the Synology.

Maybe this post might help https://forum.proxmox.com/threads/permssion-denied-os-error-13-on-nfs-share.74805/post-507340

 

[GHMC]

Thanks for your help. Tried the suggestions in that thread, but they all did not work for me. Taking to much time, so I'm kicking PBS out and keep backing up from PVE.