Can't create backup | CIFS

[R00bin]

Hello. I am trying to create backups with the Proxmox Backup Server.
For this I have mount an external storage (CIFS) in my system.

My source directory is: mnt/storage-01

The monut is carried out with the following command in /ect/fstab

//<host>/mnt/storage-01       cifs    iocharset=utf8,rw,credentials=/etc/backup-credentials.txt,file_mode=0777,dir_mode=0777 0  0

The storage is also displayed corrently in the web interface of the backup server.
An integration into the Proxmox server itself also seems to have worked without problems.

However, when I now try to create a backup, i get the following errors.

Proxmox (Server) Console:

NFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/1655/root/ to /var/tmp/vzdumptmp2198260_101
INFO: first sync finished - transferred 681.18M bytes in 2s
INFO: suspending guest
INFO: starting final sync /proc/1655/root/ to /var/tmp/vzdumptmp2198260_101
INFO: final sync finished - transferred 0 bytes in 0s
INFO: resuming guest
INFO: guest is online again after <1 seconds
INFO: creating Proxmox Backup Server archive 'ct/101/2022-10-13T12:38:00Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp2198260_101/etc/vzdump/pct.conf fw.conf:/var/tmp/vzdumptmp2198260_101/etc/vzdump/pct.fw root.pxar:/var/tmp/vzdumptmp2198260_101 --include-dev /var/tmp/vzdumptmp2198260_101/. --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 101 --backup-time 1665664680 --repository root@pam@host:storage-01
INFO: Starting backup: ct/101/2022-10-13T12:38:00Z
INFO: Client name: prox
INFO: Starting backup protocol: Thu Oct 13 14:38:02 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp2198260_101/etc/vzdump/pct.conf' to 'root@pam@host:8007:storage-01' as pct.conf.blob
INFO: Error: fchmod "/mnt/storage-01/backup/ct/101/2022-10-13T12:38:00Z/pct.conf.tmp_MB6GVy" failed: EPERM: Operation not permitted
ERROR: Backup of VM 101 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp2198260_101/etc/vzdump/pct.conf fw.conf:/var/tmp/vzdumptmp2198260_101/etc/vzdump/pct.fw root.pxar:/var/tmp/vzdumptmp2198260_101 --include-dev /var/tmp/vzdumptmp2198260_101/. --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 101 --backup-time 1665664680 --repository root@pam@host:storage-01' failed: exit code 255
INFO: Failed at 2022-10-13 14:38:03
INFO: Backup job finished with errors
TASK ERROR: job errors

Proxmox backup server console output:

()
2022-10-13T14:38:02+02:00: starting new backup on datastore 'storage-01': "ct/101/2022-10-13T12:38:00Z"
2022-10-13T14:38:02+02:00: protocol upgrade done
2022-10-13T14:38:02+02:00: GET /previous_backup_time
2022-10-13T14:38:02+02:00: POST /blob
2022-10-13T14:38:02+02:00: POST /blob: 400 Bad Request: fchmod "/mnt/storage-01/backup/ct/101/2022-10-13T12:38:00Z/pct.conf.tmp_MB6GVy" failed: EPERM: Operation not permitted
2022-10-13T14:38:02+02:00: backup ended and finish failed: backup ended but finished flag is not set.
2022-10-13T14:38:02+02:00: removing unfinished backup
2022-10-13T14:38:02+02:00: TASK ERROR: backup ended but finished flag is not set.

Does anyone have any idea what this could be? I would be very grateful for any help.

FYI: I have intentionally renamed te hostname in the logs.

Versions:
Proxmox Virtual Environment 7.2-11
Proxmox Backup Server 2.2-7

 

[R00bin]

What is also worth mentioning is that folders up to /ct/101/ alone are created.
e.g. 2022-10-13T12:38:00Z not.

But the "owner" files exist with content:
root@pam

 

[Dunuin]

With unprivileged LXCs you often run into permission problems when backing up, because the LXCs root user is UID 100000 and that user needs write rights to the temporary folder and this is usually also your SMB share in case you didn‘t changed "tmpdir" in the vzdump.conf. But this shouldn't be a problem here, as you mounted that share with 777 rights.

Edit:
Ignore the above. Thats just the case for vzdump, not PBS.

 

[R00bin]

With unprivileged LXCs you often run into permission problems when backing up, because the LXCs root user is UID 100000 and that user needs write rights to the temporary folder and this is usually also your SMB share in case you didn‘t changed "tmpdir" in the vzdump.conf. But this shouldn't be a problem here, as you mounted that share with 777 rights.

Yes, the mount is with 777 rights (file_mode=0777,dir_mode=0777) - so i tink at.
The error also occurs with VMs

Outputs from ls -la /mnt/storage-01/backup/

drwxrwxrwx 2 root root 0 Oct 13 14:37 . drwxrwxrwx 2 root root 0 Oct 13 13:40 .. drwxrwxrwx 2 root root 0 Oct 13 14:08 .chunks drwxrwxrwx 2 root root 0 Oct 13 14:37 ct -rwxrwxrwx 1 root root 0 Oct 13 13:55 .lock

Is there perhaps the problem here?

 

[R00bin]

I have found this in the syslogs, maybe this is an explanation for the problem?

Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: starting new backup on datastore 'storage-01': "ct/101/2022-10-13T13:43:02Z"
Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: protocol upgrade done
Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: GET /previous_backup_time
Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: POST /blob
Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: POST /blob: 400 Bad Request: fchmod "/mnt/storage-01/backup/ct/101/2022-10-13T13:43:02Z/pct.conf.tmp_vzqLjz" failed: EPERM: Operation not permitted
Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: backup ended and finish failed: backup ended but finished flag is not set.
Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: removing unfinished backup
Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: removing backup snapshot "/mnt/storage-01/backup/ct/101/2022-10-13T13:43:02Z"
Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: TASK ERROR: backup ended but finished flag is not set.

 

[Dunuin]

Do you maybe got any ACLs set that prevent writing to it?

 

[R00bin]

Do you maybe got any ACLs set that prevent writing to it?

Not that i know of :c

 

[mmhx]

"/mnt/storage-01/backup/ct/101/2022-10-13T13:43:02Z/pct.conf.tmp_vzqLjz"
Not sure about unix, but under windows a path including ':' ist probably not allowed.
Edit: seems to not cause any issues in linux...

 

[R00bin]

"/mnt/storage-01/backup/ct/101/2022-10-13T13:43:02Z/pct.conf.tmp_vzqLjz"
Not sure about unix, but under windows a path including ':' ist probably not allowed.
Edit: seems to not cause any issues in linux...

This may be where the problem lies. Unfortunately I don't know if you can change the name of the backup server archive or how to format it.

 

[Dunuin]

Didn't see anything in the documentation, so I think this should be hardcoded. Keep in mind that neither Windows nor network shares in genral are officially supported.
You might want to create a feature request for it, if this is really the problem: https://bugzilla.proxmox.com/enter_bug.cgi

 

[fiona]

Hi,
did you actually verify that you can't create a directory with a colon in it? Does the backup user (UID 34) have permissions to write on the mounted share?

 

[R00bin]

Hi,
did you actually verify that you can't create a directory with a colon in it? Does the backup user (UID 34) have permissions to write on the mounted share?

With e.g. mkdir 2022-10-13T13:43:02Z I can create the folder without problems. If the user would have no rights, he could not create the folder "ct", "101" and the file "owner", or am I wrong?

 

[fiona]

Oct 13 15:43:04 pbs proxmox-backup-proxy[547]: POST /blob: 400 Bad Request: fchmod "/mnt/storage-01/backup/ct/101/2022-10-13T13:43:02Z/pct.conf.tmp_vzqLjz" failed: EPERM: Operation not permitted[/CODE]

It's the fchmod call that fails. Can you (as the backup user) try to create a file and run chmod 644 on it?

 

[R00bin]

It's the fchmod call that fails. Can you (as the backup user) try to create a file and run chmod 644 on it?

I manage to create a new folder with the backup user, but as soon as I try to execute the chmod command on the folder with this user, I get this error: chmod: changing permissions of 'test/': Operation not permitted