[SOLVED] Unable to create backup group "/mnt/datastore/<datastore>" - Permission denied (os error 13)

[hibouambigu]

Greetings Proxmoxians

I've just deployed a PBS server with a single ZFS datastore (zbackup in the backup log below) and cannot seem to figure out why backup tests are failing with these permission errors.

The PVE cluster is authenticated against PBS with a user eeboo@pbs who has Admin permissions at / and also /datastore/zbackup

proxmox-backup-manager acl list

┌───────────┬────────────────────┬───────────┬────────┐
│ ugid      │ path               │ propagate │ roleid │
╞═══════════╪════════════════════╪═══════════╪════════╡
│ eeboo@pbs │ /                  │         1 │ Admin  │
├───────────┼────────────────────┼───────────┼────────┤
│ eeboo@pbs │ /datastore/zbackup │         1 │ Admin  │
└───────────┴────────────────────┴───────────┴────────┘

ls -alh /mnt/datastore/zbackup/

total 12M
drwxr-xr-x     3 backup backup   4 Dec 20 10:31 .
drwxr-xr-x     3 root   root     3 Dec 18 10:44 ..
drwxr-x--- 65538 backup backup 65K Dec 18 10:44 .chunks
-rw-r--r--     1 backup backup   0 Dec 18 10:44 .lock

Backup job errors seen from running backup on Proxmox VE host

INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --node pxn13 --storage pbs-zbackup
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-12-20 10:19:24
INFO: status = running
INFO: VM Name: pvr.int.eeboo.io
INFO: include disk 'scsi0' 'nfs-vmshare:101/vm-101-disk-0.qcow2' 43212M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/101/2020-12-20T15:19:24Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: enabling encryption
ERROR: VM 101 qmp command 'backup' failed - backup connect failed: command error: unable to create backup group "/mnt/datastore/zbackup/vm/101" - Permission denied (os error 13)
INFO: aborting backup job
ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup connect failed: command error: unable to create backup group "/mnt/datastore/zbackup/vm/101" - Permission denied (os error 13)
INFO: Failed at 2020-12-20 10:19:27
INFO: Backup job finished with errors
TASK ERROR: job errors

Have I missed something here?

Any other informations I can provide to help figure this out?

Thanks!!!

 

[fabian]

can you run mkdir /mnt/datastore/zbackup/{vm,host,ct} on the PBS system logged in as user 'backup' ?

 

[hibouambigu]

can you run mkdir /mnt/datastore/zbackup/{vm,host,ct} on the PBS system logged in as user 'backup' ?

I'm not sure I'm understanding correct. Here's what I tried (from pbs shell):

root@pbs:~# runuser -l backup -c 'mkdir /mnt/datastore/zbackup/{vm,host,ct}'
This account is currently not available.

root@pbs:~# runuser -l backup -c 'mkdir /mnt/datastore/zbackup/test'
This account is currently not available.

So the weird thing is, the scheduled backups actually completed last night, and this error seems to be gone. I can see the folders with the VM backups now on PBS, and verify their integrity.

Nothing about the configuration had been changed by me since this forum post and debugging yesterday.

Do you think it could have been something that didn't propagate to the PVE hosts yet? Very odd. Servers have not been rebooted or anything either.

 

[fabian]

the errors above are probably because the 'backup' user cannot login (has no password). not sure what your original problem was, maybe the dataset was not (yet) mounted properly when PVE attempted the backup?

 

[hibouambigu]

the errors above are probably because the 'backup' user cannot login (has no password). not sure what your original problem was, maybe the dataset was not (yet) mounted properly when PVE attempted the backup?

Thanks fabian. I'll report back if I run into this issue again, but at this point not looking to try and reproduce it. It seemed more like an authentication issue somehow than a mount issue, since I was able to list the volumes on PBS iirc.

Will report back if I discover anything new regarding it. Will mark as solved for now.

 

[Rick van Leeuwen]

Thanks fabian. I'll report back if I run into this issue again, but at this point not looking to try and reproduce it. It seemed more like an authentication issue somehow than a mount issue, since I was able to list the volumes on PBS iirc.

Will report back if I discover anything new regarding it. Will mark as solved for now.

i set a password in the shell and added backup user in GUI with the same pw, that solved it

 

[FelixJ]

Hi everyone, I ran into that error as well, seams to me, as if "Permission denied (os error 13)" is a rather generic error code, as in my case it was the missing .lock file in the root of the backup-repository directory.
I ran into that, when I recovered my PBS repository from tape (I do not use the pbs tape management, as I have a bunch of other files I need to have on the tape aside the vm/ct repository that comes from PBS.
Anyhow, after the tape-restore was complete, and I created a datastore via gui and then changed the path in /etc/proxmox-backup/datastore.cfg to match my restored tape path. For some reason, the .lock file was not on the tape, and that caused this error - so this is just to let everyone now, if one runs into that error!
regards, Felix

 

[fabian]

that sounds like it was missing AND it was not possible to create because of permissions (e.g., missing +w on the datastore root for the backup user?)

 

[FelixJ]

that sounds like it was missing AND it was not possible to create because of permissions (e.g., missing +w on the datastore root for the backup user?)

Hi Fabian,
I am not to sure, if that was a question or a remark, so here is the full version of what I did:
I first created in the GUI the datastore, which was then accessible without any problem.
I restored data from the tape using tar -C /path/to/datastore but tar by default restores the full path, so unter my datastore-path, I had the full path down to the original repository. So I edited the datastore.cfg to match the correct path.
This is, when I got the os 13 error.
I then removed the created .chuncks folder under the previously created datastore-path and replaced it with the restored one, so I did with the vm-folder.
The Problem was persistent.
Then I created an additional datastore to see the differences between both folders. This is when I realized, the .lock file was missing.
The most important thing is now, that it works. The less nice thing about it is, that the error message is not quiet catchy.... Would be nice, to evaluate, prior to try to read the contend of the datastore, if the .lock file is present and, if missing, re-create it!
So far I'm glad, I found the solution. I only wanted to contribute that to this thread, as it seamed to me, the reason for my os error 13 is different then the one from the originator of that thread!

IMHO, that thread can be re-closed and is, once again, solved!
regards,
Felix

 

[fabian]

yeah - the lock file is already re-created if possible - so the error you get is because it was not possible to (re)-create the lockfile, which indicates that the datastore root does not have the correct permissions (i.e., access by root is possible, access by backup is not).