permission denied ... mid-backup. huh?

[tcabernoch]

This machine won't backup. No other machines have this problem.
I've moved it to a new host. Same issue.

The disks are just in the regular local-zfs mount, like all the other VMs. Nothing weird.
Its not encrypted. Not running qemu-guest-agent.

The backup does use fleecing. It's writing to a PBS. (The PBS also works fine for other VMs.)

So look at this. It's reading a disk. Midway through the disk, it says permission denied.
If this made any sense at all, it would have a permission error upon first trying to open the disk, not midway.

INFO: 36% (196.2 GiB of 544.0 GiB) in 12m 20s, read: 270.7 MiB/s, write: 8.8 MiB/s
INFO: 37% (201.5 GiB of 544.0 GiB) in 12m 37s, read: 317.9 MiB/s, write: 0 B/s
INFO: 37% (203.7 GiB of 544.0 GiB) in 12m 48s, read: 203.3 MiB/s, write: 0 B/s
ERROR: job failed with err -13 - Permission denied
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 203 failed - job failed with err -13 - Permission denied
INFO: Failed at 2024-08-24 03:23:58

Of course, then it leaves behind a fleecing file, which kills subsequent backups.
But when I nuke the fleecing file and try again, same error. And then another fleecing file to purge.

I'm willing to try any reasonable suggestion. Considering cloning it. Casting about for other options.
(Hmm. I'm gonna copy the disks to the NAS and back. Probly won't help, but its something.)
It's a log server. If I can't fix it, I'll tell the stakeholder to redeploy and don't do anything weird this time.

 

[tcabernoch]

So, nobody else out there is getting this?

ERROR: job failed with err -13 - Permission denied

It's only one VM, and I've tried all the really obvious things to fix it.
This post did not help.
https://forum.proxmox.com/threads/vm-backup-error-job-failed-with-err-13-permission-denied.146695/

Error has occurred on hosts with qemu 8.1.5-6.

It's mysterious. Just that. Shouldn't be acting that way. Doesn't make sense.
This is one of those questions where I hope to see some staff feedback.

 

[tcabernoch]

This error is not occurring when I use a NAS target.
That's not really an answer. More of a data point.
I don't have a next move. It affects 2 VMs that I know of now.
One of them, I can rebuild.
The other one happens to be my jumpbox and workstation in that colo.
I'd like to fix this and get PBS backups working again.

 

[wbk]

I remember having a permissions error once in the middle of a backup process, where, I think, there was a mismatch between the owner of the backup job and the owner of the storage media. It's a different situation than yours, but it might give you inspiration in one direction or another.

Good luck!

 

[tcabernoch]

Thanks.
After a successful backup to the NAS, one of the afflicted VMs is backing up with PBS correctly now ... but the backup only took seconds.
I'm not sure if it will have problems going forward. Gonna have to wait a day and see.

I'm going to try backing the other one up to the NAS, and see if it clears the error condition on it too.

Also, it occurred to me, if I have a good backup on the NAS, I could just restore the machine from that backup.
That will rewrite every bit on the box. Might help.
I'll make sure to do the dance and shake the chicken if/when I perform this voodoo.

Edit: So I ran full backups on both against the NAS. Then I switched em back to PBS backups. They appear to be fixed.
And that tells me ... nothing. Nothing at all. I don't know if I fixed anything. Don't know what the root cause was.

 

[tcabernoch]

I have a new case of this -13 backup error.
It is on a PVE host that did this before. I've patched the host, but not rebooted yet.

INFO: Starting Backup of VM 234 (qemu)
INFO: Backup started at 2024-08-31 23:19:58
INFO: status = running
INFO: VM Name:
INFO: include disk 'scsi0' 'local-zfs:vm-234-disk-0' 140G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/234/2024-09-01T06:19:58Z'
INFO: drive-scsi0: attaching fleecing image local-zfs:vm-234-fleece-0 to QEMU
INFO: started backup task 'a86ef604-46d0-4155-a5b9-824a8cb653d3'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (732.0 MiB of 140.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 732.0 MiB dirty of 140.0 GiB total
INFO: 12% (92.0 MiB of 732.0 MiB) in 3s, read: 30.7 MiB/s, write: 30.7 MiB/s
INFO: 19% (144.0 MiB of 732.0 MiB) in 6s, read: 17.3 MiB/s, write: 17.3 MiB/s
INFO: 22% (168.0 MiB of 732.0 MiB) in 7s, read: 24.0 MiB/s, write: 24.0 MiB/s
ERROR: job failed with err -13 - Permission denied
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 234 failed - job failed with err -13 - Permission denied
INFO: Failed at 2024-08-31 23:20:10
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors


I moved it to a new box, and no error. No backup even. Pushes a couple hundred mb of delta and finishes.
It thinks it has a complete backup though.

I would like to know _where_ the permission is being denied.
Is it being denied read?
Is it being denied write access to the backup archive?
Is it trying to update the dirty-bitmap and failing?
I've recently added site to site Sync Jobs, could that cause it?
Or ... This _fleecing_ thing is a brand-spankin-new feature. Maybe its trying to write to the fleecing file and failing.
@fiona am I running into a bug?

What's going on there? The answer would probably point me at a specific failing that I could then rectify.
Thanks for any insight.

 

[fiona]

Hi,

I would like to know _where_ the permission is being denied.
Is it being denied read?
Is it being denied write access to the backup archive?
Is it trying to update the dirty-bitmap and failing?
I've recently added site to site Sync Jobs, could that cause it?
Or ... This _fleecing_ thing is a brand-spankin-new feature. Maybe its trying to write to the fleecing file and failing.
@fiona am I running into a bug?

yes, it's likely that it's hitting the timeout of 45 seconds for writing to the fleecing image. Newer QEMU versions 9.x have an improved error message here, because the "permission denied" message was a red herring: https://git.proxmox.com/?p=pve-qemu.git;a=commit;h=db293008eeaf65057481eb15d73602232efd9df0

 

[tcabernoch]

Ok, thank you. I've patched.
This error is more transient than I thought. If it didn't leave behind a busted fleecing file that prevents further backups, it would be of less concern.
I have the feeling I'm going to keep chasing this for a while.