VM Backup ERROR: job failed with err -13 - Permission denied

[linhu]

I have a proxmox system with one backupserver, for all backups the option fleecing is enabled the fleecing storrage is a local zpool1 on all vm servers.
So this vm is failing but others are done.
The zpool1 has 1Tb space and there is free space.

INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2024-05-09 02:00:01
INFO: status = running
INFO: VM Name: ns1
INFO: include disk 'scsi0' 'nfs-truenas:100/vm-100-disk-0.qcow2' 16G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2024-05-09T00:00:01Z'
INFO: drive-scsi0: attaching fleecing image zpool1:vm-100-fleece-0 to QEMU
INFO: started backup task '845cc857-e11e-466e-bed4-aee0a5ff7059'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO: 0% (32.0 MiB of 16.0 GiB) in 3s, read: 10.7 MiB/s, write: 1.3 MiB/s
INFO: 1% (164.0 MiB of 16.0 GiB) in 54s, read: 2.6 MiB/s, write: 80.3 KiB/s
INFO: 2% (328.0 MiB of 16.0 GiB) in 3m 15s, read: 1.2 MiB/s, write: 0 B/s
INFO: 3% (496.0 MiB of 16.0 GiB) in 4m 24s, read: 2.4 MiB/s, write: 0 B/s
INFO: 4% (716.0 MiB of 16.0 GiB) in 5m 30s, read: 3.3 MiB/s, write: 0 B/s
INFO: 5% (840.0 MiB of 16.0 GiB) in 5m 39s, read: 13.8 MiB/s, write: 455.1 KiB/s
INFO: 6% (1000.0 MiB of 16.0 GiB) in 5m 50s, read: 14.5 MiB/s, write: 0 B/s
INFO: 7% (1.1 GiB of 16.0 GiB) in 6m 7s, read: 9.9 MiB/s, write: 0 B/s
INFO: 8% (1.3 GiB of 16.0 GiB) in 6m 46s, read: 5.1 MiB/s, write: 0 B/s
INFO: 9% (1.4 GiB of 16.0 GiB) in 6m 49s, read: 37.3 MiB/s, write: 0 B/s
INFO: 10% (1.7 GiB of 16.0 GiB) in 7m 13s, read: 10.5 MiB/s, write: 0 B/s
INFO: 11% (1.8 GiB of 16.0 GiB) in 7m 16s, read: 32.0 MiB/s, write: 0 B/s
INFO: 12% (1.9 GiB of 16.0 GiB) in 7m 25s, read: 18.7 MiB/s, write: 0 B/s
INFO: 13% (2.1 GiB of 16.0 GiB) in 8m 44s, read: 1.8 MiB/s, write: 0 B/s
INFO: 14% (2.3 GiB of 16.0 GiB) in 9m 1s, read: 12.2 MiB/s, write: 0 B/s
INFO: 15% (2.4 GiB of 16.0 GiB) in 9m 13s, read: 11.3 MiB/s, write: 0 B/s
INFO: 16% (2.7 GiB of 16.0 GiB) in 9m 22s, read: 28.9 MiB/s, write: 0 B/s
INFO: 17% (2.8 GiB of 16.0 GiB) in 9m 34s, read: 6.7 MiB/s, write: 0 B/s
INFO: 18% (2.9 GiB of 16.0 GiB) in 9m 44s, read: 13.6 MiB/s, write: 0 B/s
INFO: 19% (3.1 GiB of 16.0 GiB) in 10m 15s, read: 5.9 MiB/s, write: 0 B/s
INFO: 20% (3.2 GiB of 16.0 GiB) in 10m 56s, read: 3.4 MiB/s, write: 0 B/s
INFO: 21% (3.4 GiB of 16.0 GiB) in 11m 9s, read: 16.3 MiB/s, write: 0 B/s
INFO: 22% (3.6 GiB of 16.0 GiB) in 11m 12s, read: 48.0 MiB/s, write: 0 B/s
INFO: 23% (3.7 GiB of 16.0 GiB) in 11m 16s, read: 35.0 MiB/s, write: 0 B/s
INFO: 24% (3.9 GiB of 16.0 GiB) in 11m 30s, read: 14.6 MiB/s, write: 0 B/s
INFO: 25% (4.0 GiB of 16.0 GiB) in 11m 51s, read: 5.9 MiB/s, write: 0 B/s
INFO: 26% (4.2 GiB of 16.0 GiB) in 13m 10s, read: 2.5 MiB/s, write: 155.5 KiB/s
INFO: 27% (4.3 GiB of 16.0 GiB) in 13m 28s, read: 8.4 MiB/s, write: 0 B/s
INFO: 28% (4.5 GiB of 16.0 GiB) in 13m 46s, read: 8.7 MiB/s, write: 0 B/s
INFO: 29% (4.7 GiB of 16.0 GiB) in 14m 1s, read: 14.1 MiB/s, write: 0 B/s
INFO: 30% (4.8 GiB of 16.0 GiB) in 14m 12s, read: 12.0 MiB/s, write: 0 B/s
INFO: 31% (5.0 GiB of 16.0 GiB) in 14m 19s, read: 21.1 MiB/s, write: 0 B/s
INFO: 32% (5.2 GiB of 16.0 GiB) in 14m 31s, read: 14.7 MiB/s, write: 0 B/s
INFO: 33% (5.3 GiB of 16.0 GiB) in 14m 51s, read: 7.0 MiB/s, write: 0 B/s
INFO: 34% (5.5 GiB of 16.0 GiB) in 15m 20s, read: 7.0 MiB/s, write: 0 B/s
INFO: 35% (5.7 GiB of 16.0 GiB) in 15m 23s, read: 69.3 MiB/s, write: 0 B/s
INFO: 36% (5.9 GiB of 16.0 GiB) in 15m 26s, read: 73.3 MiB/s, write: 0 B/s
INFO: 37% (5.9 GiB of 16.0 GiB) in 15m 30s, read: 10.0 MiB/s, write: 0 B/s
INFO: 38% (6.1 GiB of 16.0 GiB) in 16m 46s, read: 2.5 MiB/s, write: 269.5 KiB/s
INFO: 39% (6.3 GiB of 16.0 GiB) in 17m 7s, read: 6.7 MiB/s, write: 3.0 MiB/s
INFO: 40% (6.4 GiB of 16.0 GiB) in 17m 37s, read: 5.1 MiB/s, write: 0 B/s
INFO: 41% (6.6 GiB of 16.0 GiB) in 17m 41s, read: 42.0 MiB/s, write: 0 B/s
INFO: 42% (6.7 GiB of 16.0 GiB) in 17m 45s, read: 36.0 MiB/s, write: 0 B/s
INFO: 43% (6.9 GiB of 16.0 GiB) in 17m 53s, read: 28.0 MiB/s, write: 0 B/s
INFO: 44% (7.1 GiB of 16.0 GiB) in 17m 56s, read: 45.3 MiB/s, write: 0 B/s
INFO: 45% (7.3 GiB of 16.0 GiB) in 18m 3s, read: 26.3 MiB/s, write: 0 B/s
INFO: 46% (7.4 GiB of 16.0 GiB) in 19m 45s, read: 1.3 MiB/s, write: 0 B/s
ERROR: job failed with err -13 - Permission denied
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - job failed with err -13 - Permission denied
INFO: Failed at 2024-05-09 02:19:59

 

[fiona]

Hi,
please make sure you are using a version of pve-qemu-kvm >= 8.1.5-6 which contains: https://git.proxmox.com/?p=pve-qemu.git;a=commit;h=c93a5bfd4f29642e060d796e9b8a4d5117052cd5

VMs need to be Rebooted via UI (inside the guest is not enough) or migrated to a node that already has the update to pick up the new version.

 

[linhu]

apt search pve-qemu-kvm
Sorting... Done
Full Text Search... Done
pve-qemu-kvm/stable,now 8.1.5-6 amd64 [installed]
Full virtualization on x86 hardware

and pveversion -v says:
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1

all servers are rebootet, but i will check this again.
is a bulk reboot ok?

 

[fiona]

The other possibility is that there was an IO error. In QEMU 9.0 (not released for Proxmox VE yet), the error message will be improved to show the actual root cause.

AFAIK, there is no bulk reboot functionality right now. You could combine bulk shutdown followed by a bulk start of course.