[SOLVED] Chunks permission error affecting only one specific vm backup

[Por12]

Hello,

I've been having an error on the backup of a specific vm. It happens every time I try to backup this specific VM and always at this specific % (12%). The error code seems to refer to a permission error, but I'm at a loss on how to solve it.

root@pbs:/mnt/pbs-backup# ls -ld .chunks/4437
drw-r--r-- 1 backup backup 238 Sep 24 10:01 .chunks/4437

root@pbs:/mnt/pbs-backup# ls -ld .chunks/4437/*
-rw-r--r-- 1 backup backup   94138 Sep 24 10:01 .chunks/4437/4437d38b4d951e4ecc4aa71f9526b84a87837161de1d7ef1bacbd42774f6931c
-rw-r--r-- 1 backup backup 2211149 Sep 18 11:57 .chunks/4437/4437f466352b16b40bd9a5c3e25c83538671afa82d34eabdb60e646d0ced224a
-rw-r--r-- 1 backup backup 4194316 Sep 18 11:59 .chunks/4437/4437faf5b38d38db305ddcca6306cdfd4f71d0ddc85c00f467048649eadeec12

root@pbs:/mnt/pbs-backup# stat .chunks/4437/*
  File: .chunks/4437/4437d38b4d951e4ecc4aa71f9526b84a87837161de1d7ef1bacbd42774f6931c
  Size: 94138           Blocks: 184        IO Block: 4096   regular file
Device: 0,37    Inode: 648799838517122081  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   34/  backup)   Gid: (   34/  backup)
Access: 2023-10-25 00:00:30.447727083 +0200
Modify: 2023-09-24 10:01:21.123644151 +0200
Change: 2023-10-25 07:31:40.467298324 +0200
 Birth: -
  File: .chunks/4437/4437f466352b16b40bd9a5c3e25c83538671afa82d34eabdb60e646d0ced224a
  Size: 2211149         Blocks: 4320       IO Block: 4096   regular file
Device: 0,37    Inode: 648799838516566541  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   34/  backup)   Gid: (   34/  backup)
Access: 2023-10-25 00:02:30.917764674 +0200
Modify: 2023-09-18 11:57:43.413532733 +0200
Change: 2023-10-25 00:02:30.917764674 +0200
 Birth: -
  File: .chunks/4437/4437faf5b38d38db305ddcca6306cdfd4f71d0ddc85c00f467048649eadeec12
  Size: 4194316         Blocks: 8200       IO Block: 4096   regular file
Device: 0,37    Inode: 648799838516566539  Links: 1
Access: (0644/-rw-r--r--)  Uid: (   34/  backup)   Gid: (   34/  backup)
Access: 2023-10-25 00:02:32.803749622 +0200
Modify: 2023-09-18 11:59:58.662762587 +0200
Change: 2023-10-25 00:02:32.803749622 +0200
 Birth: -

INFO: starting new backup job: vzdump 401 --remove 0 --mode snapshot --notes-template '{{guestname}}' --notification-mode auto --storage pbs-unraid-vm --node apollo
INFO: Starting Backup of VM 401 (qemu)
INFO: Backup started at 2024-01-09 12:01:39
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: win11
INFO: include disk 'virtio0' 'local:401/vm-401-disk-2.raw' 128G
INFO: include disk 'efidisk0' 'local:401/vm-401-disk-0.qcow2' 528K
INFO: include disk 'tpmstate0' 'local:401/vm-401-disk-1.raw' 17K
INFO: creating Proxmox Backup Server archive 'vm/401/2024-01-09T11:01:39Z'
INFO: starting kvm to execute backup task
swtpm_setup: Not overwriting existing state file.
INFO: attaching TPM drive to QEMU for backup
INFO: started backup task '0df4517d-9e4a-4541-8de9-d82c703032de'
INFO: efidisk0: dirty-bitmap status: created new
INFO: tpmstate0-backup: dirty-bitmap status: created new
INFO: virtio0: dirty-bitmap status: created new
INFO:   0% (568.0 MiB of 128.0 GiB) in 3s, read: 189.3 MiB/s, write: 165.3 MiB/s
INFO:   1% (1.4 GiB of 128.0 GiB) in 7s, read: 216.0 MiB/s, write: 215.0 MiB/s
INFO:   2% (2.7 GiB of 128.0 GiB) in 13s, read: 216.0 MiB/s, write: 216.0 MiB/s
INFO:   3% (3.9 GiB of 128.0 GiB) in 18s, read: 249.6 MiB/s, write: 248.0 MiB/s
INFO:   4% (5.3 GiB of 128.0 GiB) in 25s, read: 205.1 MiB/s, write: 202.9 MiB/s
INFO:   5% (6.5 GiB of 128.0 GiB) in 31s, read: 210.7 MiB/s, write: 210.7 MiB/s
INFO:   6% (7.7 GiB of 128.0 GiB) in 37s, read: 200.7 MiB/s, write: 198.7 MiB/s
INFO:   7% (9.1 GiB of 128.0 GiB) in 45s, read: 179.0 MiB/s, write: 178.0 MiB/s
INFO:   8% (10.3 GiB of 128.0 GiB) in 50s, read: 254.4 MiB/s, write: 249.6 MiB/s
INFO:   9% (11.6 GiB of 128.0 GiB) in 53s, read: 444.0 MiB/s, write: 438.7 MiB/s
INFO:  10% (12.8 GiB of 128.0 GiB) in 59s, read: 200.7 MiB/s, write: 194.0 MiB/s
INFO:  11% (14.4 GiB of 128.0 GiB) in 1m 8s, read: 182.2 MiB/s, write: 177.8 MiB/s
INFO:  12% (15.6 GiB of 128.0 GiB) in 1m 15s, read: 167.4 MiB/s, write: 165.7 MiB/s
INFO:  12% (15.8 GiB of 128.0 GiB) in 1m 16s, read: 220.0 MiB/s, write: 220.0 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: inserting chunk on store 'pbs-backup' failed for 4437faf5b38d38db305ddcca6306cdfd4f71d0ddc85c00f467048649eadeec12 - mkstemp "/mnt/pbs-backup/.chunks/4437/4437faf5b38d38db305ddcca6306cdfd4f71d0ddc85c00f467048649eadeec12.tmp_XXXXXX" failed: EACCES: Permission denied
INFO: aborting backup job
INFO: stopping kvm after backup task
trying to acquire lock...
 OK
ERROR: Backup of VM 401 failed - backup write data failed: command error: write_data upload error: pipelined request failed: inserting chunk on store 'pbs-backup' failed for 4437faf5b38d38db305ddcca6306cdfd4f71d0ddc85c00f467048649eadeec12 - mkstemp "/mnt/pbs-backup/.chunks/4437/4437faf5b38d38db305ddcca6306cdfd4f71d0ddc85c00f467048649eadeec12.tmp_XXXXXX" failed: EACCES: Permission denied
INFO: Failed at 2024-01-09 12:02:57
INFO: Backup job finished with errors

Thanks in advance!

 

[Por12]

Could it have to be with the 644 permission on the chunks/4437 folder?

 

[Chris]

Hi,
seems like you messed up the access permissions for the chunk folder structure, you will have to set all directories located in the .chunks folder to have 750 permissions, so that the backup user and group can enter the directories.

 

[Por12]

To double check, to do that I should run "chmod 750 -R /mnt/Proxmox_Backup/.chunks", right?

Thanks

 

[Chris]

To double check, to do that I should run "chmod 750 -R /mnt/Proxmox_Backup/.chunks", right?

Thanks

No, you don't want to change the permissions recursively, as this would also change the chunk files. Since the list of folders is large, the best is use find to change the mode via e.g. find /mnt/pbs-backup/.chunks -maxdepth 1 -type d -exec chmod 750 {} \;, assuming the path to your chunk folder is correct (note that your log output and the path you showed in your suggested command do not match).

 

[Por12]

That's amazing. Thanks for pointing out the command, I copied it from another forum post.

All is solved now and PBS is back to working fine.

Thanks!