Proxmox taking Backup on PBS and facing the issue

[arslan]

Hi, community,

I have proxmox server 6.3-2 and running some VMs and containers on it. For the backups, I am using the PBS server (version 2.4.3-1 amd64). I have configured the backup on a daily basis. Everything was working fine a month back but now, some of the VMs are successfully backed up on the PBS server but some VMs failed the backup with the following error.
______________________________________________________________________________________________
INFO: Starting Backup of VM 701 (qemu)
INFO: Backup started at 2023-07-27 03:00:18
INFO: status = running
INFO: VM Name: st.vteamslabs.com
INFO: include disk 'scsi0' 'Data:701/vm-701-disk-0.qcow2' 50G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/701/2023-07-26T22:00:18Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '8e350fb1-c42a-4139-90b0-1259e8ca3122'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (328.0 MiB of 50.0 GiB) in 3s, read: 109.3 MiB/s, write: 70.7 MiB/s
INFO: 1% (532.0 MiB of 50.0 GiB) in 6s, read: 68.0 MiB/s, write: 66.7 MiB/s
INFO: 2% (1.0 GiB of 50.0 GiB) in 14s, read: 63.5 MiB/s, write: 61.0 MiB/s
INFO: 3% (1.5 GiB of 50.0 GiB) in 23s, read: 57.8 MiB/s, write: 56.4 MiB/s
INFO: 4% (2.1 GiB of 50.0 GiB) in 31s, read: 68.5 MiB/s, write: 68.5 MiB/s
INFO: 5% (2.6 GiB of 50.0 GiB) in 39s, read: 63.5 MiB/s, write: 63.5 MiB/s
INFO: 6% (3.0 GiB of 50.0 GiB) in 47s, read: 60.5 MiB/s, write: 60.5 MiB/s
INFO: 7% (3.7 GiB of 50.0 GiB) in 56s, read: 71.1 MiB/s, write: 67.1 MiB/s
INFO: 8% (4.0 GiB of 50.0 GiB) in 1m 2s, read: 62.0 MiB/s, write: 61.3 MiB/s
INFO: 9% (4.6 GiB of 50.0 GiB) in 1m 11s, read: 61.8 MiB/s, write: 61.8 MiB/s
INFO: 10% (5.0 GiB of 50.0 GiB) in 1m 19s, read: 59.0 MiB/s, write: 59.0 MiB/s
INFO: 11% (5.5 GiB of 50.0 GiB) in 1m 28s, read: 57.8 MiB/s, write: 55.6 MiB/s
INFO: 12% (6.0 GiB of 50.0 GiB) in 1m 37s, read: 58.2 MiB/s, write: 58.2 MiB/s
INFO: 13% (6.5 GiB of 50.0 GiB) in 1m 46s, read: 56.9 MiB/s, write: 56.9 MiB/s
INFO: 14% (7.1 GiB of 50.0 GiB) in 1m 56s, read: 61.6 MiB/s, write: 61.6 MiB/s
INFO: 15% (7.5 GiB of 50.0 GiB) in 2m 2s, read: 64.0 MiB/s, write: 62.0 MiB/s
INFO: 16% (8.0 GiB of 50.0 GiB) in 2m 10s, read: 67.0 MiB/s, write: 66.5 MiB/s
INFO: 17% (8.5 GiB of 50.0 GiB) in 2m 19s, read: 52.9 MiB/s, write: 52.9 MiB/s
INFO: 17% (8.5 GiB of 50.0 GiB) in 2m 20s, read: 4.0 MiB/s, write: 4.0 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: inserting chunk on store 'pbs-28.82.122' failed for ffe763e638e1c9e9bd2dc9d14a7864a5834a7dda6745b03c5fc09128015f746d - mkstemp "/mnt/datastore/pbs-28.82.122/.chunks/ffe7/ffe763e638e1c9e9bd2dc9d14a7864a5834a7dda6745b03c5fc09128015f746d.tmp_XXXXXX" failed: ENOENT: No such file or directory
INFO: aborting backup job
ERROR: Backup of VM 701 failed - backup write data failed: command error: write_data upload error: pipelined request failed: inserting chunk on store 'pbs-28.82.122' failed for ffe763e638e1c9e9bd2dc9d14a7864a5834a7dda6745b03c5fc09128015f746d - mkstemp "/mnt/datastore/pbs-28.82.122/.chunks/ffe7/ffe763e638e1c9e9bd2dc9d14a7864a5834a7dda6745b03c5fc09128015f746d.tmp_XXXXXX" failed: ENOENT: No such file or directory
INFO: Failed at 2023-07-27 03:02:40
_________________________________________________________________________________________________________________________________________________

Okay, so I identified two things that describe the state when the backup is successfully taken and when the backup will face some problems.

First Case:

INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared

INFO: 0% (328.0 MiB of 50.0 GiB) in 3s, read: 109.3 MiB/s, write: 70.7 MiB/s
INFO: 1% (532.0 MiB of 50.0 GiB) in 6s, read: 68.0 MiB/s, write: 66.7 MiB/s

when I see the status above in red lines, the backup is going to be failed in any case.


Second Case:

INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (508.0 MiB of 20.0 GiB dirty)

INFO: using fast incremental mode (dirty-bitmap), 508.0 MiB dirty of 20.0 GiB total
INFO: 46% (236.0 MiB of 508.0 MiB) in 3s, read: 78.7 MiB/s, write: 73.3 MiB/s
INFO: 77% (392.0 MiB of 508.0 MiB) in 6s, read: 52.0 MiB/s, write: 30.7 MiB/s
INFO: 100% (508.0 MiB of 508.0 MiB) in 10s, read: 29.0 MiB/s, write: 10.0 MiB/s
INFO: Waiting for server to finish backup validation...
INFO: backup was done incrementally, reused 19.66 GiB (98%)
INFO: transferred 508.00 MiB in 12 seconds (42.3 MiB/s)
INFO: running 'proxmox-backup-client prune' for 'vm/700'
INFO: Finished Backup of VM 700 (00:00:15)
INFO: Backup finished at 2023-07-27 03:00:18
Result: {
"data": null
}


When I see the status above in the red lines, The backup gonna be successful. This is what I have identified.


Please let me know what is this issue and how can i resolve the issue of not successfully taking the backup.

 

[ggoller]

Hi!
please have a look at this thread (https://forum.proxmox.com/threads/g...iled-enoent-no-such-file-or-directory.125842/) and check if the `.chunks/ffe7/` directory exists and which permissions it has. I suspect either `.chunks/ffe7/` doesn't exist or it isn't owned by `backup:backup`.
PS: this is the national support, so german only. The english part of the forum is here: https://forum.proxmox.com/forums/proxmox-backup-installation-and-configuration.24/