LXC container backup ends with error

[vdiogo]

I have a backup for all my containers running against an external s3fs bucket (hosted in iDrive).

I also have this script running that, once the backup finishes, it deletes the backup files from the local disk

I recently created a new container where I mapped two drives which I use to put the files (images and videos) processed by the application (Frigate NVR).

Since then my backup fails for this new container only.

See bellow logs for a working backup:

104: 2024-04-07 01:06:12 INFO: Starting Backup of VM 104 (lxc)
104: 2024-04-07 01:06:12 INFO: status = running
104: 2024-04-07 01:06:12 INFO: CT Name: uptimekuma
104: 2024-04-07 01:06:12 INFO: including mount point rootfs ('/') in backup
104: 2024-04-07 01:06:12 INFO: backup mode: snapshot
104: 2024-04-07 01:06:12 INFO: ionice priority: 7
104: 2024-04-07 01:06:12 INFO: create storage snapshot 'vzdump'
104: 2024-04-07 01:06:13 INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-104-2024_04_07-01_06_12.tar.zst'
104: 2024-04-07 01:07:04 INFO: Total bytes written: 1924833280 (1.8GiB, 36MiB/s)
104: 2024-04-07 01:07:04 INFO: archive file size: 521MB
104: 2024-04-07 01:07:04 INFO: adding notes to backup
104: 2024-04-07 01:07:04 INFO: prune older backups with retention: keep-last=1, keep-monthly=2, keep-weekly=1
104: 2024-04-07 01:07:04 INFO: pruned 0 backup(s)
104: 2024-04-07 01:07:04 INFO: cleanup temporary 'vzdump' snapshot
104: 2024-04-07 01:07:04 INFO: Finished Backup of VM 104 (00:00:52)

And the log for the failed backup:

105: 2024-04-07 01:07:04 INFO: Starting Backup of VM 105 (lxc)
105: 2024-04-07 01:07:04 INFO: status = running
105: 2024-04-07 01:07:04 INFO: CT Name: frigate
105: 2024-04-07 01:07:04 INFO: including mount point rootfs ('/') in backup
105: 2024-04-07 01:07:04 INFO: including mount point mp0 ('/mnt/frigatedata') in backup
105: 2024-04-07 01:07:04 INFO: including mount point mp1 ('/mnt/frigatedatacctv') in backup
105: 2024-04-07 01:07:04 INFO: mode failure - some volumes do not support snapshots
105: 2024-04-07 01:07:04 INFO: trying 'suspend' mode instead
105: 2024-04-07 01:07:04 INFO: backup mode: suspend
105: 2024-04-07 01:07:04 INFO: ionice priority: 7
105: 2024-04-07 01:07:04 INFO: CT Name: frigate
105: 2024-04-07 01:07:04 INFO: including mount point rootfs ('/') in backup
105: 2024-04-07 01:07:04 INFO: including mount point mp0 ('/mnt/frigatedata') in backup
105: 2024-04-07 01:07:04 INFO: including mount point mp1 ('/mnt/frigatedatacctv') in backup
105: 2024-04-07 01:07:04 INFO: starting first sync /proc/16462/root/ to /var/lib/vz/dump/vzdump-lxc-105-2024_04_07-01_07_04.tmp
105: 2024-04-07 01:24:41 ERROR: Backup of VM 105 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/16462/root//./ /proc/16462/root//./mnt/frigatedata /proc/16462/root//./mnt/frigatedatacctv /var/lib/vz/dump/vzdump-lxc-105-2024_04_07-01_07_04.tmp' failed: exit code 11

When asking Gemini for help I get as potential causes the following, which I can't for the sake of me troubleshoot:
Potential Causes:

1. Permission Issues:
   • The backup process might lack sufficient permissions to read from the source directories or write to the destination directory.
   • Check and adjust permissions if necessary using tools like chmod or chown.
2. File System Issues:
   • The source or destination filesystem might have errors or inconsistencies that are preventing the transfer.
   • Use tools like fsck to check and repair any filesystem problems.
3. Network Issues:
   • If the backup is occurring over a network, network connectivity problems might be causing errors.
   • Verify network stability and troubleshoot any interruptions.
4. Resource Constraints:
   • The system might have run out of disk space, memory, or other resources during the backup process.
   • Monitor resource usage and ensure adequate availability.