Hello, we have 3 proxmox servers, we take weekly backups of our containers on an external drive using NFS. The first 2 server's backups are successfully being stored. However, the 3rd server seems to get stuck while taking a backup.
Here is the backup job logs, we had added a write bandwidth limit of 10000 KB/s, it seemed to fix the problem for a few weeks
```
INFO: starting new backup job: vzdump 101 105 106 107 108 109 111 112 103 --storage externalBackups --mode snapshot --node rns-server2 --compress lzo --prune-backups 'keep-weekly=2' --mailto yasin@rnssol.com --mailnotification always --quiet 1
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2022-11-12 23:59:00
INFO: status = running
INFO: CT Name: wazuh-siem
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: bandwidth limit: 10000 KB/s
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
Logical volume "snap_vm-101-disk-0_vzdump" created.
INFO: creating vzdump archive '/mnt/pve/externalBackups/dump/vzdump-lxc-101-2022_11_12-23_59_00.tar.lzo'
INFO: Total bytes written: 8601344000 (8.1GiB, 9.7MiB/s)
INFO: archive file size: 4.12GB
INFO: prune older backups with retention: keep-weekly=2
INFO: removing backup 'externalBackups:backup/vzdump-lxc-101-2022_10_08-23_59_00.tar.lzo'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-101-disk-0_vzdump" successfully removed
INFO: Finished Backup of VM 101 (00:14:20)
INFO: Backup finished at 2022-11-13 00:13:20
INFO: Starting Backup of VM 103 (lxc)
INFO: Backup started at 2022-11-13 00:13:20
INFO: status = running
INFO: CT Name: troves-testing-Sevser
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: bandwidth limit: 10000 KB/s
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
Logical volume "snap_vm-103-disk-0_vzdump" created.
WARNING: Sum of all thin volume sizes (2.56 TiB) exceeds the size of thin pool pve2/data2 and the size of whole volume group (<1.82 TiB).
INFO: creating vzdump archive '/mnt/pve/externalBackups/dump/vzdump-lxc-103-2022_11_13-00_13_20.tar.lzo'
```
The journalctl logs of the day of the backup are attached in the file, it doesn't show any significant errors which we can debug, help would be appreciated.
Thanks.
Here is the backup job logs, we had added a write bandwidth limit of 10000 KB/s, it seemed to fix the problem for a few weeks
```
INFO: starting new backup job: vzdump 101 105 106 107 108 109 111 112 103 --storage externalBackups --mode snapshot --node rns-server2 --compress lzo --prune-backups 'keep-weekly=2' --mailto yasin@rnssol.com --mailnotification always --quiet 1
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2022-11-12 23:59:00
INFO: status = running
INFO: CT Name: wazuh-siem
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: bandwidth limit: 10000 KB/s
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
Logical volume "snap_vm-101-disk-0_vzdump" created.
INFO: creating vzdump archive '/mnt/pve/externalBackups/dump/vzdump-lxc-101-2022_11_12-23_59_00.tar.lzo'
INFO: Total bytes written: 8601344000 (8.1GiB, 9.7MiB/s)
INFO: archive file size: 4.12GB
INFO: prune older backups with retention: keep-weekly=2
INFO: removing backup 'externalBackups:backup/vzdump-lxc-101-2022_10_08-23_59_00.tar.lzo'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-101-disk-0_vzdump" successfully removed
INFO: Finished Backup of VM 101 (00:14:20)
INFO: Backup finished at 2022-11-13 00:13:20
INFO: Starting Backup of VM 103 (lxc)
INFO: Backup started at 2022-11-13 00:13:20
INFO: status = running
INFO: CT Name: troves-testing-Sevser
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: bandwidth limit: 10000 KB/s
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
Logical volume "snap_vm-103-disk-0_vzdump" created.
WARNING: Sum of all thin volume sizes (2.56 TiB) exceeds the size of thin pool pve2/data2 and the size of whole volume group (<1.82 TiB).
INFO: creating vzdump archive '/mnt/pve/externalBackups/dump/vzdump-lxc-103-2022_11_13-00_13_20.tar.lzo'
```
The journalctl logs of the day of the backup are attached in the file, it doesn't show any significant errors which we can debug, help would be appreciated.
Thanks.