I have a setup where I backup PVE VMs to Backup Server. This setup has run for more than a year without any configuration change. Two months ago I observed a steep increase in the time it takes to backup one of the linux VMs. It went up from about 10min to more than one hour. And I fail to find any reasons for this increase.
Versions of PVE and Backup Server are current with respect to community version.
This is a copy of a part of the backup log that shows where the additional time is spent (see red text):
04: 2025-02-15 22:15:26 INFO: 88% (46.4 GiB of 52.5 GiB) in 14m 7s, read: 208.0 MiB/s, write: 169.3 MiB/s
104: 2025-02-15 22:15:29 INFO: 89% (47.1 GiB of 52.5 GiB) in 14m 10s, read: 225.3 MiB/s, write: 176.0 MiB/s
104: 2025-02-15 22:15:32 INFO: 90% (47.6 GiB of 52.5 GiB) in 14m 13s, read: 198.7 MiB/s, write: 166.7 MiB/s
104: 2025-02-15 22:15:35 INFO: 92% (48.4 GiB of 52.5 GiB) in 14m 16s, read: 268.0 MiB/s, write: 266.7 MiB/s
104: 2025-02-15 22:15:38 INFO: 93% (49.2 GiB of 52.5 GiB) in 14m 19s, read: 245.3 MiB/s, write: 230.7 MiB/s
104: 2025-02-15 22:15:41 INFO: 94% (49.7 GiB of 52.5 GiB) in 14m 22s, read: 189.3 MiB/s, write: 162.7 MiB/s
104: 2025-02-15 22:15:44 INFO: 95% (50.3 GiB of 52.5 GiB) in 14m 25s, read: 197.3 MiB/s, write: 193.3 MiB/s
104: 2025-02-15 22:15:47 INFO: 96% (50.7 GiB of 52.5 GiB) in 14m 28s, read: 157.3 MiB/s, write: 157.3 MiB/s
104: 2025-02-15 22:15:50 INFO: 97% (51.3 GiB of 52.5 GiB) in 14m 31s, read: 182.7 MiB/s, write: 182.7 MiB/s
104: 2025-02-15 22:15:53 INFO: 98% (51.8 GiB of 52.5 GiB) in 14m 34s, read: 169.3 MiB/s, write: 169.3 MiB/s
104: 2025-02-15 22:15:56 INFO: 99% (52.4 GiB of 52.5 GiB) in 14m 37s, read: 222.7 MiB/s, write: 222.7 MiB/s
104: 2025-02-15 22:15:59 INFO: 100% (52.5 GiB of 52.5 GiB) in 14m 40s, read: 26.7 MiB/s, write: 26.7 MiB/s
104: 2025-02-15 22:15:59 INFO: Waiting for server to finish backup validation...
104: 2025-02-15 23:08:09 INFO: backup is sparse: 36.00 MiB (0%) total zero data
104: 2025-02-15 23:08:09 INFO: backup was done incrementally, reused 8.69 TiB (99%)
104: 2025-02-15 23:08:09 INFO: transferred 52.51 GiB in 4010 seconds (13.4 MiB/s)
104: 2025-02-15 23:08:09 INFO: adding notes to backup
104: 2025-02-15 23:08:09 INFO: prune older backups with retention: keep-daily=7, keep-monthly=2, keep-weekly=2, keep-yearly=1
104: 2025-02-15 23:08:10 INFO: running 'proxmox-backup-client prune' for 'vm/104'
104: 2025-02-15 23:08:10 INFO: pruned 1 backup(s) not covered by keep-retention policy
104: 2025-02-15 23:08:10 INFO: HOOK: backup-end snapshot 104
104: 2025-02-15 23:08:10 INFO: Finished Backup of VM 104 (01:06:58)
I guess the server backup validation is causing the delay. Is that something new introduced recently. I am wondering since there was noch backup configuration change on my side. And is that backup validation something else than the backup verification that can be initiated separately after the backup has been done?
Regards,
Christoph
Versions of PVE and Backup Server are current with respect to community version.
This is a copy of a part of the backup log that shows where the additional time is spent (see red text):
04: 2025-02-15 22:15:26 INFO: 88% (46.4 GiB of 52.5 GiB) in 14m 7s, read: 208.0 MiB/s, write: 169.3 MiB/s
104: 2025-02-15 22:15:29 INFO: 89% (47.1 GiB of 52.5 GiB) in 14m 10s, read: 225.3 MiB/s, write: 176.0 MiB/s
104: 2025-02-15 22:15:32 INFO: 90% (47.6 GiB of 52.5 GiB) in 14m 13s, read: 198.7 MiB/s, write: 166.7 MiB/s
104: 2025-02-15 22:15:35 INFO: 92% (48.4 GiB of 52.5 GiB) in 14m 16s, read: 268.0 MiB/s, write: 266.7 MiB/s
104: 2025-02-15 22:15:38 INFO: 93% (49.2 GiB of 52.5 GiB) in 14m 19s, read: 245.3 MiB/s, write: 230.7 MiB/s
104: 2025-02-15 22:15:41 INFO: 94% (49.7 GiB of 52.5 GiB) in 14m 22s, read: 189.3 MiB/s, write: 162.7 MiB/s
104: 2025-02-15 22:15:44 INFO: 95% (50.3 GiB of 52.5 GiB) in 14m 25s, read: 197.3 MiB/s, write: 193.3 MiB/s
104: 2025-02-15 22:15:47 INFO: 96% (50.7 GiB of 52.5 GiB) in 14m 28s, read: 157.3 MiB/s, write: 157.3 MiB/s
104: 2025-02-15 22:15:50 INFO: 97% (51.3 GiB of 52.5 GiB) in 14m 31s, read: 182.7 MiB/s, write: 182.7 MiB/s
104: 2025-02-15 22:15:53 INFO: 98% (51.8 GiB of 52.5 GiB) in 14m 34s, read: 169.3 MiB/s, write: 169.3 MiB/s
104: 2025-02-15 22:15:56 INFO: 99% (52.4 GiB of 52.5 GiB) in 14m 37s, read: 222.7 MiB/s, write: 222.7 MiB/s
104: 2025-02-15 22:15:59 INFO: 100% (52.5 GiB of 52.5 GiB) in 14m 40s, read: 26.7 MiB/s, write: 26.7 MiB/s
104: 2025-02-15 22:15:59 INFO: Waiting for server to finish backup validation...
104: 2025-02-15 23:08:09 INFO: backup is sparse: 36.00 MiB (0%) total zero data
104: 2025-02-15 23:08:09 INFO: backup was done incrementally, reused 8.69 TiB (99%)
104: 2025-02-15 23:08:09 INFO: transferred 52.51 GiB in 4010 seconds (13.4 MiB/s)
104: 2025-02-15 23:08:09 INFO: adding notes to backup
104: 2025-02-15 23:08:09 INFO: prune older backups with retention: keep-daily=7, keep-monthly=2, keep-weekly=2, keep-yearly=1
104: 2025-02-15 23:08:10 INFO: running 'proxmox-backup-client prune' for 'vm/104'
104: 2025-02-15 23:08:10 INFO: pruned 1 backup(s) not covered by keep-retention policy
104: 2025-02-15 23:08:10 INFO: HOOK: backup-end snapshot 104
104: 2025-02-15 23:08:10 INFO: Finished Backup of VM 104 (01:06:58)
I guess the server backup validation is causing the delay. Is that something new introduced recently. I am wondering since there was noch backup configuration change on my side. And is that backup validation something else than the backup verification that can be initiated separately after the backup has been done?
Regards,
Christoph
