Backup to PBS not incremental ?

H

HomerJ.S.

New Member
Aug 7, 2024
16
1
3
Hi community,

I have again a beginner question. I use a setup with a PVE and a PBS, both local in the same network. The PVE has a lot of vm and ct (most only container) and three of them use disk space between 700 GB and 3 TB. Every Saturday a backup job starts in the morning.

And here's what I do not understand. The first time the backup took very long, ok that's obvious. But shouldn't the following backups run very much faster? My setup needs every Saturday more than 10 hours, and for example the nextcloud backup does transfer more than 1 TB, that's the whole containers size.

I thought the backups to the PBS are by default incremental, is that right?
My backups are all suspend backups, mostly because I cannot accept long downtime for some of the servers. Is that part of the problem?
My vm and ct are stored on a nvm ssd, the data is stored on a hdd. Both are ext4. Is that the point?

Here is the part of my backup log, the nextcloud part:

Code: 
INFO: CT Name: NextcloudAIO
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/mnt/daten') in backup
INFO: starting first sync /proc/8159/root/ to /mnt/pve/Daten3-12TB/BackupCacheDir/vzdumptmp775776_105/
INFO: first sync finished - transferred 1.24T bytes in 20107s
INFO: suspending guest
INFO: starting final sync /proc/8159/root/ to /mnt/pve/Daten3-12TB/BackupCacheDir/vzdumptmp775776_105/
INFO: final sync finished - transferred 49.33K bytes in 256s
INFO: resuming guest
INFO: guest is online again after 256 seconds
INFO: creating Proxmox Backup Server archive 'ct/105/2025-11-02T09:12:09Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/mnt/pve/Daten3-12TB/BackupCacheDir/vzdumptmp775776_105//etc/vzdump/pct.conf root.pxar:/mnt/pve/Daten3-12TB/BackupCacheDir/vzdumptmp775776_105/ --include-dev /mnt/pve/Daten3-12TB/BackupCacheDir/vzdumptmp775776_105//. --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 105 --backup-time 1762074729 --entries-max 1048576 --repository root@pam@192.168.1.6:PBS_Backup_4TB
INFO: Starting backup: ct/105/2025-11-02T09:12:09Z   
INFO: Client name: proxmox   
INFO: Starting backup protocol: Sun Nov  2 15:51:32 2025   
INFO: Downloading previous manifest (Sun Oct 26 10:11:45 2025)   
INFO: Upload config file '/mnt/pve/Daten3-12TB/BackupCacheDir/vzdumptmp775776_105//etc/vzdump/pct.conf' to 'root@pam@192.168.1.6:8007:PBS_Backup_4TB' as pct.conf.blob   
INFO: Upload directory '/mnt/pve/Daten3-12TB/BackupCacheDir/vzdumptmp775776_105/' to 'root@pam@192.168.1.6:8007:PBS_Backup_4TB' as root.pxar.didx   
INFO: processed 12.969 GiB in 1m, uploaded 130.661 MiB
INFO: processed 26.301 GiB in 2m, uploaded 130.661 MiB
INFO: processed 39.726 GiB in 3m, uploaded 130.661 MiB
INFO: processed 53.18 GiB in 4m, uploaded 130.661 MiB
INFO: processed 66.562 GiB in 5m, uploaded 130.661 MiB
INFO: processed 79.916 GiB in 6m, uploaded 130.661 MiB
INFO: processed 93.307 GiB in 7m, uploaded 130.661 MiB
INFO: processed 106.703 GiB in 8m, uploaded 130.661 MiB
INFO: processed 120.068 GiB in 9m, uploaded 130.661 MiB
INFO: processed 133.431 GiB in 10m, uploaded 130.661 MiB
INFO: processed 146.509 GiB in 11m, uploaded 130.661 MiB
INFO: processed 159.018 GiB in 12m, uploaded 130.661 MiB
INFO: processed 171.128 GiB in 13m, uploaded 130.661 MiB
INFO: processed 183.231 GiB in 14m, uploaded 130.661 MiB
INFO: processed 195.325 GiB in 15m, uploaded 130.661 MiB
INFO: processed 207.396 GiB in 16m, uploaded 130.661 MiB
INFO: processed 219.484 GiB in 17m, uploaded 130.661 MiB
INFO: processed 231.448 GiB in 18m, uploaded 130.661 MiB
INFO: processed 243.431 GiB in 19m, uploaded 130.661 MiB
INFO: processed 256.41 GiB in 20m, uploaded 130.661 MiB
INFO: processed 269.841 GiB in 21m, uploaded 130.661 MiB
INFO: processed 283.208 GiB in 22m, uploaded 130.661 MiB
INFO: processed 296.569 GiB in 23m, uploaded 130.661 MiB
INFO: processed 309.891 GiB in 24m, uploaded 130.661 MiB
INFO: processed 323.276 GiB in 25m, uploaded 130.661 MiB
INFO: processed 336.598 GiB in 26m, uploaded 130.661 MiB
INFO: processed 349.921 GiB in 27m, uploaded 130.661 MiB
INFO: processed 363.233 GiB in 28m, uploaded 130.661 MiB
INFO: processed 376.596 GiB in 29m, uploaded 130.661 MiB
INFO: processed 389.912 GiB in 30m, uploaded 130.661 MiB
INFO: processed 403.245 GiB in 31m, uploaded 130.661 MiB
INFO: processed 416.583 GiB in 32m, uploaded 130.661 MiB
INFO: processed 429.883 GiB in 33m, uploaded 130.661 MiB
INFO: processed 443.219 GiB in 34m, uploaded 130.661 MiB
INFO: processed 456.485 GiB in 35m, uploaded 130.661 MiB
INFO: processed 469.707 GiB in 36m, uploaded 130.661 MiB
INFO: processed 482.986 GiB in 37m, uploaded 130.661 MiB
INFO: processed 496.098 GiB in 38m, uploaded 130.661 MiB
INFO: processed 507.039 GiB in 39m, uploaded 130.661 MiB
INFO: processed 518.421 GiB in 40m, uploaded 130.661 MiB
INFO: processed 529.778 GiB in 41m, uploaded 130.661 MiB
INFO: processed 540.933 GiB in 42m, uploaded 130.661 MiB
INFO: processed 551.704 GiB in 43m, uploaded 130.661 MiB
INFO: processed 561.53 GiB in 44m, uploaded 130.661 MiB
INFO: processed 571.353 GiB in 45m, uploaded 130.661 MiB
INFO: processed 580.051 GiB in 46m, uploaded 130.661 MiB
INFO: processed 589.375 GiB in 47m, uploaded 130.661 MiB
INFO: processed 598.398 GiB in 48m, uploaded 130.661 MiB
INFO: processed 608.448 GiB in 49m, uploaded 130.661 MiB
INFO: processed 618.48 GiB in 50m, uploaded 130.661 MiB
INFO: processed 628.127 GiB in 51m, uploaded 130.661 MiB
INFO: processed 637.492 GiB in 52m, uploaded 130.661 MiB
INFO: processed 647.691 GiB in 53m, uploaded 130.661 MiB
INFO: processed 658.019 GiB in 54m, uploaded 130.661 MiB
INFO: processed 669.664 GiB in 55m, uploaded 130.661 MiB
INFO: processed 681.448 GiB in 56m, uploaded 130.661 MiB
INFO: processed 693.237 GiB in 57m, uploaded 130.661 MiB
INFO: processed 704.115 GiB in 58m, uploaded 130.661 MiB
INFO: processed 715.436 GiB in 59m, uploaded 130.661 MiB
INFO: processed 726.761 GiB in 1h, uploaded 130.661 MiB
INFO: processed 738.053 GiB in 1h 1m, uploaded 130.661 MiB
INFO: processed 750.223 GiB in 1h 2m, uploaded 132.876 MiB
INFO: processed 763.748 GiB in 1h 3m, uploaded 132.876 MiB
INFO: processed 776.153 GiB in 1h 4m, uploaded 132.876 MiB
INFO: processed 786.632 GiB in 1h 5m, uploaded 132.876 MiB
INFO: processed 796.982 GiB in 1h 6m, uploaded 132.876 MiB
INFO: processed 807.225 GiB in 1h 7m, uploaded 132.876 MiB
INFO: processed 817.178 GiB in 1h 8m, uploaded 542.212 MiB
INFO: processed 821.711 GiB in 1h 9m, uploaded 582.532 MiB
INFO: processed 833.202 GiB in 1h 10m, uploaded 582.532 MiB
INFO: processed 843.292 GiB in 1h 11m, uploaded 845.368 MiB
INFO: processed 854.16 GiB in 1h 12m, uploaded 845.368 MiB
INFO: processed 865.514 GiB in 1h 13m, uploaded 845.368 MiB
INFO: processed 877.13 GiB in 1h 14m, uploaded 845.368 MiB
INFO: processed 888.811 GiB in 1h 15m, uploaded 845.368 MiB
INFO: processed 900.97 GiB in 1h 16m, uploaded 845.368 MiB
INFO: processed 911.507 GiB in 1h 17m, uploaded 845.368 MiB
INFO: processed 922.463 GiB in 1h 18m 0s, uploaded 845.368 MiB
INFO: processed 935.413 GiB in 1h 19m 0s, uploaded 845.368 MiB
INFO: processed 946.342 GiB in 1h 20m 0s, uploaded 845.368 MiB
INFO: processed 958.554 GiB in 1h 21m 0s, uploaded 845.368 MiB
INFO: processed 970.397 GiB in 1h 22m 0s, uploaded 845.368 MiB
INFO: processed 981.421 GiB in 1h 23m 0s, uploaded 845.368 MiB
INFO: processed 992.828 GiB in 1h 24m 0s, uploaded 845.368 MiB
INFO: processed 1000.882 GiB in 1h 25m 0s, uploaded 845.368 MiB
INFO: processed 1012.648 GiB in 1h 26m 0s, uploaded 845.368 MiB
INFO: processed 1022.654 GiB in 1h 27m 0s, uploaded 845.368 MiB
INFO: processed 1.008 TiB in 1h 28m 0s, uploaded 845.368 MiB
INFO: processed 1.018 TiB in 1h 29m 0s, uploaded 845.368 MiB
INFO: processed 1.019 TiB in 1h 30m 0s, uploaded 845.368 MiB
INFO: processed 1.032 TiB in 1h 31m 0s, uploaded 845.368 MiB
INFO: processed 1.044 TiB in 1h 32m 0s, uploaded 845.368 MiB
INFO: processed 1.057 TiB in 1h 33m 0s, uploaded 845.368 MiB
INFO: processed 1.069 TiB in 1h 34m 0s, uploaded 845.368 MiB
INFO: processed 1.081 TiB in 1h 35m 0s, uploaded 845.368 MiB
INFO: processed 1.093 TiB in 1h 36m 0s, uploaded 845.368 MiB
INFO: processed 1.1 TiB in 1h 37m 0s, uploaded 958.471 MiB
INFO: processed 1.105 TiB in 1h 38m 0s, uploaded 1.379 GiB
INFO: processed 1.111 TiB in 1h 39m 0s, uploaded 1.834 GiB
INFO: processed 1.116 TiB in 1h 40m 0s, uploaded 2.779 GiB
INFO: processed 1.12 TiB in 1h 41m 0s, uploaded 6.132 GiB
INFO: processed 1.127 TiB in 1h 42m 0s, uploaded 9.346 GiB
INFO: root.pxar: had to backup 9.387 GiB of 1.128 TiB (compressed 4.893 GiB) in 6124.32 s (average 1.57 MiB/s)
INFO: root.pxar: backup was done incrementally, reused 1.119 TiB (99.2%)
INFO: Uploaded backup catalog (14.77 MiB)
INFO: Duration: 6129.53s   
INFO: End Time: Sun Nov  2 17:33:41 2025   
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=4, keep-monthly=3, keep-weekly=2, keep-yearly=1
INFO: running 'proxmox-backup-client prune' for 'ct/105'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 105 (07:23:25)
INFO: Backup finished at 2025-11-02 17:35:34
INFO: Starting Backup of VM 111 (lxc)
INFO: Backup started at 2025-11-02 17:35:34
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7

I would appreciate every hints and tips.

Thank you
Alex
 
HomerJ.S. said:
INFO: processed 1.127 TiB in 1h 42m 0s, uploaded 9.346 GiB INFO: root.pxar: had to backup 9.387 GiB of 1.128 TiB (compressed 4.893 GiB) in 6124.32 s (average 1.57 MiB/s) INFO: root.pxar: backup was done incrementally, reused 1.119 TiB (99.2%)
Click to expand...
1,1 TB ct105 disk were processed in 1h 45min but only 9,5 GB were changed and incremental saved, so more than 99% were not transfered again.
 
Maybe folks are missing something like CBT mechanism as in vSphere...

Edit: Well, seemingly there is CBT. But... why does a backup like in #1 take so long?
 
Last edited:
You can see it here:
Code: 
INFO: root.pxar: had to backup 9.387 GiB of 1.128 TiB (compressed 4.893 GiB) in 6124.32 s (average 1.57 MiB/s)
 
Thanks for your help.

On both server there are brand new Seagate 12 TB HDD installed, enterprise class. I thought that's enough for a small home server.
The whole sync start with a full copy to the BackupCacheDir. I think that could be the bottleneck. Would a zfs file system make this cache write process obsolete?
 
While a new backup will transfer only changed data, PVE first has to decide which data has been changed.

On a running VM there is a "dirty bitmap", which knows exactly which bytes had been modified since the last backup.

For a VM which has has been stopped in the meantime this is not possible, the "dirty bitmap" has been lost!

Those VMs have to be read from disk. Completely! That takes a long time for multi-TB rotating rust...

It's easy: if you want your backups to finish really quickly then the VMs have to run continuously.


Adding a little bit uncertainty: I know the above was true for a long time. But I am not sure regarding the current state of this behavior: there were plans for a persistent bitmap, right? What's the state?
 
  • Like
Reactions: waltar
HomerJ.S. said:
HDD
Click to expand...
https://pbs.proxmox.com/docs/installation.html#minimum-server-requirements-for-evaluation
“
  • Prefer fast storage that delivers high IOPS for random IO workloads; use only enterprise SSDs for best results.
  • If HDDs are used: Using a metadata cache is highly recommended, for example, add a ZFS special device mirror.”


HomerJ.S. said:
backups to the PBS are by default incremental
Click to expand...
Yes and no. All backups in PVE are complete backups. PBS deduplicates chunks.
 
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom