Backups are not using dirty-bitmap like I thought

[jayg30]

I have PBS running as a VM on node 1 of my 3 node Proxmox Cluster.
I setup a backup job within PVE to backup my running VMs everyday at 1:30.
The backup occurs but inspection of the logs it appears to be reading all blocks and not using dirty-bitmap.
I'm at a lose as to why.

I've tested manually performing backups of 1 particular VM to the same datastore within a few minutes and I receive the line in the logs about dirty-bitmap. All VM's are running during the backups and haven't been shutdown or restarted between backups. What other requirements are there? Perhaps it's is using the dirty-bitmap but it's not being written to the logs? IDK.

For instance I just manually ran a backup to PBS for this 1 VM (which is part of my backup job). There were already 3 existing backups;

INFO: starting new backup job: vzdump 131 --mode snapshot --node pve02 --remove 0 --storage pbs-store-01
INFO: Starting Backup of VM 131 (qemu)
INFO: Backup started at 2020-07-28 21:05:01
INFO: status = running
INFO: VM Name: vdc01
INFO: include disk 'scsi0' 'local-zfs:vm-131-disk-0' 35328M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/131/2020-07-29T01:05:01Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '2934633a-e811-4a81-b93d-2186609a544a'
INFO: resuming VM again
INFO: using fast incremental mode (dirty-bitmap), 496.0 MiB dirty of 34.5 GiB total
INFO: status: 99% (492.0 MiB of 496.0 MiB), duration 3, read: 164.0 MiB/s, write: 164.0 MiB/s
INFO: status: 100% (496.0 MiB of 496.0 MiB), duration 4, read: 4.0 MiB/s, write: 4.0 MiB/s
INFO: backup was done incrementally, reused 34.02 GiB (98%)
INFO: transferred 496.00 MiB in 4 seconds (124.0 MiB/s)
INFO: Finished Backup of VM 131 (00:00:04)
INFO: Backup finished at 2020-07-28 21:05:05
INFO: Backup job finished successfully
TASK OK

Cool, we see fast incremental mode (dirty-bitmap) and the job finished in (00:00:04).

Look back at the last scheduled backup in PVE to PBS and we see;

INFO: starting new backup job: vzdump 100 103 107 130 131 --mode snapshot --mailnotification always --quiet 1 --mailto user@domain.com --storage pbs-store-01 --compress zstd
INFO: skip external VMs: 107, 130
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2020-07-28 01:30:02
INFO: status = running
INFO: VM Name: test
INFO: include disk 'scsi0' 'local-zfs:vm-100-disk-0' 92G
INFO: include disk 'scsi1' 'local-zfs:vm-100-disk-1' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2020-07-28T05:30:02Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '420110d1-498a-4ac6-a5f3-8db5f2d4e96d'
INFO: resuming VM again
INFO: status: 0% (780.0 MiB of 192.0 GiB), duration 3, read: 260.0 MiB/s, write: 260.0 MiB/s
INFO: status: 1% (2.0 GiB of 192.0 GiB), duration 11, read: 158.0 MiB/s, write: 158.0 MiB/s
...
INFO: status: 99% (190.3 GiB of 192.0 GiB), duration 921, read: 226.2 MiB/s, write: 226.2 MiB/s
INFO: status: 100% (192.0 GiB of 192.0 GiB), duration 929, read: 220.5 MiB/s, write: 220.5 MiB/s
INFO: backup was done incrementally, reused 177.45 GiB (92%)
INFO: transferred 192.00 GiB in 929 seconds (211.6 MiB/s)
INFO: Finished Backup of VM 100 (00:15:32)
INFO: Backup finished at 2020-07-28 01:45:34

INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2020-07-28 01:45:34
INFO: status = running
INFO: VM Name: app01
INFO: include disk 'scsi0' 'local-zfs:vm-103-disk-0' 34G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/103/2020-07-28T05:45:34Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '91c23859-dcf0-462e-986c-63fdd21597d9'
INFO: resuming VM again
INFO: status: 1% (696.0 MiB of 34.0 GiB), duration 3, read: 232.0 MiB/s, write: 232.0 MiB/s
INFO: status: 3% (1.3 GiB of 34.0 GiB), duration 6, read: 213.3 MiB/s, write: 213.3 MiB/s
...
INFO: status: 99% (33.7 GiB of 34.0 GiB), duration 153, read: 229.3 MiB/s, write: 229.3 MiB/s
INFO: status: 100% (34.0 GiB of 34.0 GiB), duration 155, read: 174.0 MiB/s, write: 174.0 MiB/s
INFO: backup was done incrementally, reused 33.29 GiB (97%)
INFO: transferred 34.00 GiB in 155 seconds (224.6 MiB/s)
INFO: Finished Backup of VM 103 (00:02:36)
INFO: Backup finished at 2020-07-28 01:48:10

INFO: Starting Backup of VM 131 (qemu)
INFO: Backup started at 2020-07-28 01:48:10
INFO: status = running
INFO: VM Name: vdc01
INFO: include disk 'scsi0' 'local-zfs:vm-131-disk-0' 35328M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/131/2020-07-28T05:48:10Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '3527fb90-89b0-41ce-9603-53236eaa95d9'
INFO: resuming VM again
INFO: status: 1% (688.0 MiB of 34.5 GiB), duration 3, read: 229.3 MiB/s, write: 229.3 MiB/s
INFO: status: 3% (1.3 GiB of 34.5 GiB), duration 6, read: 210.7 MiB/s, write: 210.7 MiB/s
INFO: status: 5% (1.9 GiB of 34.5 GiB), duration 9, read: 216.0 MiB/s, write: 216.0 MiB/s
...
INFO: status: 96% (33.4 GiB of 34.5 GiB), duration 153, read: 220.0 MiB/s, write: 220.0 MiB/s
INFO: status: 98% (34.0 GiB of 34.5 GiB), duration 156, read: 221.3 MiB/s, write: 221.3 MiB/s
INFO: status: 100% (34.5 GiB of 34.5 GiB), duration 159, read: 156.0 MiB/s, write: 156.0 MiB/s
INFO: backup was done incrementally, reused 33.53 GiB (97%)
INFO: transferred 34.50 GiB in 159 seconds (222.2 MiB/s)
INFO: Finished Backup of VM 131 (00:02:40)
INFO: Backup finished at 2020-07-28 01:50:50
INFO: Backup job finished successfully
TASK OK

Here we see 3 VMs are being backed up as part of this job, the last one being the same VM as manually backed up above. It doesn't say anything about dirty-bitmap. It appears to be reading the disks and it takes (00:02:40) to finish. Looking at all the logs for the scheduled backup produce the same results, always about the same amount of time to perform the backups of the VM and no dirty-bitmap line. Something isn't right...

Thanks

PS: PVE and PBS are both fully updated against the community repo

 

[jayg30]

I did find this information;
https://lists.proxmox.com/pipermail/pve-user/2020-July/171884.html

Which states that if you try to backup a VM to multiple datastores you'll essentially invalidate the qemu dirty-bitmap as PBS doesn't track this per datastore currently.

I did try this (pbs-store-01 & pbs-store-02) for some testing, but I had removed the 2nd datastore and scheduled backup job in PVE at least 2 backup jobs ago. If that could still be causing the problem someone let me know because I don't see how.

That did raise a question. How will standard vzdump backups in PVE impact PBS backups? Meaning if I have PVE do a regular full backup of a VM between PBS backups will the dirty-bitmap be wiped? I'd think not since you're not (currently) using it to allow incremental backups in PVE.

 

[jayg30]

I guess performing the manual backup of my VMs must have fixed it because it worked correctly from the scheduled backup this time.

Either that or deleting the unused secondary datastore. That's the only other thing I did.