[SOLVED] dirty-bitmap status: created new

[hfo]

Hi,

relative new here and evaluating Proxmox VE and PBS. And they are looking great.
But I am wondering about backup time of a few VMs, all of them are up and running and auto started, using Linux as guest with guest-agend installed.

Running a scheduled backup I get the following message in the backup log:
103: 2020-09-11 12:00:03 INFO: virtio0: dirty-bitmap status: created new

Manually triggering the same backupjob I will read:
103: 2020-09-10 10:45:40 INFO: virtio0: dirty-bitmap status: OK (536.0 MiB of 20.0 GiB dirty)
103: 2020-09-10 10:45:40 INFO: using fast incremental mode (dirty-bitmap), 536.0 MiB dirty of 20.0 GiB total

Just after a few minutes after the scheduled backup I triggered the job manually again:
103: 2020-09-11 12:44:02 INFO: virtio0: dirty-bitmap status: OK (68.0 MiB of 20.0 GiB dirty)
103: 2020-09-11 12:44:02 INFO: using fast incremental mode (dirty-bitmap), 68.0 MiB dirty of 20.0 GiB total

And both manual triggered jobs are bit faster ;-) Lets say nine times faster, great job.

I observed this behavior now for a few days. VMs are up and running during this time. Any way to get the fast incremental mode also on scheduled mode?

BR Henning

 

[t.lamprecht]

Hi,

could it be that you use "stop" mode for the scheduled backup jobs? As the VM gets probably stopped and thus a new dirty-bitmap created.

 

[hfo]

Hi,

could it be that you use "stop" mode for the scheduled backup jobs? As the VM gets probably stopped and thus a new dirty-bitmap created.

Hi,

negativ, mode is snapshot (just checked).

 

[t.lamprecht]

Does something else stops your VM inbetween? Could you please post the two differing task log as a whole here?

 

[hfo]

Hm, yes, maybe the host was started up again.
I will observe and report back here (with logs if necessary).

 

[hfo]

Hi,

I just observed the backups over the weekend and yes, it runs as expected. I also checked the log of the host and of course I did a few restarts...
Thank you.

BR Henning

 

[high_performer]

Probably depending on ondisk-format? QCOW2 vs raw?
I noticed a complete read of the vm if its a raw format! (Here on CEPH). This read lasts long - even though there are just some bytes to write. Dedup etc looks fine, besides.
Probably worth a new thread...

 

[fabian]

no, the bitmap support is not related to image format at all. the first backup after restarting the VM always needs to read the image fully. the bitmap will also be cleared in other cases:
- image resized
- switch from encrypted to plain or vice versa
- previous backup no longer available on the server

 

[high_performer]

somewhat clearer - but further questions:
If I do a "move disk" in PVE the ondisk format might change. Next backup starts: would this be again a full-backup?
What happens after migrate vm to another host?

 

[fabian]

somewhat clearer - but further questions:
If I do a "move disk" in PVE the ondisk format might change. Next backup starts: would this be again a full-backup?

yes

What happens after migrate vm to another host?

yes as well

in both cases it is possible in theory to avoid it (the size and data from the guest's point of view remains the same), but it is not yet implemented.

 

[high_performer]

Seems i am testing / trying too often too much too different things!
I was astonished to see again a "full" backup starting after doing things. Move disk (with and without changing ondisk-format, local, nfs, ceph, iscsi), migrate, vzdump on stopped vm etc.

Fabian, thanks for your answers.
I will continue my tests.
PBS is great! Really great rocket science! Thanks!

 

[Timo Goetze]

I switched from "standard" full backup type to pbs. I put some VMs on the new PBS Storage and removed from other backup storage which were never started since I configured them to use PBS backup storage.

I would expect that these switched-off VMs are backupped once and after this dirty bitmap says: 100% is the same as last backup - so i dont need to copy data.. But they are fully backupped over and over again.

Is this expected behaviour or is there a bug with persisting the dirty bitmap if VM is switched off?

In other running VMs I watch normal expected behaviour.

7.1-10 and 2.1-5

Thanks!

 

[Dunuin]

Someone of the staff once explained why it is not possible to have persistent dirty bitmaps (would only be possible with qcow2). As soon as you shutdown a VM (or the VM get shutdown by a "stop" type backup task) the dirty bitmap is lost and the whole virtual disk needs to be read again.

 

[fabian]

yes, like @Dunuin said - the dirty bitmap is not persistent. the issue with making it (the technical obstacles are easy to overcome ) is that as soon as anything modifies the disk image between VM starts, the bitmap is invalid but we have no way to tell (except reading the full disk image and comparing it against a hash, which is exactly the cost that the persistent bitmap is supposed to get rid of). even worse, since we can't tell, the error is propagated from one backup to the next, and all the backups are (possibly very subtly) corrupt.

 

[JesseI]

yes, like @Dunuin said - the dirty bitmap is not persistent. the issue with making it (the technical obstacles are easy to overcome ) is that as soon as anything modifies the disk image between VM starts, the bitmap is invalid but we have no way to tell (except reading the full disk image and comparing it against a hash, which is exactly the cost that the persistent bitmap is supposed to get rid of). even worse, since we can't tell, the error is propagated from one backup to the next, and all the backups are (possibly very subtly) corrupt.

So is it abnormal that every single backup to the PBS says "INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared", even if there was no reboot or modification of the VM? Our disks are all in raw format (imported from VMWare).

And would we be better off converting over to QCOW format to have more persistent dirty bitmaps? Right now the backup jobs that run every 2 hours are taking about 6 hours to run.

 

[fabian]

So is it abnormal that every single backup to the PBS says "INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared", even if there was no reboot or modification of the VM? Our disks are all in raw format (imported from VMWare).

if that happens either the latest snapshot (that the bitmap corresponds to) no longer exists, or something else is changing that causes the bitmap to become invalid. are you always backing up to the same storage/PBS datastore+namespace? do you have naming conflicts there (e.g., multiple clusters with the same guest IDs backing up into the same datastore+namespace)?

And would we be better off converting over to QCOW format to have more persistent dirty bitmaps? Right now the backup jobs that run every 2 hours are taking about 6 hours to run.

no, the question of persistent bitmaps is entirely unrelated (and there is no support yet for it in any case). you do have a bitmap, it just gets cleared for some reason.

 

[masgo]

I am having the same issues with PBS and do not yet understand the dirty-bitmap topic. Let me ask a more specific question:
How should I do the backups in order to have (a) consistend backups and (b) make use of the dirty-bitmap feature.

If I configure the backup mode to be "Stop", will this make use of the dirty-bitmap?
(Stop is the best option in terms of consistency. )

What about "Suspend" mode? When does it use dirty-maps?
Consistency is slightly limited, but probably still ok. The biggest problem I see is that suspend needs to write the RAM to disk (which might take a while for VMs with a lot of ram.


Assumptions:
- VM stays on the same host between backups
- No changes to the VM config between backups. No added, removed or resized disks.
- All backups done to the same PBS, same namespace, no ID conflicts.
- The VMs are not manually turned off in between backups

 

[Dunuin]

How should I do the backups in order to have (a) consistend backups and (b) make use of the dirty-bitmap feature.

If I configure the backup mode to be "Stop", will this make use of the dirty-bitmap?
(Stop is the best option in terms of consistency. )

Dirty-bitmaps get dropped once the VM stops. A "stop" mode backup will stop the VM. And LXCs won't use dirty-bitmaps at all.
So what you want are snapshot-mode backups of VMs with installed QEMU guest agent. The guest agent is important, because otherwise data integrity of snapshot-mode backups can't be ensured because the caches won't be dropped before doing the snapshot.

Not quite sure about suspend-mode backups.