[SOLVED] Dirty bitmap after reboot

[showiproute]

Hello everyone,

I am a bit confused or maybe I am misinterpreting something: As soon as I reboot a PVE server the dirty-bitmap is all gone:

INFO: started backup task 'd3444593-26bf-4cc7-8ff3-0335701b2440'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO: scsi1: dirty-bitmap status: created new
INFO: scsi10: dirty-bitmap status: created new
INFO: scsi2: dirty-bitmap status: created new
INFO: scsi3: dirty-bitmap status: created new
INFO: scsi4: dirty-bitmap status: created new
INFO: scsi7: dirty-bitmap status: created new
INFO:   0% (1.6 GiB of 20.4 TiB) in 3s, read: 544.0 MiB/s, write: 40.0 MiB/s
INFO:   1% (208.5 GiB of 20.4 TiB) in 17m 41s, read: 200.3 MiB/s, write: 3.8 MiB/s
INFO:   2% (417.1 GiB of 20.4 TiB) in 38m 41s, read: 169.5 MiB/s, write: 0 B/s
INFO:   3% (625.6 GiB of 20.4 TiB) in 1h 3m 52s, read: 141.3 MiB/s, write: 0 B/s
INFO:   4% (834.0 GiB of 20.4 TiB) in 1h 33m 3s, read: 121.9 MiB/s, write: 0 B/s
INFO:   5% (1.0 TiB of 20.4 TiB) in 2h 2m 13s, read: 122.1 MiB/s, write: 0 B/s

and so on...


As you can see it only reads data but mostly does not write anything to PBS (which resides on the same physical server).

If the status is completed after one day and a couple of hours it "remembers" the dirty bitmap.

Is this an expected behaviour?

 

[fiona]

Hi,
yes, the dirty bitmap is not saved when the VM is stopped. It'd be very difficult to ensure that the bitmap stays consistent with the disk, e.g. doing a restore/rollback/etc. would break the consistency.

It only writes little data, because the other chunks are already known to PBS and so don't have to be re-uploaded.

 

[showiproute]

Servus @Fabian_E thanks for clarification!
So is there any possiblitiy to speed up the backup process?
20 TB on regular spinners usually take a while.

 

[fiona]

If the storage is shared (otherwise migration will take forever ) you can migrate the VM to another node before rebooting the node it was on. That will preserve the dirty bitmap. But I'm not aware of any other way.

 

[showiproute]

No unfortunately it's not a shared storage backend.
Anyway thanks for your support and clarification!

 

[showiproute]

@Fabian_E : What would you/Proxmox team think about adding a "Pause" button for a backup?
Due to the recent issues with PVE kernel and unable to run any Windows VM I had to reboot the server several time (I am using PVE & PBS on the same server).

Everytime I start the backup it starts at 0 and needs to build this bitmap which takes around two days. During that time the IO delay is usually at around 70 - 75 and some other more time critical things takes longer.

So from a user perspecitve it might be a greate feature to temporarily pause a backup process do the more time critical stuff and continue the backup without loosing any progress so far.
Currently I need to stop the backup process and rerun it where it starts at 0 again.

INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi10: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi2: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi3: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi4: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi7: dirty-bitmap status: existing bitmap was invalid and has been cleared

 

[fiona]

I think it'd be pretty difficult from a technical perspective to implement such a feature. The backup mechanism would need to switch to using a persistent snapshot or similar and it's currently not designed for that.