backup job backing up entire disk although mostly empty

[proxwolfe]

Hi,

I have set up a VM with a passed through physical disk.

When I back it up using PVE's backup feature (to a PBS), it seems to backup the entire physical disk although the disk is still mostly empty.

Is that normal/expected?

If so, how I can avoid this, i.e. only backup that part of the disk that is actually occupied?

Thanks

 

[fabian]

is the disk really empty (as in, filled with zeros), or is space just unused? the latter is not really detectable from Qemu's point of view if you fill the "empty" space with zeros, it should all be compacted to a single 4MB chunk. if you don't have a valid dirty bitmap, a backup task will still have to scan the full disk to realize it's full of zeroes though, depending on size and read speed that might still take a while.

the output contained in a full backup log should give you an indication of whats going on.

 

[proxwolfe]

is the disk really empty (as in, filled with zeros), or is space just unused?

No zeros, just freshly formatted.

if you don't have a valid dirty bitmap, a backup task will still have to scan the full disk to realize it's full of zeroes though

Oh, so it is only the first time this takes so long. From the next backup on, it will no where the changes were and only back those up? That, I could live with.

If it's not like that and the next time will still take so long, are there other, more effecient ways to backup? Like only backup files? Can that be configured in any way? Or is that what the PBS client does?

Thanks!

 

[fabian]

No zeros, just freshly formatted.

Oh, so it is only the first time this takes so long. From the next backup on, it will no where the changes were and only back those up? That, I could live with.

If it's not like that and the next time will still take so long, are there other, more effecient ways to backup? Like only backup files? Can that be configured in any way? Or is that what the PBS client does?

Thanks!

the bitmap is lost when the VM is stopped. you can also do a file-based backup inside the VM if it's linux based - that is outside of the scope of PVE though

 

[proxwolfe]

the bitmap is lost when the VM is stopped

When you say "stopped" - do you mean technically stopped as in the PVE web gui "stop" or would the bitmap also be lost in a normal shutdown, reboot/restore(from backup)?

you can also do a file-based backup inside the VM if it's linux based

I might try and combine the two: A nightly full backup from PVE (ideally incrementally, but if not it won't disrupt service too much during the night) and hourly incremental file backups out of the VM via PBS client.

 

[fabian]

When you say "stopped" - do you mean technically stopped as in the PVE web gui "stop" or would the bitmap also be lost in a normal shutdown, reboot/restore(from backup)?

stopped as in if the Qemu process exits. it is migrated when doing a live migration.

 

[proxwolfe]

stopped as in if the Qemu process exits. it is migrated when doing a live migration.

Is there any way to preserve the dirty bitmap (other than migrating the VM to another node)?

 

[fabian]

no. this has been discussed at length in the forum - the issue is that once a VM has been stopped, the bitmap is possibly invalid (as someone/something can modify the disk image without PVE knowing, before the VM starts again). to determine whether it's still valid we need to do the same work we'd need for recreating it (reading and hashing the full disk). to just use it without determining whether it's still valid is dangerous - it would mean creating invalid backups!

 

[proxwolfe]

to just use it without determining whether it's still valid is dangerous - it would mean creating invalid backups!

Understood. Thanks!