PBS Incremental Backup Question

[theprez1980]

So in reading the forums, I understand that each backup is a full backup due to how deduplication works. But I'm unclear as to if there's any impact to network traffic.

Said differently, does each backup (nightly in my case) transfer all the same data to the remote server each night and it then deduplicates or is there some intelligence to simply transfer the chunks that have changed and nothing more?

I ask because it seems like subsequent backups occur quicker than the first but maybe I'm dreaming that up.

Thanks

 

[Chris]

So in reading the forums, I understand that each backup is a full backup due to how deduplication works. But I'm unclear as to if there's any impact to network traffic.

Said differently, does each backup (nightly in my case) transfer all the same data to the remote server each night and it then deduplicates or is there some intelligence to simply transfer the chunks that have changed and nothing more?

I ask because it seems like subsequent backups occur quicker than the first but maybe I'm dreaming that up.

Thanks

Hi,
yes there is an optimization in place to only transfer chunks not present in the previous backup snapshot located in the same group on the Proxmox Backup Server. This is true for all: VM, container and host backups. For VM backups, the dirty bitmap is used to track changed blocks which need to be resend, this is however only possible if the dirty bitmap is not invalidated in between (e.g. by stopping the VM). For container and host backups, all data is read and chunked, only uploading the new chunks.

For host and container backups there is now also (at the time of writing still experimental) change detection mode switch, see https://pbs.proxmox.com/docs/backup-client.html#change-detection-mode

 

[ftodino]

Not wanting to thread hijack, but if the VM is off for weeks at a time, would you expect a daily backup of it to take longer than if it was running 24/7?? I have a deployment VM that I only turn on when I want to do re-imaging. Daily it backs up but for some reason has a dirty bitmap meaning it copies the entire vdisk in every backup. SInce it has been off and therefore it doesn't change, I would expect this backup to take zero time as there should be nothing to backup!!!

 

[UdoB]

has a dirty bitmap meaning it copies the entire vdisk in every backup.

No.

Yes, the dirty bitmap is lost when the VM had been stopped and is restarted now.

Because of this the Backup process needs to read each and every occupied byte of data from the virtual disks to re-calculate the checksum of the corresponding chunk.

But it will transmit only that checksum of a chunk to the PBS. In a second step and only if the checksums are different the new data of these chunks is send to the PBS.

Clever, isn't it

 

[ftodino]

No.

Yes, the dirty bitmap is lost when the VM had been stopped and is restarted now.

Because of this the Backup process needs to read each and every occupied byte of data from the virtual disks to re-calculate the checksum of the corresponding chunk.

But it will transmit only that checksum of a chunk to the PBS. In a second step and only if the checksums are different the new data of these chunks is send to the PBS.

Clever, isn't it

That's a great explanation. Thanks

 

[KGundermann]

Yes, the dirty bitmap is lost when the VM had been stopped and is restarted now.

Is the dirty bitmap only a in-memory table? Why couldn't the map still exist after the VM is shut down?
We have some test systems, which are normally shut down but still in the backup schedule.
So it takes over an 1/2 hour to backup a single VM even the system was shut down and no byte on the disk has changed.

 

[Chris]

Is the dirty bitmap only a in-memory table? Why couldn't the map still exist after the VM is shut down?

See the discussion about this in issue https://bugzilla.proxmox.com/show_bug.cgi?id=3233