backup vm pve to pbs, how to enable incremental

[roonsie22]

I have setup an Proxmox Backup Server beta. On my PVE the PBS storage is available.

So i can run and schedule QEMU VM backups to the PBS storage, but they seem to run full every time.

Can i enable incremental backups this way (through the PVE GUI)?

In another thread (Windows backup in seconds) i saw a line "INFO: using fast incremental mode (dirty-bitmap), 1.2 GiB dirty of 32.0 GiB total" in the logfile.

Do i need to enable QEMU dirty bits somewhere in the config, or is there another way to make incremental VM backups?

 

[t.lamprecht]

So i can run and schedule QEMU VM backups to the PBS storage, but they seem to run full every time.

Is the VM running? As the dirty-bitmap support needs the VM to be running.

Can i enable incremental backups this way (through the PVE GUI)?

It is always done incrementally, as a later log line like:

1890: 2020-07-11 19:36:56 INFO: status: 100% (64.0 GiB of 64.0 GiB), duration 124, read: 100.0 MiB/s, write: 100.0 MiB/s
1890: 2020-07-11 19:36:56 INFO: backup was done incrementally, reused 64.00 GiB (100%)

Should show, it's just that it has to read everything from the storage to be safe, whereas with a running VM the dirty-bitmap info is in memory and thus can be done with reading only what really changed.

 

[roonsie22]

Thank you, this is correct, VM was powered off indeed, incremental works, and Qemu dirty bits backup is like a flash of thunder!

Great work!

 

[Helle]

So I am testing backups of my VMs too and have a few VM's that are shut down.

Will those take a lot of time like many hours (one VM is 3.1 TB) every time the backup is run since there is no dirty-bit support ?
If so, any VM not running during backup will potentially render the backup unsuccessful since that can likely force the backup to run over next scheduled backup cycle ??

If this is true, there should be an option to skip non-running VM's

Is this a PVE or PBS "issue" or by design and is this good/bad/acceptable ?

One could argue that VM's normally should be up and running and that is usually true but there are many possible ways this could cause problems, me thinks.

Having a large VM and shutting it down and forgetting to manually untick the VM from a backup job should not break everything (which it now seems to do)

To be clear, I am very enthusiastic about PBS and only want to understand and possibly improve how things work.

/Helle

 

[Helle]

I my case the VM currently backing up has progressed 47% in 4hours and 15 minutes and was backed up yesterday and have not been used since last backup..

 

[t.lamprecht]

Will those take a lot of time like many hours (one VM is 3.1 TB) every time the backup is run since there is no dirty-bit support ?

For now yes. Albeit, after the first backup was made it will take way less time than a classic full backup will, as while all data needs to be read only that what really changed needs to be uploaded (written) to the backup server.

If so, any VM not running during backup will potentially render the backup unsuccessful since that can likely force the backup to run over next scheduled backup cycle ??

There's a per-node backup lock, which future jobs will wait on if a currently running job hasn't finished yet.

Is this a PVE or PBS "issue" or by design and is this good/bad/acceptable ?

The main reason is that when the VM is running it has full control over its storage, so we can tell for sure if something on the disk was changed or not. Once stopped, anything could theoretically happen to those VM disk images - so we play it safe and verify all of it. If we find a guaranteed sane way optimizing this, we may do so in the future, for now we wanted to play safe - backups are really important after all.

Having a large VM and shutting it down and forgetting to manually untick the VM from a backup job should not break everything (which it now seems to do)

That won't break anything, it may take long but that is not an issue.

 

[Helle]

For now yes. Albeit, after the first backup was made it will take way less time than a classic full backup will, as while all data needs to be read only that what really changed needs to be uploaded (written) to the backup server.

There's a per-node backup lock, which future jobs will wait on if a currently running job hasn't finished yet.

My test environment is a single node backing up to a vm in another single node

The main reason is that when the VM is running it has full control over its storage, so we can tell for sure if something on the disk was changed or not. Once stopped, anything could theoretically happen to those VM disk images - so we play it safe and verify all of it. If we find a guaranteed sane way optimizing this, we may do so in the future, for now we wanted to play safe - backups are really important after all.

Ok, makes sense but....

I have two larger VMs on the same PVE that consumes over 7TB each and the initial backup seems to take 21hours per VM so my question is.

Will the dirty-bit tracking persist if I reboot the VM nicely, IE soft reboot, warm reboot? and only a shutdown where the KVM machine instance is destroyed will kill the dirty bit tracking ?

If that is the case, regular maintenance of the VM is possible but if both scenarios kills Dirti-bit, I can not use this for backing up machines that will take several days to catch up and render every other daily backup on same node delayed.

I can admit that my test environment is not the best but it is what it is and the question had to be asked.

That won't break anything, it may take long but that is not an issue.

To have daily backups delayed by a week is at least a break of SLA in most cases.

 

[t.lamprecht]

Will the dirty-bit tracking persist if I reboot the VM nicely, IE soft reboot, warm reboot? and only a shutdown where the KVM machine instance is destroyed will kill the dirty bit tracking ?

Currently, it normally stays as long as the KVM process persists, so a reboot from inside the VM won't drop the bitmap.
We plan to at least sync it out and in if there's a VM full restart triggered over the PVE API.

I can not use this for backing up machines that will take several days to catch up and render every other daily backup on same node delayed.

how does it gets backed up currently?

 

[Helle]

Currently, it normally stays as long as the KVM process persists, so a reboot from inside the VM won't drop the bitmap.
We plan to at least sync it out and in if there's a VM full restart triggered over the PVE API.

That is super nice !! That is how most backup systems that I have come across do it. a bitmap tracking file...

how does it gets backed up currently?

They are not backed up because:
1. Non important machines that are just for lab usage
2. Had no good way to back them up until PBS happend

I am arguing just for the good cause to make PBS suitable for a hypothetical case.

 

[EuroDomenii]

It seems that incremental doesn't work with running KVM on LVM-thin storage ...

 

[tom]

It seems that incremental doesn't work with running KVM on LVM-thin storage ...

Works here. Post your backup log.

 

[MAgno Santos]

Hello everyone,

Amazing JOB being done. I'm a Fan

Regarding the Dirty Bitmap i understand the it needs de VM to be running

Currently, it normally stays as long as the KVM process persists, so a reboot from inside the VM won't drop the bitmap.
We plan to at least sync it out and in if there's a VM full restart triggered over the PVE API.

But isn't there any any when the VM shutdowns to "save it" it? Isn't even safer if a backup is made to a VM that was powered off between backups to detected that no change were made at all without the need to fully read it?

 

[t.lamprecht]

But isn't there any any when the VM shutdowns to "save it" it?

Currently, not really. Persistent bitmaps are only a thing for qcow2, and we do not yet support it there. As said, we'll still look into optimizing this.

Isn't even safer if a backup is made to a VM that was powered off between backups to detected that no change were made at all without the need to fully read it?

If you're using the QEMU guest agent a live backup is normally safe too - depends a bit on workload though.
But yes, a stop mode backup is normally the safest option if one wants a general approach.

 

[MAgno Santos]

But yes, a stop mode backup is normally the safest option if one wants a general approach.

What i wanted to meant was: why is not then possible to implement a Fast Incremental Backup on Non-Running VMs? Is not possible (even if for just ZFS Storage VMs) to detected that there was no changes on the VMs Disk? For instance, i have a couple of VMs that are Off but they belong to a pool which is regularly backed up... having all the read process just to identify that 100% of previous backup was used isn't the most efficient way.

But in any case, everything is a lot better than before PBS

 

[EuroDomenii]

Works here. Post your backup log.

My bad... Even the VM was running, I was trying to do backup after restart, so dirty-bitmaps from RAM were lost. Incremental works on LVM-thin storage also, if the VM stays running

 

[t.lamprecht]

to detected that there was no changes on the VMs Disk?

Yes, by verifying that like we do now Else, no, there's no general way of knowing it - the underlying block device could have been accessed by a third party we have no knowledge and no control over.

having all the read process just to identify that 100% of previous backup was used isn't the most efficient way.

Sure, but for a thing like backups safety beats "fast but maybe the backups is unusable" easily.

There's an idea that we do a storage snapshot for those who support it, and only do the diff - like rbd diff for ceph has a relatively simple data format defiintion which could work for that, possibly it could be done for ZFS too. But as said, just an idea, nobody did any serious work on that yet.

 

[carsten2]

A backup solution which is not able to do fast incremental backups for unchanged and switched of VMs (the safest method), is not really a good solution. There should be a solution.

 

[t.lamprecht]

So, out of interest, what good backup solution being able to do both do you use now for that?

Anyway, there are already plans to save the dirty-bitmap state under certain controlled circumstances, like API triggered guest reboot or a stop-mode backup of a running VM, and then re-sync it - so some situations will be still improved (note, this is still a beta )

 

[carsten2]

Currently I use znapzend. WIth this I can:

1) Start a VM directory from the backup at any remote location
2) Restore a previous version of the VM by just reverting to the snapshot
3) Clone a snapshot to start a previous version directly from the backup for testing.
4) Clone a snapshot and mount the file system (like ntfs) directly from the backup for restoring files.

And all this without an explicit or time consuming restore operation.
Unfortunately there is no proxmox UI support for this solution (it would be really nice, for peope using only ZFS).

BTW: How to solve 4 with PBS, i.e. how to restore a single from a 1 TB zvol backup in ntfs?

 

[t.lamprecht]

With the big asterisks of being limited to a single filesystem, which cannot be made realtime-replicated/clustered (at list sanely), we wanted to make a general solution. Also, as you mentioned, it has no Proxmox VE awareness, so you need to manage that all yourself.

If you want to do snapshots as a backup strategy (which one should really plan out well, as if they ain't synced somewhere safe its worth nothing) you could also just use Proxmox VE Replication (or the more general pve-zsync), which can do this for years.

2) Restore a previous version of the VM by just reverting to the snapshot

In ZFS you cannot simply rollback to an arbitrary snapshot, you can clone it, but that what you suggest is not possible - at least if you are not willing to destroy all snapshots in-between. Snapshots also have to be a list, they can not be an arbitrary tree. PBS does not care about this at all and is as flexible as one can imagine.

BTW: How to solve 4 with PBS, i.e. how to restore a single from a 1 TB zvol backup in ntfs?

This is already possible with a bit of manual work now, and it is currently being better integrated:
You can use our QEMU Proxmox Backup Server block-backend to access any image on a backup server of your choice, either directly from a VM or through NBD on the host, this can then be mounted and used just fine.
https://lists.proxmox.com/pipermail/pve-user/2020-July/171883.html

The upcoming "more native integrated" version will be a simple "proxmox-backup-client map/unmap" command.