Best way to configure big disk backups

[plokko]

I'm ruinning PBE as a backup service for my PVE systems and, as i plan to backup large set of data, i want to know what's the best approach to my problem.

I have a VM with 3 disks:
one system disk, one database disk and one for storage.
The first two disk are reasonably small (50 and 300Gb) but the last one is huge, at least 1.4Tb and will grow esponentially.
The system is used for storage of critical files, i need to always have at least one copy of everything.

Is the backup differential? (ex. if the total size of VM is 3Tb and i write 1Gb per day, 3 backups for 3 consequent days will occupy 9.3Tb or 3.3Tb?)

If i neet to have different retention periods for different VMs (ex. dalily backup for VM but keep only last 30 for VM 100 and only last 3 backup for VM 101) do i have to create different datastores?
If so is there a problem if i put the different datastores in the same path or do i have to create one folder for each datastore?
Can the retention periods be set by disk in the same VM? (ex.system and database disk will retain last 10 copies, data disk last 2 copies)

 

[aaron]

Is the backup differential?

There are two things to consider. First how the backups are stored on the PBS. The data is split into chunks. If the chunks did not change in between backups, no new data needs to be stored and that chunk is then referenced by the new backup as well. So if existing data on the disk isn't changing a lot, then there should only be minimal additional space usage with each backup.

Secondly, how the VMs are backed up. This is done by tracking which parts of the disk has been written to since the last backup. This is done using Qemus dirty bitmap feature. IIRC the dirty bitmap does not survive a cold boot of the VM. So after you cold boot the VM, the first backup will take some time because the whole disk needs to be read. But only changed areas will be using up additional space (see the previous paragraph). After the first backup, each following backup will only look at the parts of the disk that have been written to since the last backup. Depending on how much (or little) has changed since the last backup, it can be finished very quickly.

If i neet to have different retention periods for different VMs (ex. dalily backup for VM but keep only last 30 for VM 100 and only last 3 backup for VM 101) do i have to create different datastores?

As far as I know yes as this is configured on a datastore level. But depending on how well the VMs can use the same chunks, you might be able to just have the longer retention period for the other VMs as well.

If so is there a problem if i put the different datastores in the same path or do i have to create one folder for each datastore?

You absolutely need to have different locations for different datastores!

Can the retention periods be set by disk in the same VM? (ex.system and database disk will retain last 10 copies, data disk last 2 copies)

No.

 

[plokko]

There are two things to consider. First how the backups are stored on the PBS. The data is split into chunks. If the chunks did not change in between backups, no new data needs to be stored and that chunk is then referenced by the new backup as well. So if existing data on the disk isn't changing a lot, then there should only be minimal additional space usage with each backup.

Secondly, how the VMs are backed up. This is done by tracking which parts of the disk has been written to since the last backup. This is done using Qemus dirty bitmap feature. IIRC the dirty bitmap does not survive a cold boot of the VM. So after you cold boot the VM, the first backup will take some time because the whole disk needs to be read. But only changed areas will be using up additional space (see the previous paragraph). After the first backup, each following backup will only look at the parts of the disk that have been written to since the last backup. Depending on how much (or little) has changed since the last backup, it can be finished very quickly.


As far as I know yes as this is configured on a datastore level. But depending on how well the VMs can use the same chunks, you might be able to just have the longer retention period for the other VMs as well.


You absolutely need to have different locations for different datastores!


No.

Thank you very much, really clear and to the point.
The VM should only add new files in the larger datastore and write files in the database disk (mysql) so backup difference should be negligible.