[SOLVED] Several PBS systems for same Proxmox/Ceph cluster.

[SOLTECSIS - Carles Munyoz]

Hi,
I have a Proxmox/Ceph cluster and two PBS systems, one local and the other one in a remote data center.

The local PBS is bigger, it has enough disk space for keep lot of backups of the virtual machines. In this one I have the next prune policy:
Keep last: 30 Keep weekly: 24 Keep monthly: 12

The remote PBS is smaller and I can keep only the las 15 versions.

If I make the first copy over the local PBS I can see this message related with the dirty-bitmap:
scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared

The second copy I see the next message and the copy is done very quick, in matter of seconds.
scsi0: dirty-bitmap status: OK (35.9 GiB of 250.0 GiB dirty) INFO: using fast incremental mode (dirty-bitmap), 35.9 GiB dirty of 250.0 GiB total

If I make the copy now over the remote PBS, I again get the message indicating that the dirty-bitmap is invalid:
scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared

If every day I make backup over the local PBS and then over the remote PBS, the dirty-bitmap is useless because it is invalid every time I switch from one PBS to the other.

This is a problem, because if I want to make copy of the virtual machines every hour in the local and remote PBS, it is not possible because the copy process takes so much time due that the dirty-bitmap is invalid every time I switch from PBS.

How can I avoid it?
Which is the best way of manage this two PBS, the local one with lot of disk space and the remote one with less disk space?

Thank you very much for your help.

 

[dietmar]

How can I avoid it?

I guess you can use a sync job to copy the data to the remote host (instead of a second backup).

 

[SOLTECSIS - Carles Munyoz]

The problem is that in the remote PBS I have less space for copies.
Is it possible to sync only the last 15 backups from the local PBS to the remote one ?

 

[fabian]

no, but you can have different prune settings on the remote end (you'll sync some short-lived snapshots, but if traffic/bandwidth is not your bottle neck that doesn't really hurt)

 

[SOLTECSIS - Carles Munyoz]

This is not possible because local PBS has more used disk space in the datastore than space disk available in the remote one. Then, it is not possible to make a full sync of the local datastore over the remote one and then prune it.

Moreover, it is not very efficient ... imagine that we have 100 snapshots in the local PBS but we want only 15 in the remote PBS. Every day we will have to sync all the snapshots and remove 85 snapshots.

Is there another option ?

 

[fabian]

This is not possible because local PBS has more used disk space in the datastore than space disk available in the remote one. Then, it is not possible to make a full sync of the local datastore over the remote one and then prune it.

Moreover, it is not very efficient ... imagine that we have 100 snapshots in the local PBS but we want only 15 in the remote PBS. Every day we will have to sync all the snapshots and remove 85 snapshots.

Is there another option ?

you wouldn't need to sync all snapshots each day, only the new ones.

 

[SOLTECSIS - Carles Munyoz]

I don't understand ...

When I enable in my local PBS a sync of a datastore over the remote PBS, all the content of the local datastore will be synced to the remote one, ¿is it correct?
But if the remote one doesn't has enough disk space, then the sync will fail, ¿correct?

Another scenario, imagine that I have enough disk space in the remote PBS for sync all the datastore, but in the local datastore I have 100 snapshots and in the remote one I only want 15. If i prune the remote datastore for keep only the last 15 snapshots and run the garbage collect every day, after the garbage collector removes the data of the pruned snapshots, the next time the sync is run, these removed data will be sent again to the remote PBS datastore. Am I right?

 

[fabian]

I don't understand ...

When I enable in my local PBS a sync of a datastore over the remote PBS, all the content of the local datastore will be synced to the remote one, ¿is it correct?
But if the remote one doesn't has enough disk space, then the sync will fail, ¿correct?

yes

Another scenario, imagine that I have enough disk space in the remote PBS for sync all the datastore, but in the local datastore I have 100 snapshots and in the remote one I only want 15. If i prune the remote datastore for keep only the last 15 snapshots and run the garbage collect every day, after the garbage collector removes the data of the pruned snapshots, the next time the sync is run, these removed data will be sent again to the remote PBS datastore. Am I right?

no, only snapshots (and their chunks) which are newer than the last local one are pulled from the remote

 

[SOLTECSIS - Carles Munyoz]

Then the solution is:

• Setup the Proxmox/Ceph cluster to make backups over the remote PBS, the one with less disk space.
• In the remote PBS keep only the last 15 snapshots.
• Setup a sync job in the local PBS for sync the datastore from the remote PBS.
• In the local PBS setup the prune policy for keep the last 30 snapshots, 24 weekly and 12 monthly.

This way I will not lost the dirty-map because the backups will be done always over the same PBS server and in the local one I can keep all the snapshots I want because the sync job only will transfer snapshots (and their chunks) newer than the last local one.

Is it correct?

 

[UdoB]

I guess you can use a sync job to copy the data to the remote host (instead of a second backup).

Just for my personal understanding as this seems to be important to me: running Backups from PVE to PBS server A today and to server B tomorrow (and so on, alternating) will lose the benefits of the dirty-bitmap-mechanism?

Best regards

 

[SOLTECSIS - Carles Munyoz]

Just for my personal understanding as this seems to be important to me: running Backups from PVE to PBS server A today and to server B tomorrow (and so on, alternating) will lose the benefits of the dirty-bitmap-mechanism?

Best regards

Correct.
I think that the Proxmox team is making changes for allow several bitmaps for avoid this problem, but I don't know the status of this development.
Perhaps some of the Proxmox staff members may inform us about it.

 

[UdoB]

Correct.

Thanks for the confirmation.
I've just taken a look into my task-logs. Should have done that earlier...

Best regards

 

[SOLTECSIS - Carles Munyoz]

You are welcome.
I think that the best and simplest solution is the multiple dirty-bitmaps, but meanwhile the solution I have explained can be valid.
But I would like that the Proxmox staff confirms that it is right.

 

[fabian]

Then the solution is:

• Setup the Proxmox/Ceph cluster to make backups over the remote PBS, the one with less disk space.
• In the remote PBS keep only the last 15 snapshots.
• Setup a sync job in the local PBS for sync the datastore from the remote PBS.
• In the local PBS setup the prune policy for keep the last 30 snapshots, 24 weekly and 12 monthly.

This way I will not lost the dirty-map because the backups will be done always over the same PBS server and in the local one I can keep all the snapshots I want because the sync job only will transfer snapshots (and their chunks) newer than the last local one.

Is it correct?

that works - how much of a tradeoff it is depends on whether the connection from PVE to remote PBS is much slower than from PVE to local PBS (the backup is more latency and bandwidth sensitive than the sync, since the former will slow down I/O in the guest, the latter won't)

 

[fabian]

multiple dirty bitmaps are not free either unfortunately. in almost all cases using one PBS backup target and syncing from there is the better solution.

 

[SOLTECSIS - Carles Munyoz]

Conclusion: The optimal solution for our scenario is:

• Setup the Proxmox/Ceph cluster to make backups over the LOCAL PBS.
• In the LOCAL PBS setup the prune policy for keep the last 30 snapshots, 24 weekly and 12 monthly.
• Setup a sync job in the REMOTE PBS for sync the datastore from the LOCAL PBS. In this sync only snapshots (and their chunks) which are newer than the last one in the REMOTE PBS are pulled from the LOCAL PBS.
• In the REMOTE PBS keep only the last 15 snapshots and run garbage collector every day for remove old data.

Is it correct?

 

[fabian]

yes

 

[SOLTECSIS - Carles Munyoz]

Ok, great.
Thank you very much for your help.