Two PBS Servers, one syncs from the other - Do I Prune and GC on both?

[helojunkie]

As the title says, I have two PBS servers and one (the backup unit) pulls a sync from the primary. I have my primary doing pruning and garbage collections already, do I also need to do that on the backup server that is pulling the sync or will it automatically prune based on the sync?

I assume I would need to do garbage collection on both, but not sure about the pruning.

Thank You

 

[Dunuin]

As the title says, I have two PBS servers and one (the backup unit) pulls a sync from the primary. I have my primary doing pruning and garbage collections already, do I also need to do that on the backup server that is pulling the sync or will it automatically prune based on the sync?

Depends if you enable "removed-vanished" for the sync job or not. But keep in mind that when enabling that it won't be great for ransomware protection.

 

[helojunkie]

Thanks, Dunuin; by Ransomware protection, are you referring to my PBS primary getting compromised? I have on system backups and snaps now, and I have my primary PBS doing routine backups all day (not a lot of VMs/CTs, 100 or less) for my case, keeping 5 last, 48 hourly, 13 daily, 24 weekly, 24 monthly and 5 yearly. In my mind, maybe incorrectly, I thought that was an effective way to manage risk, but I wanted to add one more copy (the golden 3-2-1) by adding a third backup.

My primary servers are all 100% NVMe-backed local storage. My Primary PBS is 100% NVMe-backed storage, and my Secondary is spinning rust.

From your comment, it seems to me that you are saying never to prune on the secondary as a further mitigation against ransomware, or am I just misreading that? I had not really heard that or thought about it in that sense, so I would enjoy hearing more of your thoughts on it.

Thank you for taking the time to share.

On another note, I did not have 'remove-vanished' selected, so I assume I can just go ahead and hit that and it will remove the pruned backups off the primary? I also assume I would still need a GC event on my secondary, correct?

And a quick edit, the secondary PBS server that syncs with my primary is hosted on my primary TrueNAS Scale server. That server is also backed up to a secondary TrueNAS Scale server. So I guess I really have four backups.

 

[Dunuin]

by Ransomware protection, are you referring to my PBS primary getting compromised? I have on system backups and snaps now, and I have my primary PBS doing routine backups all day (not a lot of VMs/CTs, 100 or less) for my case, keeping 5 last, 48 hourly, 13 daily, 24 weekly, 24 monthly and 5 yearly. In my mind, maybe incorrectly, I thought that was an effective way to manage risk, but I wanted to add one more copy (the golden 3-2-1) by adding a third backup.

Yes. But not only the primary PBS but also the PVE. Examples:
A.) You add a PBS datastore to PVE with privileges that allow PVE to prune any backup snapshot on the datastore. The compromised PVE will prune all backup snapshots on the primary PBS and the sync will then also prune all backup snapshots on the secondary PBS even if the PVE got no access to the secondary PBS at all (or is limited to restore-only privileges).
B.) You don't allow PVE to prune backup snapshots but instead of using "houry" and "daily" you use something like 20 last, 12 weekly, 12 monthly. The compromised PVE will then encrypt your guests and create 20 new backups overwriting the 20 last healty backups. Once the PBSs prune and sync job has finished all recent backup snapshots on both PBSs will be lost and all whats left are some outdated weekly and monthly backups.
c.) human error and some admin manually prunes the wrong backups on the primary PBSs webUI by accident. You won't believe how many people here ask if it is possible to undelete an accidentally pruned backup in case the GC hasn't run yet...and the answer is no...if there is no sync to another PBS or some automated ZFS snapshots. Would be bad if the sync job then also would automatically destroy that backup on the secondary PBS.

From your comment, it seems to me that you are saying never to prune on the secondary as a further mitigation against ransomware, or am I just misreading that? I had not really heard that or thought about it in that sense, so I would enjoy hearing more of your thoughts on it.

Pruning is fine. But I would prefer if only the secondary PBS would be allowed to prune itself. Without the compromised PVEs or compromised primary PBS being able to destroy any backups on the secondary PBS. Also useful in case you want different backup retentions. Like a fast but small primary PBS that only keeps the last few weeks for very fast restores of recent backups and a bigger but slower secondary PBS that stores the backups for years for long-term archival.
If you want both to store the same you could set up a prune task on the secondary PBS using "keeping 5 last, 48 hourly, 13 daily, 24 weekly, 24 monthly and 5 yearly" too. Then both PBSs should store the same backup snapshots and in case the primary PBS gets compromised and all backup snapshots deleted, the secondary won't be affected and after wiping the primary PBS you could sync all the backups snapshots back.

On another note, I did not have 'remove-vanished' selected, so I assume I can just go ahead and hit that and it will remove the pruned backups off the primary?

Yes.

I also assume I would still need a GC event on my secondary, correct?

Yes, GC and (Re-)Verify to actually free up space and make sure backups won't silently corrupt.

 

[helojunkie]

Dunuin - wow, thank you for that great explanation. That is very well thought out. I did not even consider the threat vector PVE to PBS, but it makes perfect sense that that could, in fact, happen should the PVE become compromised.

So the best way would be to create a user on PBS that has only the ability to write to it but nothing else, backup using that user so if the PVEs become compromised, they can do a little damage locally and no damage to the PBS itself. And then continue to do the same thing with the PBS servers - My backup PBS server should be able to read from my primary but not prune, and my primary should not have any access at all to my backup. Do 100% of my pruning at each local PBS.

 

[dietmar]

For Ransomware protection, Tape backup is also a good idea..

 

[Dunuin]

So the best way would be to create a user on PBS that has only the ability to write to it but nothing else, backup using that user so if the PVEs become compromised, they can do a little damage locally and no damage to the PBS itself. And then continue to do the same thing with the PBS servers - My backup PBS server should be able to read from my primary but not prune, and my primary should not have any access at all to my backup. Do 100% of my pruning at each local PBS.

Yes. So PVE should be only allowed to backup and restore to/from primary PBS, no pruning and only allowed to restore from secondary PBS. Primary PBS prunes itself. Secondary PBS prunes itself too and pulls backups from the primary PBS without trusting the primary PBS or PVEs.

There is also a chapter in the documentation: https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery