I use both PBS and restic for backup on different systems. I'm very happy with both tools for their job. In the restic documentation I read about a possible threat and wonder how it is handles in PBS. I have not found any mention of the threat or a mitigation in the documentation of PBS.
Attack
I would maybe name the attack "backup stuffing" or "pruning side channel". Usually you want to make sure, that the backup source can not delete the backups on the destination. Because, the source system can not only die, but could also be compromised by a potential intelligent adversary.
The usual way to prevent this is to give the backup source only permission to create new backups and not to delete backups.
In order to still be able to delete old backups and enforce a backup retention, the backup destination now has to delete backups. This is done based on pruning settings. These pruning settings are used in a pruning operation on the backup destination to decide which backups should remain and which should be deleted.
For this decision, the number of backups and the timestamp of the backups is used. And this is the information that is manipulated in this attack.
When a pruning setting of keep-last: X is set, the adversary can just create X new trash backups (without useable data). Then the pruning on the destination will delete the old good backups and our data is gone.
Pruning settings of keep-weekly (and similar) are also exploitable under the following assumption: The backup source can set arbitrary timestamps to the backups.
When the adversary on the backup source creates a trash backup with a slightly newer timestamp for each good backup, the pruning will delete the good backups.
Restic's mitigation to this problem is the keep-within option.
It prevents the deletion of any backups that are younger than X days (compared to the current time of the pruning system, not the time of the "newest" backup, since that could also be faked to the future). So you have X days to notice the compromise and in that time, no backup is deleted.
See their docs also for their description of the threat: https://restic.readthedocs.io/en/v0.16.3/060_forget.html#security-considerations-in-append-only-mode
(It should also be noted that restic, despite sharing great characteristics, is different to PBS. In restic the backup destination is just a blob storage and has no possibility to remember when a backup was actually written to it. PBS could do that, since it has a service at the destination.)
Exploitation scenarios
For PBS I see two possible source-destination scenario where this could be exploited:
1. proxmox-backup-client to PBS
In order to be able to perform the described attack, a malicious proxmox-backup-client must be able to create backups with arbitrary timestamp.
I found nothing in the documentation that says this isn't possible. However, the description of the backup protocol API suggest to me that it is not possible. But let assume it for this scenario.
(Knowledge about the pruning settings is, to some extend, could also be needed for the adversary. However, within the meaning of Kerckhoffs's principle, the knowledge must be assumed.)
There is a proxmox-backup-client running on a machine, let's call it A. There is a separate machine running a PBS, called B.
A has access to B with the DatastoreBackup role in order to create daily backups. B has the prune settings "keep-weekly: 4" installed. They have run for some time.
A is now completely compromised by an intelligent adversary and has therefore DatastoreBackup access to B. B is otherwise not compromised.
A creates four trash backups with timestamps that are slightly younger than the existing ones. On the next prune and GC job, our good (from before the attack on A) backups are gone.
2. PBS to PBS via sync job
In order to be able to perform the described attack, a malicious PBS must be able to create backups with arbitrary timestamps in its own datastore and a pull jobs will pull those potential old backups.
I see no reason why this should not be possible. The sync job is designed to sync two datastores and therefore also pulls older backups.
We have a backup source (may it be again a proxmox-backup-client), called A and a PBS (e.g. in the local network, maybe under the same SSO), called B. Machine A backups its data to B.
We have another PBS, called C, that is e.g. located offsite and is as much as possible isolated from B. On C there is a sync job configured to pull the backups regularly (sensible, without remove-vanished set). Also, on C there are pruning settings set (e.g. similar to the former scenario).
Machines A and B are now both completely compromised (e.g. compromise of SSO), again by an intelligent adversary. C is not compromised.
B deletes the old backups and creates trash backups with slightly newer timestamps. C will pull those backups and later prune the good backups.
Questions
Is the assumption in scenario 1 correct/feasible?
How about scenario 2?
Are there mitigation to this attack?
Mitigation Ideas
I have two ideas for mitigate this attack:
Receiver-side generation of timestamps. The PBS generates the timestamp of a backup on its own during receiving. These timestamps are used to decide when to prune a backup. However on a sync job, this is a bit more tricky. (This is the mitigation restic can't use, as the destination is just a dumb blob storage, as mentioned above)
Sender-side definition of pruning/backup retention: The source of the backup saves in each backup the timestamp until the backup has to be kept. Now, malicious backups can't use the side channel of pruning to delete other backups. This comes with other complications in management, configuration etc.
Attack
I would maybe name the attack "backup stuffing" or "pruning side channel". Usually you want to make sure, that the backup source can not delete the backups on the destination. Because, the source system can not only die, but could also be compromised by a potential intelligent adversary.
The usual way to prevent this is to give the backup source only permission to create new backups and not to delete backups.
In order to still be able to delete old backups and enforce a backup retention, the backup destination now has to delete backups. This is done based on pruning settings. These pruning settings are used in a pruning operation on the backup destination to decide which backups should remain and which should be deleted.
For this decision, the number of backups and the timestamp of the backups is used. And this is the information that is manipulated in this attack.
When a pruning setting of keep-last: X is set, the adversary can just create X new trash backups (without useable data). Then the pruning on the destination will delete the old good backups and our data is gone.
Pruning settings of keep-weekly (and similar) are also exploitable under the following assumption: The backup source can set arbitrary timestamps to the backups.
When the adversary on the backup source creates a trash backup with a slightly newer timestamp for each good backup, the pruning will delete the good backups.
Restic's mitigation to this problem is the keep-within option.
It prevents the deletion of any backups that are younger than X days (compared to the current time of the pruning system, not the time of the "newest" backup, since that could also be faked to the future). So you have X days to notice the compromise and in that time, no backup is deleted.
See their docs also for their description of the threat: https://restic.readthedocs.io/en/v0.16.3/060_forget.html#security-considerations-in-append-only-mode
(It should also be noted that restic, despite sharing great characteristics, is different to PBS. In restic the backup destination is just a blob storage and has no possibility to remember when a backup was actually written to it. PBS could do that, since it has a service at the destination.)
Exploitation scenarios
For PBS I see two possible source-destination scenario where this could be exploited:
1. proxmox-backup-client to PBS
In order to be able to perform the described attack, a malicious proxmox-backup-client must be able to create backups with arbitrary timestamp.
I found nothing in the documentation that says this isn't possible. However, the description of the backup protocol API suggest to me that it is not possible. But let assume it for this scenario.
(Knowledge about the pruning settings is, to some extend, could also be needed for the adversary. However, within the meaning of Kerckhoffs's principle, the knowledge must be assumed.)
There is a proxmox-backup-client running on a machine, let's call it A. There is a separate machine running a PBS, called B.
A has access to B with the DatastoreBackup role in order to create daily backups. B has the prune settings "keep-weekly: 4" installed. They have run for some time.
A is now completely compromised by an intelligent adversary and has therefore DatastoreBackup access to B. B is otherwise not compromised.
A creates four trash backups with timestamps that are slightly younger than the existing ones. On the next prune and GC job, our good (from before the attack on A) backups are gone.
2. PBS to PBS via sync job
In order to be able to perform the described attack, a malicious PBS must be able to create backups with arbitrary timestamps in its own datastore and a pull jobs will pull those potential old backups.
I see no reason why this should not be possible. The sync job is designed to sync two datastores and therefore also pulls older backups.
We have a backup source (may it be again a proxmox-backup-client), called A and a PBS (e.g. in the local network, maybe under the same SSO), called B. Machine A backups its data to B.
We have another PBS, called C, that is e.g. located offsite and is as much as possible isolated from B. On C there is a sync job configured to pull the backups regularly (sensible, without remove-vanished set). Also, on C there are pruning settings set (e.g. similar to the former scenario).
Machines A and B are now both completely compromised (e.g. compromise of SSO), again by an intelligent adversary. C is not compromised.
B deletes the old backups and creates trash backups with slightly newer timestamps. C will pull those backups and later prune the good backups.
Questions
Is the assumption in scenario 1 correct/feasible?
How about scenario 2?
Are there mitigation to this attack?
Mitigation Ideas
I have two ideas for mitigate this attack:
Receiver-side generation of timestamps. The PBS generates the timestamp of a backup on its own during receiving. These timestamps are used to decide when to prune a backup. However on a sync job, this is a bit more tricky. (This is the mitigation restic can't use, as the destination is just a dumb blob storage, as mentioned above)
Sender-side definition of pruning/backup retention: The source of the backup saves in each backup the timestamp until the backup has to be kept. Now, malicious backups can't use the side channel of pruning to delete other backups. This comes with other complications in management, configuration etc.
) - if you think you find an issue that might have security implications, it would be appreciated to use the process outlined in