The situation is that because of unforeseen events the VM got deleted from prox cluster and so did the backups from PBS.
Which kind of unforeseen events? Ransomware or human error? You can't do much against human error but you can at least limit it (together with ransomware) if you set your permissions that you can only remove backups on PBS from the PBS UI with according permissions or a prune job and (important) don't activate the "remote vanished" option in PBS (so removed backups on the PVE won't be removed in the PBS until triggered by manual removal or a prune job):
https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery
https://pbs.proxmox.com/docs/managing-remotes.html#sync-jobs
In both cases however you shouldn't have just one backup but more fitting the 3-2-1 rule: "In short, the rule states that one should create
3 backups on at least
2 different types of storage media, of which
1 copy is kept off-site." (
https://pbs.proxmox.com/docs/storage.html#the-3-2-1-rule-with-proxmox-backup-server )
Storage is CEPH Proxmox 8 latest , and PBS ZFS store (prune job has not been run).
Are you sure about that? The prune jobs are the thing (together with manual remove actions) which remove the snapshots (references to the data). The garbage collection is the task which will actual remove the data (anything not referenced in a snapshots for more than 24h+5m). It doesn't matter in your case anyhow: If the snapshots you are missing were removed, they can't be reconstructed.
Can anyone help me restore backup from PBS and is that even possible?
If you have no copies anywhere else: No. PBS splits the backups data in a lot of small files called chunks. These chunks are referenced in the snapshot so if you have a backup of lets say VM 420 it just contains the configuration of VM420 plus the references to said chunks. Now if this snapshot was removed how should anybody be able to reconstruct them? He would need to go all of them, uncompress and decrypt them (if you encrypted them) then figure out how everything fits together. This isn't feasible on the scale we are talking here.
Just to have an example, this is from the log of the latest GC job on my PBS (around 294 GB raw data on the datastore):
Code:
2025-09-01T02:00:19+02:00: Chunk cache: hits 35799635, misses 262321 (hit ratio 99.27%)
2025-09-01T02:00:19+02:00: Removed garbage: 4.143 GiB
2025-09-01T02:00:19+02:00: Removed chunks: 6865
2025-09-01T02:00:19+02:00: Pending removals: 2.977 GiB (in 5756 chunks)
2025-09-01T02:00:19+02:00: Original data usage: 136.036 TiB
2025-09-01T02:00:19+02:00: On-Disk usage: 291.077 GiB (0.21%)
2025-09-01T02:00:19+02:00: On-Disk chunks: 262322
2025-09-01T02:00:19+02:00: Deduplication factor: 478.57
2025-09-01T02:00:19+02:00: Average chunk size: 1.136 MiB
2025-09-01T02:00:19+02:00: queued notification (id=e7a941f6-3201-44d5-a79b-7e51c10ccc33)
2025-09-01T02:00:19+02:00: TASK OK
So let's say I miss one backup and would want to try to put it together from the chunks, then I would need to go through 262322 files at the size of 1,2 MB. This just isn't feasible, here is a listing of some of the files inside one of the folders inside the chunks storage folder in the datastore:
Code:
/mnt/datastore/local/.chunks/47bd# ls -lh
total 11M
-rw-r--r-- 1 backup backup 2.0M Apr 29 22:15 47bd10fbb4e076256b40b483f26c6c9effcc3ed66565c4daa58c1e073670cc72
-rw-r--r-- 1 backup backup 1.7M Feb 26 2025 47bd71bbe20d22c0cc1e36048df04224740290ddd5e27ae1f2e9c331c1a66c7b
-rw-r--r-- 1 backup backup 3.1M Jan 31 2025 47bd768461f20e2dde8af55acba87e9ccc5df957ce5395616044456f44eef629
-rw-r--r-- 1 backup backup 2.2M Apr 9 02:15 47bdd6277be5fa2cb96c03988481ee384f4bdd0f827dd1b18e1287e27aa96f7e
-rw-r--r-- 1 backup backup 1.8M Nov 28 2024 47bdd8ff15cf657405802db06d033ffc2be0d154ba021426051f5375b9bdb836
-rw-r--r-- 1 backup backup 20K Nov 25 2024 47bddf081d82c787139384970dfb16869e55fbe11e878b4f440bbfa86dc6b0a2
However: Did you happen to have the datastore on zfs and something like zfs-auto-snapshots or manual snapshots of it? Then there might be a chance to restore the contents of the datastore from a snapshot to a new datastore, add this restored datastore on the PBS and get it again that way.
For the future I advise to have a second PBS on an offsite location like a friends place, a cheap virtualserver or a cloud PBS provider like tuxis.nl, Inett and friends. Additionally you could use an external USB disc as removable datastore on the PBS and a backup storage of the native backup files (who doesn't need PBS but also need more storage space) of ProxmoxVE. The idea would be to not attach the USB disk most of the time but just add it from time to time so you have another (although older) copy which can't be altered from the PC (so neither by an attacker or human error), you could even store it outside of your place (again friends or family member place) so you have still an older copy in case of an desaster.
