Weird prune retention problem on S3 sync job

[w3bdevil]

Hi,

I have a strange problem with backup retention. My regular backups to local datastore work fine but I'm also syncing backups to a S3 datastore and the problem there is monthly backup retention. My local retention is to keep 7 dailies, 4 weeklies and 6 monthlies but to save a bit on cloud costs the retention settings for S3 datastore prune are only 3 dailies and 6 monthlies. I tested this on the simulator and it shows the result I'm expecting. However, looks like no monthlies at all are kept in the S3 datastore. Looking at prune logs I noticed something strange: it looks like it is actually initially keeping the monthly but then after a few days it just vanishes without a trace.

Here are the prune logs at the turn of the month for one of my VMs. On the 4th day of the month it correctly keeps the backup of the last day of last month. This shows up in the logs for a few days as expected (keep vm/104/2026-04-30T20:30:42Z) but then suddenly on the 8th day of the month it has disappeared and there is no trace in the logs of it being removed. What could be happening here? Can it be a timing issue? Why is there no log of the monthly being removed?

- regular backups to local datastore run every day at 23:30
- verify job runs at 00:00
- local sync job pulling the latest verified backups from my local datastore to S3 datastore runs at 01:45
- prune for S3 datastore runs at 02:00, retention options: --keep-daily 3 --keep-monthly 6

2026-05-03T02:00:00+03:00: Pruning group :"vm/104"
2026-05-03T02:00:00+03:00: remove vm/104/2026-04-29T20:32:32Z
2026-05-03T02:00:00+03:00: removing backup snapshot "/mnt/s3cache/vm/104/2026-04-29T20:32:32Z"
2026-05-03T02:00:00+03:00: keep vm/104/2026-04-30T20:30:42Z
2026-05-03T02:00:00+03:00: keep vm/104/2026-05-01T20:30:35Z
2026-05-03T02:00:00+03:00: keep vm/104/2026-05-02T20:30:38Z

2026-05-04T02:00:00+03:00: Pruning group :"vm/104"
2026-05-04T02:00:00+03:00: keep vm/104/2026-04-30T20:30:42Z  <-- keeping the last of the month as expected
2026-05-04T02:00:00+03:00: keep vm/104/2026-05-01T20:30:35Z
2026-05-04T02:00:00+03:00: keep vm/104/2026-05-02T20:30:38Z
2026-05-04T02:00:00+03:00: keep vm/104/2026-05-03T20:30:33Z

2026-05-05T02:00:00+03:00: Pruning group :"vm/104"
2026-05-05T02:00:00+03:00: keep vm/104/2026-04-30T20:30:42Z  <-- still here
2026-05-05T02:00:00+03:00: remove vm/104/2026-05-01T20:30:35Z
2026-05-05T02:00:00+03:00: removing backup snapshot "/mnt/s3cache/vm/104/2026-05-01T20:30:35Z"
2026-05-05T02:00:00+03:00: keep vm/104/2026-05-02T20:30:38Z
2026-05-05T02:00:00+03:00: keep vm/104/2026-05-03T20:30:33Z
2026-05-05T02:00:00+03:00: keep vm/104/2026-05-04T20:30:38Z

2026-05-06T02:00:00+03:00: Pruning group :"vm/104"
2026-05-06T02:00:00+03:00: keep vm/104/2026-04-30T20:30:42Z  <-- still here
2026-05-06T02:00:00+03:00: remove vm/104/2026-05-02T20:30:38Z
2026-05-06T02:00:00+03:00: removing backup snapshot "/mnt/s3cache/vm/104/2026-05-02T20:30:38Z"
2026-05-06T02:00:00+03:00: keep vm/104/2026-05-03T20:30:33Z
2026-05-06T02:00:00+03:00: keep vm/104/2026-05-04T20:30:38Z
2026-05-06T02:00:00+03:00: keep vm/104/2026-05-05T20:30:27Z

2026-05-07T02:00:00+03:00: Pruning group :"vm/104"
2026-05-07T02:00:00+03:00: keep vm/104/2026-04-30T20:30:42Z  <-- still here
2026-05-07T02:00:00+03:00: remove vm/104/2026-05-03T20:30:33Z
2026-05-07T02:00:00+03:00: removing backup snapshot "/mnt/s3cache/vm/104/2026-05-03T20:30:33Z"
2026-05-07T02:00:00+03:00: keep vm/104/2026-05-04T20:30:38Z
2026-05-07T02:00:00+03:00: keep vm/104/2026-05-05T20:30:27Z
2026-05-07T02:00:00+03:00: keep vm/104/2026-05-06T20:30:34Z

-- what happens here and why?
-- now the monthly is suddenly gone with no trace in logs of it being removed

2026-05-08T02:00:00+03:00: Pruning group :"vm/104"
2026-05-08T02:00:00+03:00: remove vm/104/2026-05-04T20:30:38Z
2026-05-08T02:00:00+03:00: removing backup snapshot "/mnt/s3cache/vm/104/2026-05-04T20:30:38Z"
2026-05-08T02:00:00+03:00: keep vm/104/2026-05-05T20:30:27Z
2026-05-08T02:00:00+03:00: keep vm/104/2026-05-06T20:30:34Z
2026-05-08T02:00:00+03:00: keep vm/104/2026-05-07T20:30:44Z

 

[Chris]

I have a strange problem with backup retention. My regular backups to local datastore work fine but I'm also syncing backups to a S3 datastore and the problem there is monthly backup retention

When are the sync jobs being performed? Before or after the pruning on the local datastore? Does it include a remove vanished? Check if the snapshot has even been synced to the S3 store.

Further, can you check if the snapshot is still present on the object store?

 

[w3bdevil]

When are the sync jobs being performed? Before or after the pruning on the local datastore? Does it include a remove vanished? Check if the snapshot has even been synced to the S3 store.

Further, can you check if the snapshot is still present on the object store?

Sync job runs at 01:45, after the regular backups have been run and verified. Local datastore prune has already been run at that point.

I think I might have found the reason. Latest sync job has a row:

2026-05-08T01:45:53+03:00: [vm/104]: delete vanished snapshot vm/104/2026-04-30T20:30:42Z

Turns out I had the option "Remove vanished" enabled on the sync job and the S3 datastore prune job picks different snapshots as monthlies than my local datastore, which means that once those were pruned from local it propagated to S3 also. This is probably because I have the keep-weekly option in my local retention rules.

Thanks for pointing me to the right direction!

 

[Chris]

Turns out I had the option "Remove vanished" enabled on the sync job and the S3 datastore prune job picks different snapshots as monthlies than my local datastore, which means that once those were pruned from local it propagated to S3 also. This is probably because I have the keep-weekly option in my local retention rules.

Why even enabling the remove vanished in this case? Would probably be better to only use the prune job to remove the snapshots.

Thanks for pointing me to the right direction!

Glad you found the culprit rather easily!