[SOLVED] bk stucked in verification

[filippoclikkami]

Hi, these days verify job got a problem with earliest backup of the set, the one that have to be deleted. Prune job is configured to maintain the 7 latest backups, it seems that it can't delete the first one and got stucked in the verification process.

I tried to prune the first one manually, run a garbage collection and reverify, everything goes right but the day after problem come back

 

[fabian]

please post the contents of the last verify (of that group/snapshot) and GC task log..

 

[filippoclikkami]

Thank you for the quick reply, here you are

 

[fabian]

did you maybe manually try to remove that snapshot somehow? it seems a bit weird since it is the oldest, but it is in an "incomplete" state.

what does

"ls -lha /mnt/pbsbackup/vm/101/2024-03-30T20:00:02Z/"

output?

 

[filippoclikkami]

here the result:
root@pbsk2a:~# ls -lha /mnt/pbsbackup/vm/101/2024-04-01T19:00:02Z/
total 0
drwxr-xr-x 1 backup backup 0 Apr 10 00:11 .
drwxr-xr-x 1 backup backup 330 Apr 10 00:08 ..

yesterday snap is no longer avaible, the problem still reamin on oldest one

 

[fabian]

remove the empty dir, then it is gone.. I am not aware of any other reports of PBS leaving empty directories around, is it possible you tried to clear that snapshot manually?

 

[filippoclikkami]

I think it may be a power loss during the backup process. I have deleted the snap everywhere, I 'll fill you, thanks for now

 

[fabian]

yeah, that is possible (the dir has to be created before files in it can be, so if a powerloss/crash happens at a bad time then you might end up with just the empty dir).

 

[filippoclikkami]

Hi Fabian, here we go again, the oldest one is stucked there

 

[fabian]

is there some other task that is also started at midnight? or shortly before and still running?

 

[filippoclikkami]

Prune and GC starts at midnight as default, Verify starts at 6:00AM, everything here

 

[fabian]

GC doesn't lock the snapshot, but it does read its index, it's possible there is a race there. does the problem go away if you schedule the GC a few minutes later? pruning should be pretty much instant..

 

[filippoclikkami]

I'll try to set up now, i'll fill you

 

[filippoclikkami]

I don't know how but everything seems to be back to normal, at least for today. If it will be the same tomorrow, we got it

 

[filippoclikkami]

Hi fabian, here to inform you that the 10 minutes delay for garbage collection worked, now everything is going well.
Can you help me whit this? https://forum.proxmox.com/threads/backup-and-restore-cluster-and-ceph.145282/
Thank you so much