GC failed with TASK ERROR

[villain666]

Garbage collect failed with this error again and again

TASK ERROR: unable to access chunk 2d3470d651e623c96fcdc0c41b0974736561c69125da3435ec34aa77cbb58406, required by "/backup/pbs/vm/110/2020-07-26T19:10:58Z/drive-scsi0.img.fidx" - update atime failed for chunk "/backup/pbs/.chunks/2d34/2d3470d651e623c96fcdc0c41b0974736561c69125da3435ec34aa77cbb58406" - ENOENT: No such file or directory

Forget backup - failed on other backup for this VM

how to repair?

 

[villain666]

When i forget backup ...110/2020-07-26T19:10:58Z - error exist on other backup with this chuck

My fix
Find first not affected backup (use verify) and remove all next affected backup index, restart GC and full backup for vm (run automatically on backup job, don`t use dirty-bitmap)

Verify slow (maybe create "quick verify" - check only exists chuck file and not size zero?)

 

[dcsapak]

the question is why is the chunk missing?

Verify slow (maybe create "quick verify" - check only exists chuck file and not size zero?)

this should not be necessary, since chunks should not vanish ....

 

[villain666]

no, chuck can be missing by different reason, there no question.

Question about how to fix GC, which stop after this error

 

[dcsapak]

no, chuck can be missing by different reason, there no question.

pbs only deletes chunks on garbage collection, so i think that should not happen normally (aside from fs errors, in which case you probably
do not want to delete more chunks and move away from the hardware asap)

 

[villain666]

I know, but i think fs error on heave load in this case. My test environment - unisocket Xeon E3, 16 Gb RAM, 13 VMs (1 Windows Server + 12 Linux VMs)
PBS setup as VM + datastor disk place (as vm image) on NFS share (small Synology NAS) + backup every 10 min

 

[villain666]

Question about GC - in this situation (broken fs) GC stop check with ERROR and restart on next minute - my PBS VM slowdown
(GC work all time, failed and start again). Maybe config options for restart/notrestart failed GC and email to admins?

 

[grin]

Same here: a chunk is missing (disappeared) and it's killing GC. I am still looking for how to fix, apart from extremely radical measures.

 

[t.lamprecht]

Same here: a chunk is missing (disappeared) and it's killing GC. I am still looking for how to fix, apart from extremely radical measures.

There has been a fix for this issue: https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=2f57a433b1f333f96e4ed6dc71b80e2575d6b453
which is publicly available since a few days in version 0.8.11, so please update to that and retry.

Also, while we definitively appreciate you trying out PBS and giving feedback, it does not help to post the same thing over and over in different threads, especially older ones - which get quickly outdated in a fast pacing beta.

 

[grin]

Also, while we definitively appreciate you trying out PBS and giving feedback, it does not help to post the same thing over and over in different threads, especially older ones - which get quickly outdated in a fast pacing beta.

This was not intentional, and if you look at them you may have noticed that progressively more and more information was revealed about the problem and the reason while the same time I have tried to figure out _where_ it should have been posted, and the same time realised that posts cannot be moved into the proper thread (or if they can, I don't see the means).

It is not obvious, still, how you _expect_ me to provide feedback and where. If you would be so kind as to point me to the right direction, or even if you'd mention in the first announcement post of pbs then it would greatly ease our commuication (where "we" are the userbase).

There has been a fix for this issue: https://git.proxmox.com/?p=proxmox-backup.git;a=commit;h=2f57a433b1f333f96e4ed6dc71b80e2575d6b453
which is publicly available since a few days in version 0.8.11, so please update to that and retry.

Thanks, and in fact I have realised that yesterday and upgraded already. The problem has been partially fixed: GC now doesn't spam the log, only logs once:

2020-08-25T00:00:00+02:00: TASK ERROR: unable to get exclusive lock - EAGAIN: Try again
for all Datastores.

I wonder whether this'll change in days or if it stays in that state.

 

[t.lamprecht]

This was not intentional, and if you look at them you may have noticed that progressively more and more information was revealed about the problem and the reason while the same time I have tried to figure out _where_ it should have been posted, and the same time realised that posts cannot be moved into the proper thread (or if they can, I don't see the means).

You can just edit your posts, no need to do new ones just for enhancing provided information.

It is not obvious, still, how you _expect_ me to provide feedback and where. If you would be so kind as to point me to the right direction, or even if you'd mention in the first announcement post of pbs then it would greatly ease our commuication (where "we" are the userbase).

If in doubt, just open a new thread, as always in our forum. Jumping on older threads should not be done normally, they are outdated most of the time and even if they describe similar or the same symptoms it may be a completely different cause.
You can naturally reference other threads, if you think they're relevant.

The problem has been partially fixed: GC now doesn't spam the log, only logs once:

2020-08-25T00:00:00+02:00: TASK ERROR: unable to get exclusive lock - EAGAIN: Try again
for all Datastores.

I wonder whether this'll change in days or if it stays in that state.

This seems like a completly unrelated issue, is there a GC already running? It at least seems like there's a process holding the lock actively, as else it'd time out.

 

[grin]

This seems like a completly unrelated issue, is there a GC already running? It at least seems like there's a process holding the lock actively, as else it'd time out.

No, GC haven't been able to run the last week or so.
Only backups are running continuously (I yet to see why some of the packups run 20+ hours).

However right now proxmox-backup-manager task list says only one backup job is running, and indeed now GC was able to run. So far it have generated ab out 200MB logs (417000 warnings) about missing chunks. But that's another problem, and I'll open another thread.

Thank you.