Garbage Collection is interrupted due to no empty space

[karpuzvepeynir]

Hi,

My PBS has no empty space. Garbage Collection process cannot be finished, because no space left.

Here is the log:

2021-11-21T07:15:00+03:00: starting garbage collection on store pb-60-20-110
2021-11-21T07:15:00+03:00: task triggered by schedule 'sun 07:15'
2021-11-21T07:15:00+03:00: Start GC phase1 (mark used chunks)
2021-11-21T08:35:05+03:00: marked 1% (12 of 1170 index files)
2021-11-21T08:46:55+03:00: marked 2% (24 of 1170 index files)
2021-11-21T08:52:48+03:00: marked 3% (36 of 1170 index files)
2021-11-21T08:53:15+03:00: marked 4% (47 of 1170 index files)
2021-11-21T08:54:07+03:00: marked 5% (59 of 1170 index files)
2021-11-21T08:54:23+03:00: marked 6% (71 of 1170 index files)
2021-11-21T08:55:22+03:00: marked 7% (82 of 1170 index files)
2021-11-21T08:55:58+03:00: marked 8% (94 of 1170 index files)
2021-11-21T08:56:06+03:00: marked 9% (106 of 1170 index files)
2021-11-21T11:42:22+03:00: marked 10% (117 of 1170 index files)
2021-11-21T12:22:02+03:00: marked 11% (129 of 1170 index files)
2021-11-21T12:32:00+03:00: marked 12% (141 of 1170 index files)
2021-11-21T12:32:59+03:00: marked 13% (153 of 1170 index files)
2021-11-21T12:33:05+03:00: marked 14% (164 of 1170 index files)
2021-11-21T12:33:12+03:00: marked 15% (176 of 1170 index files)
2021-11-21T12:33:30+03:00: marked 16% (188 of 1170 index files)
2021-11-21T12:33:33+03:00: marked 17% (199 of 1170 index files)
2021-11-21T12:54:06+03:00: marked 18% (211 of 1170 index files)
2021-11-21T22:31:21+03:00: marked 19% (223 of 1170 index files)
2021-11-22T14:06:11+03:00: marked 20% (234 of 1170 index files)
2021-11-22T14:23:22+03:00: marked 21% (246 of 1170 index files)
2021-11-22T14:45:56+03:00: marked 22% (258 of 1170 index files)
2021-11-22T15:06:00+03:00: marked 23% (270 of 1170 index files)
2021-11-22T15:10:20+03:00: marked 24% (281 of 1170 index files)
2021-11-22T15:22:25+03:00: marked 25% (293 of 1170 index files)
2021-11-22T15:41:16+03:00: marked 26% (305 of 1170 index files)
2021-11-22T18:20:34+03:00: marked 27% (316 of 1170 index files)
2021-11-22T19:29:06+03:00: marked 28% (328 of 1170 index files)
2021-11-22T19:44:15+03:00: marked 29% (340 of 1170 index files)
2021-11-22T20:47:55+03:00: marked 30% (351 of 1170 index files)
2021-11-22T20:53:44+03:00: marked 31% (363 of 1170 index files)
2021-11-22T20:55:11+03:00: marked 32% (375 of 1170 index files)
2021-11-22T21:03:59+03:00: marked 33% (387 of 1170 index files)
2021-11-22T21:04:46+03:00: marked 34% (398 of 1170 index files)
2021-11-22T21:32:42+03:00: marked 35% (410 of 1170 index files)
2021-11-22T21:45:01+03:00: marked 36% (422 of 1170 index files)
2021-11-22T21:54:40+03:00: marked 37% (433 of 1170 index files)
2021-11-22T22:01:08+03:00: marked 38% (445 of 1170 index files)
2021-11-22T22:03:01+03:00: marked 39% (457 of 1170 index files)
2021-11-22T22:03:14+03:00: marked 40% (468 of 1170 index files)
2021-11-22T22:18:58+03:00: marked 41% (480 of 1170 index files)
2021-11-22T22:34:51+03:00: marked 42% (492 of 1170 index files)
2021-11-22T23:26:34+03:00: marked 43% (504 of 1170 index files)
2021-11-22T23:47:14+03:00: marked 44% (515 of 1170 index files)
2021-11-22T23:49:14+03:00: marked 45% (527 of 1170 index files)
2021-11-22T23:53:32+03:00: marked 46% (539 of 1170 index files)
2021-11-23T00:14:48+03:00: marked 47% (550 of 1170 index files)
2021-11-23T00:16:01+03:00: marked 48% (562 of 1170 index files)
2021-11-23T01:45:16+03:00: TASK ERROR: update atime failed for chunk/file "/mnt/datastore/pb-60-20-110/.chunks/e0cc/e0cca5ea028dbb01740de540f237045474a4fd62960b0ef51ca96ed40c402033" - ENOSPC: No space left on device

I can delete some snapshots, but GC will not be finished again.
How can i free up disk space?

Thanks in advance.

 

[SOLTECSIS - Carles Munyoz]

Look at this another post:
https://forum.proxmox.com/threads/disk-full-unable-to-run-garbage-collection.81800/

I hope it helps you.

 

[karpuzvepeynir]

I temporary moved the some chunk files (~ 5GB) to another disk and garbage collection process started. It took more than few days due to total size of backups is 20TB. I interrupted the gc and reinstall the new version of PBS.

Thanks Carles.

 

[SOLTECSIS - Carles Munyoz]

Great! Your are welcome.