[SOLVED] S3 garbage collection error

R

rahman

Renowned Member
Nov 1, 2010
100
5
83
Hi,

I have a failed garbage collection job with this error:

Code: 
2026-06-16T00:00:00+03:00: starting garbage collection on store ulakfkm
2026-06-16T00:00:00+03:00: task triggered by schedule 'daily'
2026-06-16T00:00:01+03:00: Access time update check successful, proceeding with GC.
2026-06-16T00:00:01+03:00: Using access time cutoff 1d 5m, minimum access time is 2026-06-14T20:55:00Z
2026-06-16T00:00:01+03:00: Start GC phase1 (mark used chunks)
2026-06-16T00:00:06+03:00: marked 1% (10 of 907 index files)
2026-06-16T00:00:07+03:00: marked 2% (19 of 907 index files)
2026-06-16T00:00:08+03:00: marked 3% (28 of 907 index files)
2026-06-16T00:00:11+03:00: marked 4% (37 of 907 index files)
2026-06-16T00:00:18+03:00: marked 5% (46 of 907 index files)
2026-06-16T00:00:20+03:00: marked 6% (55 of 907 index files)
2026-06-16T00:00:22+03:00: marked 7% (64 of 907 index files)
2026-06-16T00:00:25+03:00: marked 8% (73 of 907 index files)
2026-06-16T00:00:26+03:00: marked 9% (82 of 907 index files)
2026-06-16T00:00:26+03:00: marked 10% (91 of 907 index files)
2026-06-16T00:00:27+03:00: marked 11% (100 of 907 index files)
2026-06-16T00:00:27+03:00: marked 12% (109 of 907 index files)
2026-06-16T00:00:27+03:00: marked 13% (118 of 907 index files)
2026-06-16T00:00:28+03:00: marked 14% (127 of 907 index files)
2026-06-16T00:00:29+03:00: marked 15% (137 of 907 index files)
2026-06-16T00:00:29+03:00: marked 16% (146 of 907 index files)
2026-06-16T00:00:30+03:00: marked 17% (155 of 907 index files)
2026-06-16T00:00:31+03:00: marked 18% (164 of 907 index files)
2026-06-16T00:00:32+03:00: marked 19% (173 of 907 index files)
2026-06-16T00:00:33+03:00: marked 20% (182 of 907 index files)
2026-06-16T00:00:34+03:00: marked 21% (191 of 907 index files)
2026-06-16T00:00:37+03:00: marked 22% (200 of 907 index files)
2026-06-16T00:00:38+03:00: marked 23% (209 of 907 index files)
2026-06-16T00:00:39+03:00: marked 24% (218 of 907 index files)
2026-06-16T00:00:41+03:00: marked 25% (227 of 907 index files)
2026-06-16T00:00:42+03:00: marked 26% (236 of 907 index files)
2026-06-16T00:00:42+03:00: marked 27% (245 of 907 index files)
2026-06-16T00:00:42+03:00: marked 28% (254 of 907 index files)
2026-06-16T00:00:44+03:00: marked 29% (264 of 907 index files)
2026-06-16T00:00:45+03:00: marked 30% (273 of 907 index files)
2026-06-16T00:00:45+03:00: marked 31% (282 of 907 index files)
2026-06-16T00:00:45+03:00: marked 32% (291 of 907 index files)
2026-06-16T00:00:45+03:00: marked 33% (300 of 907 index files)
2026-06-16T00:00:50+03:00: marked 34% (309 of 907 index files)
2026-06-16T00:00:51+03:00: marked 35% (318 of 907 index files)
2026-06-16T00:00:52+03:00: marked 36% (327 of 907 index files)
2026-06-16T00:00:53+03:00: marked 37% (336 of 907 index files)
2026-06-16T00:00:54+03:00: marked 38% (345 of 907 index files)
2026-06-16T00:00:55+03:00: marked 39% (354 of 907 index files)
2026-06-16T00:00:56+03:00: marked 40% (363 of 907 index files)
2026-06-16T00:00:57+03:00: marked 41% (372 of 907 index files)
2026-06-16T00:01:09+03:00: marked 42% (381 of 907 index files)
2026-06-16T00:01:11+03:00: marked 43% (391 of 907 index files)
2026-06-16T00:01:11+03:00: marked 44% (400 of 907 index files)
2026-06-16T00:01:13+03:00: marked 45% (409 of 907 index files)
2026-06-16T00:01:14+03:00: marked 46% (418 of 907 index files)
2026-06-16T00:01:15+03:00: marked 47% (427 of 907 index files)
2026-06-16T00:01:15+03:00: marked 48% (436 of 907 index files)
2026-06-16T00:01:16+03:00: marked 49% (445 of 907 index files)
2026-06-16T00:01:17+03:00: marked 50% (454 of 907 index files)
2026-06-16T00:01:17+03:00: marked 51% (463 of 907 index files)
2026-06-16T00:01:18+03:00: marked 52% (472 of 907 index files)
2026-06-16T00:01:18+03:00: marked 53% (481 of 907 index files)
2026-06-16T00:01:29+03:00: marked 54% (490 of 907 index files)
2026-06-16T00:01:30+03:00: marked 55% (499 of 907 index files)
2026-06-16T00:01:33+03:00: marked 56% (508 of 907 index files)
2026-06-16T00:01:34+03:00: marked 57% (517 of 907 index files)
2026-06-16T00:01:34+03:00: marked 58% (527 of 907 index files)
2026-06-16T00:01:35+03:00: marked 59% (536 of 907 index files)
2026-06-16T00:01:35+03:00: marked 60% (545 of 907 index files)
2026-06-16T00:01:35+03:00: marked 61% (554 of 907 index files)
2026-06-16T00:01:35+03:00: marked 62% (563 of 907 index files)
2026-06-16T00:01:35+03:00: marked 63% (572 of 907 index files)
2026-06-16T00:01:36+03:00: marked 64% (581 of 907 index files)
2026-06-16T00:01:37+03:00: marked 65% (590 of 907 index files)
2026-06-16T00:01:37+03:00: marked 66% (599 of 907 index files)
2026-06-16T00:01:38+03:00: marked 67% (608 of 907 index files)
2026-06-16T00:01:39+03:00: marked 68% (617 of 907 index files)
2026-06-16T00:01:46+03:00: marked 69% (626 of 907 index files)
2026-06-16T00:01:48+03:00: marked 70% (635 of 907 index files)
2026-06-16T00:01:48+03:00: marked 71% (644 of 907 index files)
2026-06-16T00:01:49+03:00: marked 72% (654 of 907 index files)
2026-06-16T00:01:49+03:00: marked 73% (663 of 907 index files)
2026-06-16T00:01:52+03:00: marked 74% (672 of 907 index files)
2026-06-16T00:01:52+03:00: marked 75% (681 of 907 index files)
2026-06-16T00:01:53+03:00: marked 76% (690 of 907 index files)
2026-06-16T00:01:53+03:00: marked 77% (699 of 907 index files)
2026-06-16T00:01:54+03:00: marked 78% (708 of 907 index files)
2026-06-16T00:01:55+03:00: marked 79% (717 of 907 index files)
2026-06-16T00:01:56+03:00: marked 80% (726 of 907 index files)
2026-06-16T00:01:56+03:00: marked 81% (735 of 907 index files)
2026-06-16T00:01:56+03:00: marked 82% (744 of 907 index files)
2026-06-16T00:01:57+03:00: marked 83% (753 of 907 index files)
2026-06-16T00:01:58+03:00: marked 84% (762 of 907 index files)
2026-06-16T00:01:58+03:00: marked 85% (771 of 907 index files)
2026-06-16T00:01:58+03:00: marked 86% (781 of 907 index files)
2026-06-16T00:02:06+03:00: marked 87% (790 of 907 index files)
2026-06-16T00:02:07+03:00: marked 88% (799 of 907 index files)
2026-06-16T00:02:12+03:00: marked 89% (808 of 907 index files)
2026-06-16T00:02:22+03:00: marked 90% (817 of 907 index files)
2026-06-16T00:02:22+03:00: marked 91% (826 of 907 index files)
2026-06-16T00:02:23+03:00: marked 92% (835 of 907 index files)
2026-06-16T00:02:25+03:00: marked 93% (844 of 907 index files)
2026-06-16T00:02:26+03:00: marked 94% (853 of 907 index files)
2026-06-16T00:02:28+03:00: marked 95% (862 of 907 index files)
2026-06-16T00:02:29+03:00: marked 96% (871 of 907 index files)
2026-06-16T00:02:30+03:00: marked 97% (880 of 907 index files)
2026-06-16T00:02:31+03:00: marked 98% (889 of 907 index files)
2026-06-16T00:02:32+03:00: marked 99% (898 of 907 index files)
2026-06-16T00:02:32+03:00: marked 100% (907 of 907 index files)
2026-06-16T00:02:32+03:00: Start GC phase2 (sweep unused chunks)
2026-06-16T00:30:10+03:00: <?xml version="1.0" encoding="UTF-8"?><Error><Code>RequestTimeTooSkewed</Code><RequestId>tx0000004bac5667100c9a1-006a306ee2-195c3d2f-default</RequestId><HostId>195c3d2f-default-default</HostId></Error>
2026-06-16T00:30:10+03:00: queued notification (id=67bc541a-ee18-4ac5-83de-a75a33c861e6)
2026-06-16T00:30:10+03:00: TASK ERROR: unexpected status code 403 Forbidden

This does not happen every time but occasionally I get this. This is a Ceph based S3 repo. So which side is the cause of the error, PBS or S3?

Regards

Rahman
 
it's an error returned by your S3 endpoint.. the error code indicates that maybe clocks of the PBS and ceph servers are not synchronized properly?
 
  • Like
Reactions: rahman
fabian said:
it's an error returned by your S3 endpoint.. the error code indicates that maybe clocks of the PBS and ceph servers are not synchronized properly?
Click to expand...
Thank you for quick reply. We use our own time server for pbs and all our servers, network devices etc. which is plain Debian 13 and uses default debian time servers.

Code: 
root@pbs1:~# systemctl status chrony
● chrony.service - chrony, an NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chrony.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-06-03 11:01:07 +03; 1 week 5 days ago
 Invocation: 1364069c218d4bb78e3c8020d93d6bcd
       Docs: man:chronyd(8)
             man:chronyc(1)
             man:chrony.conf(5)
    Process: 309527 ExecStart=/usr/sbin/chronyd $DAEMON_OPTS (code=exited, status=0/SUCCESS)
   Main PID: 309537 (chronyd)
      Tasks: 2 (limit: 361308)
     Memory: 2M (peak: 3.4M)
        CPU: 2.719s
     CGroup: /system.slice/chrony.service
             ├─309537 /usr/sbin/chronyd -F 1
             └─309539 /usr/sbin/chronyd -F 1

Jun 03 11:01:07 pbs1 systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Jun 03 11:01:07 pbs1 chronyd[309537]: chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBUG)
Jun 03 11:01:07 pbs1 chronyd[309537]: Loaded 0 symmetric keys
Jun 03 11:01:07 pbs1 chronyd[309537]: Using leap second list /usr/share/zoneinfo/leap-seconds.list
Jun 03 11:01:07 pbs1 chronyd[309537]: Frequency -4.496 +/- 0.023 ppm read from /var/lib/chrony/chrony.drift
Jun 03 11:01:07 pbs1 chronyd[309537]: Loaded seccomp filter (level 1)
Jun 03 11:01:07 pbs1 chronyd[309537]: Added source ID#0000000009 (time.etu)
Jun 03 11:01:07 pbs1 systemd[1]: Started chrony.service - chrony, an NTP client/server.
Jun 03 11:01:12 pbs1 chronyd[309537]: Selected source 10.0.1.7 (time.etu)
Jun 03 11:01:12 pbs1 chronyd[309537]: System clock TAI offset set to 37 seconds

I will ask the provider about the error.
 
You must log in or register to reply here.
Top Bottom
Back