[SOLVED] Garbage collect "changed" schedule and can't run now

[Klug]

Hi.

Since two days (january 15), garbage collect start at midnight.
Before that, it started during the day, at very different time (see screenshot).



Since this schedule change, it fails with this error:

Garbage collection failed: cannot continue garbage-collection safely, permission denied on:

There's nothing more that "permission denied on:", I don't know on what it's denied.

On my end, I didn't change anything in the backup schedule. or garbage collection schedule.

If I try to launch GC manually from the web interface, I get the same error (permission denied but I don't know on what).

Syslog says:

Jan 17 09:19:33 pbs proxmox-backup-proxy[6187]: Detected stopped task 'UPID:pbs:0000182B:063F1EF8:00000D58:6003F315:garbage_collection:BKP_PBS_NCP:root@pam:'

/var/log/proxmox-backup/tasks/archive says:

UPID:pbs:0000182B:063F1EF8:00000D58:6003F315:garbage_collection:BKP_PBS_NCP:root@pam: 6003F315 cannot continue garbage-collection safely, permission denied on:

Where else should I have a look?

 

[Klug]

.gs-status file still exist in the root of the mountpoint (my repo is NFS).
File date is the end of the last GC that worked correctly.

{"upid":"UPID:pbs:0000182B:063F1EF8:00000C1B:6000CCF0:garbage_collection:BKP_PBS_NCP:root@pam:","index-file-count":3309,"index-data-bytes":392547343007744,"disk-bytes":5347405054169,"disk-chunks":2794076,"removed-bytes":8748717026,"removed-chunks":6518,"pending-bytes":8035900920,"pending

 

[fabian]

maybe the datastore root has wrong permissions?

 

[Klug]

Hi Fabian,

this should not have changed (no server reboot, no new user, no known change).
Also, backup happens correctly, pruning too.

root@pbs:/mnt# ls -la
total 36
drwxr-xr-x  4 root   root   4096 Oct 19 15:33 .
drwxr-xr-x 18 root   root   4096 Jan 18 16:00 ..
drwxrwxrwx 14 backup backup   16 Jan 15 09:55 BKP
drwxr-xr-x  2 root   root   4096 Oct 19 15:23 hostrun
root@pbs:/mnt# cd BKP/
root@pbs:/mnt/BKP# ls -la
total 25787
drwxrwxrwx    14 backup backup    16 Jan 15 09:55  .
drwxr-xr-x     4 root   root    4096 Oct 19 15:33  ..
drwxr-x--- 32767 backup backup 65538 Oct 19 15:47  .chunks
drwxr-xr-x     2 backup backup     2 Oct 19 18:30  dump
-rw-r--r--     1 backup backup   332 Jan 15 09:55  .gc-status
drwxr-xr-x     4 backup backup     4 Jan 15 16:49  images
-rw-r--r--     1 backup backup     0 Oct 19 15:46  .lock
drwxr-xr-x     3 backup backup     3 Aug 14 12:18  template
drwxr-xr-x    63 backup backup    63 Jan  4 22:01  vm

I'm not sure there's a link between the time change (why mignight suddendly) and GC that doesn't work anymore.
But as both happened together...

 

[dcsapak]

just to note, your time at which the garbage collection runs did not change, if you look closely, the 'start time' (1. column) is always at 00:00, the second columns (which varies) is the 'end time' of the task

can you post the output of

ls -lhR /mnt/BKP

?

my guess is that since you also used the pbs datastore directory as pve directory storage, there are some files which cannot be accesses from the 'backup' user

 

[Klug]

You were right.

I found my error: I used this repo to create a VM (the day the GC failed) instead of the correct storage.
I'm removing the "Disk image" property of the storage right now!