[SOLVED] Garbage Collect fails, permission denied on: "/run/initramfs"

[Pcom]

Hello,

PBS 3.2.2 running as a virtual machine on Truenas.
user root@pam with 2FA enabled used only for web login, all backups run using API.

Everything works fine, except garbage collection. When I try to use Garbage Collection this is the message:

2024-05-01T14:41:23+02:00: starting garbage collection on store pbs
2024-05-01T14:41:23+02:00: Start GC phase1 (mark used chunks)
2024-05-01T14:41:23+02:00: TASK ERROR: cannot continue garbage-collection safely, permission denied on: "/run/initramfs"

I've searched this error on the forum but didn't find anything.
In my mind (but maybe I'm wrong) GC runs with root@pam user and I have only API user under "permission" but if I try to add root@pam nothing happens. Could this be the problem?



I will finish available space in a week because I'm not deleting all the old backups.

How can I solve this?
Thanks!

 

[dcsapak]

what is the path of the datastore?

 

[Pcom]

Umh.
I thought it was /datastore/pbs like in the screenshot but there is no /datastore directory on the server

I found the command "pvesm list" but server says
-bash: pvesm: command not found

So I tried to check every directory manually and I see the hidden folder /.chunks with 65536 subdirectories and huge amount of files inside every folder.
I think this is my datastore. Is this normal or I set up something wrong? (btw backup and restore works fine)



Thanks

 

[VictorSTS]

Run cat /etc/proxmox-backup/datastore.cfg to find out where did you setup your datastore.

 

[Pcom]

Something's wrong here, I think

 

[dcsapak]

yeah, having the datastore on '/' won't work

i'd probably create a new datastore somewhere else on disk, sync the snapshots there, and then delete the datastore *manually* as i don't know if our deletion mechanism in the gui/api deletes everything in that directory (which would wipe your server)

alternatively, start with a fresh pbs install and put the datastore somewhere else (e.g. when using the 'disks' menu, the datastores will always be under '/mnt/datastore/<datastore-name>')

 

[VictorSTS]

Yes, the datastore should be in a subdirectory, not root. I have filed a bug [1] because IMHO PBS should not allow to create datastores in "/".

Maybe you can recover you backups creating a new datastore in the right path and doing a sync (dcsapak was faster to reply), but haven't really tested this. You will need twice the space in your drive.

Maybe you can move your backups manually to a new datastore (untested):

- mkdir /datastore/pbs.
- If needed, mount a drive, nfs, whatever.
- mv /.chunks /datastore/pbs
- mv /.gc_status /datastore/pbs
- mv /host /datastore/pbs
- mv /vm /datastore/pbs
- mv /ct /datastore/pbs
- mv /ns /datastore/pbs
- Edit /etc/proxmox-backup/datastore.cfg and modify the "path" in your "pbs" datastore to be /datastore/pbs.
- Reboot the PBS server to make sure all services get restarted (no sure which ones may need it).

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5439

 

[Pcom]

So I manually create folder /mnt/datastore/pbs and moved here ".chunks" and "vm", the only folders I had. Then I edited datastore.cfg
Seems OK.
I still see my backups and now garbage collection is working, it cleaned 600 gb in minutes.

I think this is solved, I'm going to update you in case i'll find something strange, just for future readers.

Thanks!

 

[PapaPingouin]

So I manually create folder /mnt/datastore/pbs and moved here ".chunks" and "vm", the only folders I had. Then I edited datastore.cfg
Seems OK.
I still see my backups and now garbage collection is working, it cleaned 600 gb in minutes.

I think this is solved, I'm going to update you in case i'll find something strange, just for future readers.

Thanks!

Yes, the datastore should be in a subdirectory, not root. I have filed a bug [1] because IMHO PBS should not allow to create datastores in "/".

Maybe you can recover you backups creating a new datastore in the right path and doing a sync (dcsapak was faster to reply), but haven't really tested this. You will need twice the space in your drive.

Maybe you can move your backups manually to a new datastore (untested):

- mkdir /datastore/pbs.
- If needed, mount a drive, nfs, whatever.
- mv /.chunks /datastore/pbs
- mv /.gc_status /datastore/pbs
- mv /host /datastore/pbs
- mv /vm /datastore/pbs
- mv /ct /datastore/pbs
- mv /ns /datastore/pbs
- Edit /etc/proxmox-backup/datastore.cfg and modify the "path" in your "pbs" datastore to be /datastore/pbs.
- Reboot the PBS server to make sure all services get restarted (no sure which ones may need it).

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=5439

I confirm, it works for me too.
- create new folder
- move content
- edit /etc/proxmox-backup/datastore.cfg
Before reboot, the GUI update with empty content. After reboot, all works fine.

Thanks guys

 

[R_2_D_2]

This started happening to me too on PBS 3.2-7 I also ran out of storage space and unable to perform GC luckily I found this thread.

Attempting to run GC:

2024-10-05T02:45:46-05:00: starting garbage collection on store Backup
2024-10-05T02:45:46-05:00: Start GC phase1 (mark used chunks)
2024-10-05T02:45:46-05:00: TASK ERROR: cannot continue garbage-collection safely, permission denied on: "/run/initramfs"

After performing:
mkdir /mnt/datastore/pbs
mv /.chunks /mnt/datastore/pbs
mv /vm /mnt/datastore/pbs

Edit and repoint to new location:
nano /etc/proxmox-backup/datastore.cfg

I then had permission issues so ran to give full read/write/execute access:
chmod -R 777 /mnt/datastore/pbs

Reboot, then ran GC in GUI and I can also confirm this workaround works and seems to be fixed now thanks guys!

 

[VictorSTS]

chmod -R 777 /mnt/datastore/pbs

A tip: use rsync -av instead of a cp to copy ACLs with the file and avoid permissions problems. I know a few places where chmod 777 is considered a crime

 

[R_2_D_2]

A tip: use rsync -av instead of a cp to copy ACLs with the file and avoid permissions problems. I know a few places where chmod 777 is considered a crime

Thanks for the tip something to remember for next time. I guess the bug of being able to create the datastore in " / " has not reached the latest version but I started with a version that allowed this took me a bit to figure it out. Yes chmod 777 is a crime good thing this is a home lab and not a production environment