Missing directory space

[kromberg]

I have a PBS running the latest non version and I am seeing an issue I cant figure out. I manually removed all the backups on the server, but it is still show about 2.9TB of 12TB being used by something. How do I reclaim the space on the directory allocated? I have run garbage collection several times. Have rebooted, and waited over a day.

 

[Dunuin]

Would be useful if you could show us the log of the last GC task. What filesystem is your disk formatted with? Does a fstrim -a free up space? Did you check the contents of your datastore folder including the hidden folder like "/mnt/datastore/backups/.chunks"? Your last GC showed a deduplication factor of 3.82, so it looks like there are still chunks that the GC didn't remove. Maybe a lot of corrupted chunks (for example because you got bad RAM)?

 

[kromberg]

fstrim -a didnt do anything. Here is the last part of the log from running GC:

2023-07-30T14:40:00-06:00: processed 94% (763694 chunks)
2023-07-30T14:40:00-06:00: processed 95% (771937 chunks)
2023-07-30T14:40:01-06:00: processed 96% (779968 chunks)
2023-07-30T14:40:02-06:00: processed 97% (788039 chunks)
2023-07-30T14:40:02-06:00: processed 98% (796189 chunks)
2023-07-30T14:40:03-06:00: processed 99% (804256 chunks)
2023-07-30T14:40:03-06:00: Removed garbage: 0 B
2023-07-30T14:40:03-06:00: Removed chunks: 0
2023-07-30T14:40:03-06:00: Pending removals: 2.66 TiB (in 812259 chunks)
2023-07-30T14:40:03-06:00: Original data usage: 0 B
2023-07-30T14:40:03-06:00: On-Disk chunks: 0
2023-07-30T14:40:03-06:00: Deduplication factor: 1.00
2023-07-30T14:40:03-06:00: TASK OK

Hmmm.... looks like there is 2.66TB pending. How can that be removed?

 

[Dunuin]

You could check the dates of those chunks, that they are for example not in the future because of a misconfigured clock or something similar. The "atime" should be at least 24 hours and 5 minutes in the past or GC will handle them as pending. And I would check if there is a stuck task, as this also could prevent chunks from being deleted.

 

[kromberg]

I have removed all the jobs on the VE servers, so nothing should be referencing the PBS server. Can I just delete everything in the .chunks directory?

 

[Dunuin]

I have removed all the jobs on the VE servers, so nothing should be referencing the PBS server.

You could check the task history. Disabling backups jobs on the PVEs isn't enough. There could still be verify, GC, sync and prune jobs running by PBS.

Can I just delete everything in the .chunks directory?

If you just want to delete everything, yes. You could remove that datastore, delete the whole datastore folder and then create a new datastore.

 

[kromberg]

Well just deleting everything in the .chunks directory caused more problems. I ended up killing the directory and recreating it. We really, really, really need to get these steps automated through the GUI to cleanly remove/delete/wipe a directory and its disk so either the directory or disk can be used again:

1. proxmox-backup-manager datastore remove backups
2. systemctl disable mnt-datastore-backups.mount
3. umount /mnt/datastore/backups
4. rm /etc/systemd/system/mnt-datastore-backups.mount
5. sgdisk --zap-all /dev/sdb
6. wipefs -a /dev/sdb
7. rmdir /mnt/datastore/backups/
   

 

[Ernst T.]

Hmmm.... looks like there is 2.66TB pending. How can that be removed?

Wait 24 hours and 5 minutes, or "manipulate" the chunk date/time.