Proxmox Backup Server with disks full

thomasddiego

Member
Dec 17, 2021
25
2
8
Proxmox Backup Server 2.1-1 server with two zfs partitions, 10TB and 12TB, forming a datastore that is 100% used. I deleted some larger backups, but the space was not freed up. Everything that is on the disks, I no longer need, how to clean the disks and make the space available again for use?
 
  • Like
Reactions: Bran-Ko
Proxmox Backup Server 2.1-1 server with two zfs partitions, 10TB and 12TB, forming a datastore that is 100% used. I deleted some larger backups, but the space was not freed up. Everything that is on the disks, I no longer need, how to clean the disks and make the space available again for use?
Hi,
do you have a garbage collection scheduled? Deleting the backups will only free the index files, not the chunks containing the actual data. Only by runnning a garbage collection job, you will free up non-referenced chunks, which are older that >24 hours.
 
  • Like
Reactions: thomasddiego
And with 100% full pools the GC will probably fail because of insufficient space. Make sure to disable all backup jobs so it won't fill up the few MBs you might be able to free um by deleting indexes.
 
  • Like
Reactions: thomasddiego
Gargabe starts and always stops pointing to different chunks

2023-08-21T12:33:07-03:00: starting garbage collection on store BackupUnit
2023-08-21T12:33:07-03:00: Start GC phase1 (mark used chunks)
2023-08-21T13:04:19-03:00: marked 1% (3 of 218 index files)
2023-08-21T13:12:42-03:00: marked 2% (5 of 218 index files)
2023-08-21T13:24:28-03:00: marked 3% (7 of 218 index files)
2023-08-21T13:29:49-03:00: marked 4% (9 of 218 index files)
2023-08-21T13:31:16-03:00: marked 5% (11 of 218 index files)
2023-08-21T13:41:49-03:00: marked 6% (14 of 218 index files)
2023-08-21T13:41:50-03:00: TASK ERROR: update atime failed for chunk/file "/mnt/datastore/BackupUnit/.chunks/e437/e437495ad2cc01deca02e9bb0a06869ee69dd7da4eb723ff55ced3e91cf8bcd7" - EPERM: Operation not permitted


The disk space of the VM where the Proxmox Backup OS is installed is ok, only the datastore where the backups are saved is having a problem.

Any other suggestion?
 

Attachments

  • proxmox_backup.png
    proxmox_backup.png
    15.4 KB · Views: 28
Gargabe starts and always stops pointing to different chunks

2023-08-21T12:33:07-03:00: starting garbage collection on store BackupUnit
2023-08-21T12:33:07-03:00: Start GC phase1 (mark used chunks)
2023-08-21T13:04:19-03:00: marked 1% (3 of 218 index files)
2023-08-21T13:12:42-03:00: marked 2% (5 of 218 index files)
2023-08-21T13:24:28-03:00: marked 3% (7 of 218 index files)
2023-08-21T13:29:49-03:00: marked 4% (9 of 218 index files)
2023-08-21T13:31:16-03:00: marked 5% (11 of 218 index files)
2023-08-21T13:41:49-03:00: marked 6% (14 of 218 index files)
2023-08-21T13:41:50-03:00: TASK ERROR: update atime failed for chunk/file "/mnt/datastore/BackupUnit/.chunks/e437/e437495ad2cc01deca02e9bb0a06869ee69dd7da4eb723ff55ced3e91cf8bcd7" - EPERM: Operation not permitted


The disk space of the VM where the Proxmox Backup OS is installed is ok, only the datastore where the backups are saved is having a problem.

Any other suggestion?
So you still have 218 index files referencing chunks. It might be worth to delete those not needed anymore before running the garbage collection.

The EPERM when updating the atime might be a sideffect of the zfs pool running out of space, as copy on write requires some space left to work, so freeing up a few more MB by deleting additional index files might be enough to fix this.

If you don't care about the backups on this datastore anymore, as stated in your initial post, you could also delete the datastore completely and recreate a new one after removing all the .chunks and index files.
 
  • Like
Reactions: thomasddiego
Good morning everybody,
I'm coming back here just to report that all the tips helped to resolve the issue.

I deleted the indexes of the backups already made, then performed garbage collection, after many hours, all the datastore space was freed. I have already reconfigured which VMs should make backups and the first run has already completed successfully.

I want to thank Chris and Dunuin for their tips and suggestions.
 
  • Like
Reactions: Chris

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!