How to recover from 100% disk use

[maomaocake]

Hi guys, I did an oopsie. I set up a new PBS server and forgot to configure a GC and a retention policy(I have one on pve), now the disk is at 100% full how do I get the GC to run now that its giving me a disk full error.

TASK ERROR: update atime failed for chunk/file "/mnt/datastore/backups/.chunks/3cd7/3cd7ece9322ea3a1e2070500555e348682a92e89cf23d6b66ad7e075805b552d" - ENOSPC: No space left on device
Error: task failed (status update atime failed for chunk/file "/mnt/datastore/backups/.chunks/3cd7/3cd7ece9322ea3a1e2070500555e348682a92e89cf23d6b66ad7e075805b552d" - ENOSPC: No space left on device)

 

[VictorSTS]

You have to run the Prune task first, that will give back little space which might be enough to run the GC afterwards. Which filesystem are you using for the datastore?

 

[maomaocake]

I'm using zfs as the storage. I pruned earlier today it dosent seem like there is any storage freed

 

[VictorSTS]

ZFS is a cow filesystem: it need some space to even be able to delete something. What's de output of df? is there any space left?

 

[maomaocake]

ZFS is a cow filesystem: it need some space to even be able to delete something. What's de output of df? is there any space left?

unfortunately 100%

 

[maomaocake]

I've decided to just nuke my current backups and re setup a new one. if anyone has any tops post it for the next person. I rather lose my current backups than to not be able to backup in the future

 

[VictorSTS]

You don't really need to do that. I don't remember the exact steps, but you can try:

1.- Place datastore in maintenance, read only
2.- Move some files from the .chunks directory to some other drive. Having like 200MB free should be enough.
3.- Run GC and hope it's able to delete some chunks. You may need to wait 24h+5m so the GC is really able to delete something [1]. You will get errors about missing chunks (those you moved in step 2).
4.- Move back the chunks to the same directory you moved them from.
5.- Run a full verify.
6.- Run GC again.
7.- If enough free space, remove maintenance mode.

 

[TErxleben]

You don't really need to do that. I don't remember the exact steps, but you can try:

1.- Place datastore in maintenance, read only
2.- Move some files from the .chunks directory to some other drive. Having like 200MB free should be enough.
3.- Run GC and hope it's able to delete some chunks. You may need to wait 24h+5m so the GC is really able to delete something [1]. You will get errors about missing chunks (those you moved in step 2).
4.- Move back the chunks to the same directory you moved them from.
5.- Run a full verify.
6.- Run GC again.
7.- If enough free space, remove maintenance mode.

@VictorSTS
Thank you for your simple and effective tip. This is how you revive a dead datastore.
I first tried expanding my ZFS datastore via zpool xyz. Unfortunately it doesn't work somehow.
The Proxmox team should definitely address the problem of full data stores. This is extremely annoying.

 

[NWTech]

I had the same problem. Was unable to put the datastore into maintenance mode.

 

[g777]

Why do you use ZFS at all? I mean it is written in manual, that

Monitor pool and file system space to make sure that they are not full. Consider using ZFS quotas and reservations to make sure file system space does not exceed 80% pool capacity.

ZFS is kinda enterprise thing if you read about it. It CAN run on single drive (I run it that way myself for now), but it's really not intended to.
If you don't use ZFS perks like snapshots, deduplication and sophisticated RAID-Z modes, just use ext4 + mdraid. It's not a bad or inferior thing in any way. It will be more like proper tool for your task.

My point is, ZFS is awesome, but it has it's own drawbacks and you at least should be aware of those to make effective use/don't use decisions.

If ZFS is tool of your choice, why don't install something like Zabbix, that takes resources next to nothing, but monitors and notifies you of your system.

 

[tcabernoch]

I did this to a system where the main storage also hosted the datastore.
Here's how you can prevent it.
Set a reservation on the dataset so you don't fill up again.

zfs set reservation=10g rpool/ROOT