Datastore is Full

Dexter23

Member
Dec 23, 2021
193
15
23
34
Hi guys i have one customer with datastore full, i remove manually all the old snapshot and run the prune, after that i start the GC but i receive this error

This is the content actually:
1693205558814.png

This is the error:

1693205587651.png

Is there a way to fix that or i need to format the zpool?
 
i remove manually all the old snapshot and run the prune
Just so I don't misunderstand: Do you mean you deleted the snapshots yourself and then pruned them?

Also, can you post the output of the df -h command?
 
Just so I don't misunderstand: Do you mean you deleted the snapshots yourself and then pruned them?

Also, can you post the output of the df -h command?
I remove the snapshot by do this click on the recycle
1693209916053.png
And then clink on "Run Now" in the "prune&gc" tab, and then click on start gc.

df-h
1693210018537.png
 
There's currently no real way to deal with that nicely, unfortunately. There are some options, however:
  • If your zpool is a mirror, you can upgrade it to a larger mirror. Replace the first drive of the mirror with a larger disk using zpool replace and then resilver using zpool resilver zpool01. After the resilver is done, repeat for each remaining drive - replace, then resilver - until all drives are upgraded. Once you're done, you should be able to run the GC.
  • If you otherwise have another zpool available somewhere, you can use zfs send and zfs receive to move your datastore to that pool. You can then upgrade your original ZFS datastore and then send your data back again.
These are the only things I can think of off the top of my head.
 
In case you sync to a remote PBS, make sure "remove vanished" isn't set for the sync job. You could then wipe the datastore, create a new one and only sync the most revent backup snapshots back.

And for the future you should set a quota with ZFS so the customer isn't able to fill the datastore to 100%.
 
  • Like
Reactions: Max Carrara
Hi

Ok i proceed with wipe the datastore and set the quota how can i do both of this operation?
 
Hi how much space free i need to reserve for Garbace Collect on for example 1TB of storage?
Not sure but have a read on ZFS basics. A ZFS pool will be come slower and fragment faster the more you fill it and above 90% it will switch to a slower allocation mode. So when dimensioning your pool you should buy more storage to compensate this. A Rule of thumb is to not fill your pool more than 80% but I personally wasn't seeing a big difference between 80 and 90%, so I usually set a 90% quota and set my monitoring to alert me when the pool exceeds 80% so I can delete stuff or add more disks before it reaches 90%.
 
Not sure but have a read on ZFS basics. A ZFS pool will be come slower and fragment faster the more you fill it and above 90% it will switch to a slower allocation mode. So when dimensioning your pool you should buy more storage to compensate this. A Rule of thumb is to not fill your pool more than 80% but I personally wasn't seeing a big difference between 80 and 90%, so I usually set a 90% quota and set my monitoring to alert me when the pool exceeds 80% so I can delete stuff or add more disks before it reaches 90%.
i set the quota with this command is correct?
1693295081645.png

And how to set monitoring to send email when exceed 80%?
 
Last edited:
Ok you have zabbix in cloud or on premises in the same network of pbs?
Is difficult to setup and configure zabbix for this purpose?
 
@Max Carrara:
Would be great to have an additional option in the datastores notification settings for running out of space warnings. Maybe a textbox where you can type in 0-100% and it will send an notification once the datastore exceeds that. Maybe this would help a bit with people willing up the datastores to 100%. Is there already such a feature request in the bug tracker?
Ok you have zabbix in cloud or on premises in the same network of pbs?
Same network.
Is difficult to setup and configure zabbix for this purpose?
You need to set up a zabbix server first. So yes, that is some work to do and probably not worth to just monitor a single ZFS pool. But you should have some kind of monitoring anyway for all the hosts and guests including your PVE nodes. Whatever you use for monitoring these it probably also supports monitoring filesystems and ZFS pools.
 
@Max Carrara:
Would be great to have an additional option in the datastores notification settings for running out of space warnings. Maybe a textbox where you can type in 0-100% and it will send an notification once the datastore exceeds that. Maybe this would help a bit with people willing up the datastores to 100%. Is there already such a feature request in the bug tracker?

Same network.

You need to set up a zabbix server first. So yes, that is some work to do and probably not worth to just monitor a single ZFS pool. But you should have some kind of monitoring anyway for all the hosts and guests including your PVE nodes. Whatever you use for monitoring these it probably also supports monitoring filesystems and ZFS pools.
I don't use monitoring system like zabbix i have only configure email notifications on proxmox and thats'it. i use uptime kuma for monitoring tcp port include proxmox, but nothing more that that.
I've tried zabbix for monitoring but he takes time to setup and configure it.
 
Is there already such a feature request in the bug tracker?
Not to my knowledge, but you can always open an RFE if you'd like!

Also, I'm currently looking into a solution to prevent this from happening - some kind of "safety guard" if your zpool ever should fill all the way up. There's not really much we can do if you're setting up a datastore on an already existing pool, but if you're creating the zpool as well as the datastore through the GUI for example, we might be able to do something.
 
  • Like
Reactions: Dunuin

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!