Datastore is Full

[Dexter23]

Hi guys i have one customer with datastore full, i remove manually all the old snapshot and run the prune, after that i start the GC but i receive this error

This is the content actually:


This is the error:



Is there a way to fix that or i need to format the zpool?

 

[Max Carrara]

i remove manually all the old snapshot and run the prune

Just so I don't misunderstand: Do you mean you deleted the snapshots yourself and then pruned them?

Also, can you post the output of the df -h command?

 

[Dexter23]

Just so I don't misunderstand: Do you mean you deleted the snapshots yourself and then pruned them?

Also, can you post the output of the df -h command?

I remove the snapshot by do this click on the recycle

And then clink on "Run Now" in the "prune&gc" tab, and then click on start gc.

df-h

 

[Max Carrara]

There's currently no real way to deal with that nicely, unfortunately. There are some options, however:

• If your zpool is a mirror, you can upgrade it to a larger mirror. Replace the first drive of the mirror with a larger disk using zpool replace and then resilver using zpool resilver zpool01. After the resilver is done, repeat for each remaining drive - replace, then resilver - until all drives are upgraded. Once you're done, you should be able to run the GC.
• If you otherwise have another zpool available somewhere, you can use zfs send and zfs receive to move your datastore to that pool. You can then upgrade your original ZFS datastore and then send your data back again.

These are the only things I can think of off the top of my head.

 

[Dunuin]

In case you sync to a remote PBS, make sure "remove vanished" isn't set for the sync job. You could then wipe the datastore, create a new one and only sync the most revent backup snapshots back.

And for the future you should set a quota with ZFS so the customer isn't able to fill the datastore to 100%.

 

[Dexter23]

Hi

Ok i proceed with wipe the datastore and set the quota how can i do both of this operation?

 

[Dunuin]

Quota can be set with zfs set quota=1234G YourPool/YourDataset

 

[Dexter23]

Quota can be set with zfs set quota=1234G YourPool/YourDataset

Hi how much space free i need to reserve for Garbace Collect on for example 1TB of storage?

 

[Dunuin]

Hi how much space free i need to reserve for Garbace Collect on for example 1TB of storage?

Not sure but have a read on ZFS basics. A ZFS pool will be come slower and fragment faster the more you fill it and above 90% it will switch to a slower allocation mode. So when dimensioning your pool you should buy more storage to compensate this. A Rule of thumb is to not fill your pool more than 80% but I personally wasn't seeing a big difference between 80 and 90%, so I usually set a 90% quota and set my monitoring to alert me when the pool exceeds 80% so I can delete stuff or add more disks before it reaches 90%.

 

[Dexter23]

Not sure but have a read on ZFS basics. A ZFS pool will be come slower and fragment faster the more you fill it and above 90% it will switch to a slower allocation mode. So when dimensioning your pool you should buy more storage to compensate this. A Rule of thumb is to not fill your pool more than 80% but I personally wasn't seeing a big difference between 80 and 90%, so I usually set a 90% quota and set my monitoring to alert me when the pool exceeds 80% so I can delete stuff or add more disks before it reaches 90%.

i set the quota with this command is correct?


And how to set monitoring to send email when exceed 80%?

 

[Dunuin]

Ok so i set to 90% and how to set monitoring to send email when exced 80%?

I use zabbix with a PBS template to monitor my PBS pool/disk health and free space.

 

[Dexter23]

Ok you have zabbix in cloud or on premises in the same network of pbs?
Is difficult to setup and configure zabbix for this purpose?

 

[Dunuin]

@Max Carrara:
Would be great to have an additional option in the datastores notification settings for running out of space warnings. Maybe a textbox where you can type in 0-100% and it will send an notification once the datastore exceeds that. Maybe this would help a bit with people willing up the datastores to 100%. Is there already such a feature request in the bug tracker?

Ok you have zabbix in cloud or on premises in the same network of pbs?

Same network.

Is difficult to setup and configure zabbix for this purpose?

You need to set up a zabbix server first. So yes, that is some work to do and probably not worth to just monitor a single ZFS pool. But you should have some kind of monitoring anyway for all the hosts and guests including your PVE nodes. Whatever you use for monitoring these it probably also supports monitoring filesystems and ZFS pools.

 

[Dexter23]

@Max Carrara:
Would be great to have an additional option in the datastores notification settings for running out of space warnings. Maybe a textbox where you can type in 0-100% and it will send an notification once the datastore exceeds that. Maybe this would help a bit with people willing up the datastores to 100%. Is there already such a feature request in the bug tracker?

Same network.

You need to set up a zabbix server first. So yes, that is some work to do and probably not worth to just monitor a single ZFS pool. But you should have some kind of monitoring anyway for all the hosts and guests including your PVE nodes. Whatever you use for monitoring these it probably also supports monitoring filesystems and ZFS pools.

I don't use monitoring system like zabbix i have only configure email notifications on proxmox and thats'it. i use uptime kuma for monitoring tcp port include proxmox, but nothing more that that.
I've tried zabbix for monitoring but he takes time to setup and configure it.

 

[Max Carrara]

Is there already such a feature request in the bug tracker?

Not to my knowledge, but you can always open an RFE if you'd like!

Also, I'm currently looking into a solution to prevent this from happening - some kind of "safety guard" if your zpool ever should fill all the way up. There's not really much we can do if you're setting up a datastore on an already existing pool, but if you're creating the zpool as well as the datastore through the GUI for example, we might be able to do something.