ZFS pool size full with incorrect actual usage.

[powersupport]

Hi,


While checking my virtual machine I got an IO-Error. I have also checked the ZFS pool but it shows it was full. but I have confirmed we just configured the hard disk for the virtual machine hard disk size 1.55 TB.
I am not sure why the ZFS disk consumes more than the assigned for the VM, Anyone please advise on this issue?

Also, may I know is there any way to remove these unwanted zfs snapshots? Please advise.

Thank you.

 

[Dunuin]

I explained it atleast 100 times in this forum. Search this forum for "padding overhead". You probably need to destroy and recreate all zvols with a bigger volblocksize so your raidz1/2/3 isn't wasting too much space because of padding overhead. If you want to understand the padding overhead problem this is a good source: https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz

 

[powersupport]

Hi,
May I know is there any way to remove these unwanted zfs snapshots? Please advise.
Thank you.

 

[UdoB]

May I know is there any way to remove these unwanted zfs snapshots?

Well..., are there any?

Via GUI: each VM has "Snapshots" in the vertical menu bar. Are there any snapshots listed? You can delete them there.

Via CLI: to check if "pure" ZFS-Snapshots exist use the <Node>-Shell in the GUI or use ssh. Run zfs list -t snapshot as root. If the output is not empty (with no error message) post the output here (in "CODE"-Tags from the formatting toolbar).

Additionally you should definitely read some of the posts Dunuin mentioned - he has done a very fine job with detailed explanations regarding the mentioned "padding overhead" and the "much more space used than expected"-effect.

Best regards

 

[powersupport]

Hi,

we have listed the snapshot using 'zfs list -t snapshot' but Could you please let me know how to remove the snapshot from command line?

Looking forward to your reply.

Regards,

 

[powersupport]

I have attached the Zfs snapshot list. Please have a look. It is urgent.

 

[powersupport]

Also, We are not able to find the snapshot via GUI, Only able to find it through CLI, So please let us know how can we remove the same through CLI?

 

[Dunuin]

Usually you can delete snapshots by zfs destroy YourPool/YourDataset/or/Zvol@Snapshotname but in case your pool is completely full this might fail. Keep in mind that ZFS is a Copy-on-Write filesystem so in order to do any changes (for example edit or delete anything) you need to write new data first and for that you need atleast some free space. Because of that you should make sure that you never ever completely write your pool complete full so it won't become read only. Getting back to a working pool might require to copy everything to empty bigger disks by using dd or something similar (so basically replacing all your pools disks with bigger ones).
For the future you should set a pool wide quota of 80-90% of the pools usable capacity so no matter whats happening you always got 10-20% of free space. In general you might want always around 20% free because otherwise the pool will become slower and will fragment faster, which is bad, as a Copy-on-Write filesystem like ZFS can't be defragmented.
I personally prefer a 90% quota combined with external monitoring like zabbix that warns me as soon as 80% pool usage is exceeded, so I can manually free up some stuff to get it below 80% again.

 

[powersupport]

Hi,

We are not sure why the ZFS disk consumes more than the assigned for the VM. So may I confirm, It is not possible to remove this snapshot using 'zfs destroy YourPool/YourDataset/or/Zvol@Snapshotname' ?

Please advise

 

[Dunuin]

Hi,

We are not sure why the ZFS disk consumes more than the assigned for the VM.

Thats normal. Without manually optimizing stuff like the volblocksize a raidz pool won't give you more space than an striped mirror (raid10). On paper it will look like you got more space but all the additional space you got is wasted because of padding overhead. Then everything will be something like 133%, 150% or 166% in size. You really should read the links I posted above. Especially https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz