ZFS pool size full with incorrect actual usage.

Jan 18, 2020
196
1
23
28
Hi,


While checking my virtual machine I got an IO-Error. I have also checked the ZFS pool but it shows it was full. but I have confirmed we just configured the hard disk for the virtual machine hard disk size 1.55 TB.
I am not sure why the ZFS disk consumes more than the assigned for the VM, Anyone please advise on this issue?

Also, may I know is there any way to remove these unwanted zfs snapshots? Please advise.

Thank you.
 

Attachments

  • replication.png
    replication.png
    39.1 KB · Views: 29
  • virtual machine.png
    virtual machine.png
    64.9 KB · Views: 30
  • zfs storage size.png
    zfs storage size.png
    34.2 KB · Views: 29
  • zfs storage.png
    zfs storage.png
    14.2 KB · Views: 26

Dunuin

Famous Member
Jun 30, 2020
9,007
2,313
156
Germany
Last edited:
  • Like
Reactions: itNGO

UdoB

Renowned Member
Nov 1, 2016
452
130
68
Germany
May I know is there any way to remove these unwanted zfs snapshots?
Well..., are there any?

Via GUI: each VM has "Snapshots" in the vertical menu bar. Are there any snapshots listed? You can delete them there.

Via CLI: to check if "pure" ZFS-Snapshots exist use the <Node>-Shell in the GUI or use ssh. Run zfs list -t snapshot as root. If the output is not empty (with no error message) post the output here (in "CODE"-Tags from the formatting toolbar).

Additionally you should definitely read some of the posts Dunuin mentioned - he has done a very fine job with detailed explanations regarding the mentioned "padding overhead" and the "much more space used than expected"-effect.

Best regards
 
Jan 18, 2020
196
1
23
28
Hi,

we have listed the snapshot using 'zfs list -t snapshot' but Could you please let me know how to remove the snapshot from command line?

Looking forward to your reply.

Regards,
 
Jan 18, 2020
196
1
23
28
Also, We are not able to find the snapshot via GUI, Only able to find it through CLI, So please let us know how can we remove the same through CLI?
 

Attachments

  • snapshot gui.png
    snapshot gui.png
    27.9 KB · Views: 7

Dunuin

Famous Member
Jun 30, 2020
9,007
2,313
156
Germany
Usually you can delete snapshots by zfs destroy YourPool/YourDataset/or/Zvol@Snapshotname but in case your pool is completely full this might fail. Keep in mind that ZFS is a Copy-on-Write filesystem so in order to do any changes (for example edit or delete anything) you need to write new data first and for that you need atleast some free space. Because of that you should make sure that you never ever completely write your pool complete full so it won't become read only. Getting back to a working pool might require to copy everything to empty bigger disks by using dd or something similar (so basically replacing all your pools disks with bigger ones).
For the future you should set a pool wide quota of 80-90% of the pools usable capacity so no matter whats happening you always got 10-20% of free space. In general you might want always around 20% free because otherwise the pool will become slower and will fragment faster, which is bad, as a Copy-on-Write filesystem like ZFS can't be defragmented.
I personally prefer a 90% quota combined with external monitoring like zabbix that warns me as soon as 80% pool usage is exceeded, so I can manually free up some stuff to get it below 80% again.
 
Last edited:
Jan 18, 2020
196
1
23
28
Hi,

We are not sure why the ZFS disk consumes more than the assigned for the VM. So may I confirm, It is not possible to remove this snapshot using 'zfs destroy YourPool/YourDataset/or/Zvol@Snapshotname' ?

Please advise
 

Dunuin

Famous Member
Jun 30, 2020
9,007
2,313
156
Germany
Hi,

We are not sure why the ZFS disk consumes more than the assigned for the VM.
Thats normal. Without manually optimizing stuff like the volblocksize a raidz pool won't give you more space than an striped mirror (raid10). On paper it will look like you got more space but all the additional space you got is wasted because of padding overhead. Then everything will be something like 133%, 150% or 166% in size. You really should read the links I posted above. Especially https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!