[SOLVED] Single VM volume filling 100% of ZFS pool

[Henrique]

Hello!

I'm having this weird behavior with Proxmox installation with a ZFS pool (RAIDZ2, 12x10 TB) with a single VM with 72 Tb allocated. However, I have noticed that since September, the volume usage went from 76,89 TB to 88.55 TB, filling the pool to 100%.



The GUI (as you can see in the picture above) reports that the .raw takes 79.46 TB. The VM itself is only using ~40% of the allocated space (32.97 TB).

root@zfsbox1:/dev/zvol/ZFS1# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ZFS1   109T   106T  3.41T        -         -    64%    96%  1.00x    ONLINE  -

root@zfsbox1:/dev/zvol/ZFS1# zfs list
NAME                    USED  AVAIL     REFER  MOUNTPOINT
ZFS1                   80.5T  1.89M      219K  /ZFS1
ZFS1/vm-188195-disk-0  80.5T  1.89M     80.5T  -

I do not understand why the pool usage went to nearly 100% without even filling out the VM volume itself. We're not using snapshops, compression is enabled, and now I'm even locked up from the VM (as it doesn't boot with the filesystem in readonly mode) and somewhat afraid that I'm going to lose the data.

It's impressive that the volume grown about 1.8TB per week, since it started growing (~Sept 9). The data on the VM didn't grow at that rate, and there is still more than half of the space available in the vm image.

I'm attaching a .txt file with a bunch of commands that we have run that might help someone with a better eye (and zfs knowledge) spotting what might be wrong here.

Any ideas why this is happening? Is there anyway to reclaim the empty space being used (we already tried running zfs trim, but the results are the same)?

If you have any suggestions on how we can recover from the 100% filled pool, I also appreciate it!

Thanks.

 

[Pifouney]

Hi,

According to your "zfs.txt" file, you can found this information:

ZFS1 compressratio 1.09x -

Don't forgot computer working with a base 8, and not base 10, this faulted calcul explain your problem:
Disk capacity used (GUI screenshot): 79.46 TB
compression ratio(zfs.txt): 1.09
bad calcul(base 10) : 79.46*1.09= 86.6117

Your CLI screenshot say the same:

NAME USED AVAIL REFER MOUNTPOINT
ZFS1 80.5T 1.89M 219K /ZFS1

Your pool seems full ^^

EDIT: ZFS take a raw file as used space

 

[Dunuin]

What volblocksize do you use? If you didn't increased it from the default 8K volblocksize you will loose massiv storage due to padding overhead.

A 12x 10TB raidz2 with ashift=12 and volblocksize=8K will result in this:
12x10 TB = 120TB raw capacity
- 20 TB parity = 100 TB
- 60 TB padding (this is indirect because everything written to a zvol will be 250% in size) = 40 TB
- 20% because a ZFS pool shouldn't be filled more then 80% or it will get slow = 32 TB = 29,1 TiB of usable storage.
And if you use snapshots nothing will be deleted, so your zvol is growing and growing over time so in that case you might want to keep a big fraction of this 32TB free for snapshots.

So you actually can only use 29,1 TiB of that 120 TB raw storage for data and snaphshots. If you don't want to loose the 60TB to padding you need to increase the vollbockize and destroy and recreate that zvol. With a 16K vollbocksize you for example only should loose 20TB to padding (everything written will be 150% in size) and with a 256K vollbocksize only loose 1.6 TB to padding.

And you need to make sure that discard is working, or ZFS won'T free up space after deleting stuff inside the VM and the zvol will grow and grow too.

For the padding overhead you might want to look at this table. Keep in mind that this tables compares number of disks vs volblocksize in blocks. So if you use a ashift of 12 (which means 4K blocks) 16 blocks for example would mean a 64K volblocksize.

 

[Henrique]

What volblocksize do you use? If you didn't increased it from the default 8K volblocksize you will loose massiv storage due to padding overhead.

A 12x 10TB raidz2 with ashift=12 and volblocksize=8K will result in this:
12x10 TB = 120TB raw capacity
- 20 TB parity = 100 TB
- 60 TB padding (this is indirect because everything written to a zvol will be 250% in size) = 40 TB
- 20% because a ZFS pool shouldn't be filled more then 80% or it will get slow = 32 TB = 29,1 TiB of usable storage.
And if you use snapshots nothing will be deleted, so your zvol is growing and growing over time so in that case you might want to keep a big fraction of this 32TB free for snapshots.

So you actually can only use 29,1 TiB of that 120 TB raw storage for data and snaphshots. If you don't want to loose the 60TB to padding you need to increase the vollbockize and destroy and recreate that zvol. With a 16K vollbocksize you for example only should loose 20TB to padding (everything written will be 150% in size) and with a 256K vollbocksize only loose 1.6 TB to padding.

And you need to make sure that discard is working, or ZFS won'T free up space after deleting stuff inside the VM and the zvol will grow and grow too.

For the padding overhead you might want to look at this table. Keep in mind that this tables compares number of disks vs volblocksize in blocks. So if you use a ashift of 12 (which means 4K blocks) 16 blocks for example would mean a 64K volblocksize.

Thank you so much for the explanation! It is, indeed, set to 8k, I believe. Can you share how you made those padding calculations? Is there any good calculator for that?

 

[Dunuin]

Thank you so much for the explanation! It is, indeed, set to 8k, I believe. Can you share how you made those padding calculations? Is there any good calculator for that?

You can look at the table I already linked above. It shows the parity+padding losses for all drive and volblocksize combinations. If you really want to calculate the padding overhead yourself here is a great blog post explaining it.

 

[Henrique]

Perfect, thank you!