Confused ...

[pvretano]

Let me start by saying that I am complete nubie with Proxmox so please forgive me if I am using terminology incorrectly or if what I am trying to do (or how I am doing it) is completely brain dead. Hopefully by describing my experiment(s) someone can point out the error of my ways.

In my environment I have a physical machine with 8 12TB HDDs (all WDC_WD120EFBX drives). All brand new drives. My plan is to create one big honking zfspool storage on which I can create virtual disks that I can attach to my virtual machines.

I have configured two virtual machines on this physical machine ... "test" and "data" with virtual O/S drives from another zfspool I created on a couple of SSDs on the physical machine.

Experiment #1:

1. Using the hypervisor Web GUI I create a zfspool in a RAIDZ configuration with compression turned off using all 8 12TB drives. That gives me 80.66TB of available space according to Proxmox. That I expect.
2. I create a virtual disk (70000GiB) and attach it to data.
3. I create a second virtual disk (70000Gib) and attach it to data.
4. I create a third virtual disk (70000Gib) and attach it to test.
5. I create a fourth virtual disk (70000Gib) and attache it to test.

I can log into the virtual machines, fdisk the drives to create partitions, create file systems, mount them, etc. How is this possible!? I have 80TB of physical space and I am adding close to 280TB of space to my virtual machines. I think the answer is that Proxmox allows me to over provision the drives.

Experiment #2:
Because I want more determinism than that, I did a second experiment that confused me even more. I decided to ignore the fact that Proxmox seemingly lets me create and attache arbitrarily large virtual drives to my virtual machines. I delete and clear everything I did in the first iteration and recreated the 80.66TB zfspool storage anew in a RAIDZ configuration without compression.

I decide to let the real physical limitations of my environment constrain my action and thus resolved to create a 60TB virtual disk and attach it to data and a 20TB virtual disk and attach it to test.
Using the Web GUI I try to create a 60TB (55879GiB) virtual disk on the 80.66TB zfspool storage. That fails with a "zfs error: cannot create 'raid-zfs/vm-100-disk-2': out of space". How can this be!? In the first iteration of this experiment Proxmox allowed me to create a 70000GiB virtual disks without a complaint ... several in fact. Why can I create oversized disks without any problem and yet can't create a disk that should comfortably fit within the 80.66TB that Proxmox is reporting?

So, if anyone can explain what I am doing wrong, I would greatly appreciate it. I am at a loss to explain the behaviour I am seeing and I really need to completely understand what is going on here before I can continue setting up this environment.

 

[Dunuin]

When using raidz there is padding overhead when using zvols with a too small volblocksize. Using the default volblocksize of 8K you will loose 50% of your raw capacity, so only 4 of 8 drives capacity will be usable. 12TB you will loose for parity, 36TB you will loose to padding overhead and only 48TB will be usable. And then you should keep in mind that a ZFS pool should always have 20% of space free or it will fragment faster (you can't defrag a ZFS pool!) and become slow. So real usable capacity you now got are 38.4 TB. And keep in mind that you create virtual disks with GiB and not GB when using the GUI. 38.4GB are just 34.9 TiB. So right now you shouldn't create more than 34.9 TiB of virtual disks.
To loose 30% less of your raw capacity to padding overhead you should increase your volblocksize from 8K to atleast 32K, destroy all zvols and create them again (volblocksize can only be set at creation).
See here for a detailed explanation: https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz

And usually LZ4 compression is a good thing and will increase performance when using a HDD pool.