Can't create 9 TB virtual hard drive on 11 TB zfs pool - out of space?

[stewlg]

I've set up a new zfs pool like so:

root@hydrox:/mnt# zpool create -m /mnt/droxcamstorage droxcamstorage raidz2 /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
root@hydrox:/mnt# zpool status
  pool: droxcamstorage
state: ONLINE
  scan: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        droxcamstorage  ONLINE       0     0     0
          raidz2-0      ONLINE       0     0     0
            sdd         ONLINE       0     0     0
            sde         ONLINE       0     0     0
            sdf         ONLINE       0     0     0
            sdg         ONLINE       0     0     0
            sdh         ONLINE       0     0     0

errors: No known data errors

  pool: rpool
state: ONLINE
  scan: scrub repaired 0B in 0 days 00:29:38 with 0 errors on Sun Aug  9 01:53:39 2020
config:

        NAME                                             STATE     READ WRITE CKSUM
        rpool                                            ONLINE       0     0     0
          mirror-0                                       ONLINE       0     0     0
            ata-SanDisk_SDSSDH3_1T00_200723A00769-part3  ONLINE       0     0     0
            ata-SanDisk_SDSSDH3_1T00_195008A007C5-part3  ONLINE       0     0     0

errors: No known data errors

The new zpool appears to have enough space to hold a large hard 9 TB drive:

root@hydrox:/mnt/droxcamstorage# df -h .
Filesystem      Size  Used Avail Use% Mounted on
droxcamstorage   11T  256K   11T   1% /mnt/droxcamstorage

But when I try to create a 9000 GB (~ 9 TB) hard drive, I get an error:

"zfs error: cannot create 'droxcamstorage/vm-100-disk-0': out of space at /usr/share/perl5/PVE/API2/Qemu.pm line 1340. (500) ."

I've experimented, and I can definitely create a 5000 GB hard drive, but not much larger than that.

What's going on?

 

[stewlg]

So I looked around and got half a clue about what the issue is here:

https://forum.proxmox.com/threads/virtual-disk-much-bigger-on-real-disk.73255/

and here:

https://forum.proxmox.com/threads/zfs-counts-double-the-space.71536/#post-320919

I'm still not straight on the precise details of why zfs requires this almost 2x overhead, but I got the message. Here's what I did to get to 9 TB.

• I destroyed my RAIDZ2 pool

root@hydrox:/mnt# zpool destroy droxcamstorage

• I added an additional drive of the same size (4 TB) to get me from 5 drives to 6, an even number. Fortunately I have a lot of them.
• I created a pool of mirrors, as recommended in the above posts:

root@hydrox:/mnt# create -m /mnt/droxruststorage droxruststorage mirror /dev/sdd /dev/sde mirror /dev/sdf /dev/sdg mirror /dev/sdh /dev/sdi
root@hydrox:/mnt# zpool status
  pool: droxruststorage
 state: ONLINE
  scan: none requested
config:

        NAME             STATE     READ WRITE CKSUM
        droxruststorage  ONLINE       0     0     0
          mirror-0       ONLINE       0     0     0
            sdd          ONLINE       0     0     0
            sde          ONLINE       0     0     0
          mirror-1       ONLINE       0     0     0
            sdf          ONLINE       0     0     0
            sdg          ONLINE       0     0     0
          mirror-2       ONLINE       0     0     0
            sdh          ONLINE       0     0     0
            sdi          ONLINE       0     0     0

errors: No known data errors
root@hydrox:/mnt# cd /mnt/droxruststorage/
root@hydrox:/mnt/droxruststorage# df -h .
Filesystem       Size  Used Avail Use% Mounted on
droxruststorage   11T  128K   11T   1% /mnt/droxruststorage
root@hydrox:/mnt/droxruststorage#

Now I was able to create the 9000 (~ 9 TB) hard drive on my Windows guest successfully.

I am generally uneasy about anything but RAIDz2 and RAIDz3 for ZFS. I have what I think is a decent understanding of RAIDz, but so far I have avoided anything but the simplest mirror setups. But, in this particular case, I am making a drive to store surveillance camera footage, which for me is not mission crtiical. So this is actually a good place to experiment with this stripped mirror drive configuration, at least for me.

I would gently suggest that the dialog that fails with the error give some feedback indicating WHY this error is happening, if it can be determined.
Perhaps something to explain that a new virtual hard drive it needs far more more space on ZFS than the disk storage space itself. It took me a long time and a lot of scrabbling at searches to stumble on the fix above.

 

[j4ys0n]

Confirmed - I tried to create a 3TB volume on a RAIDZ2 pool consisting of 6x 1TB SSDs. There was plenty of available space. I got the same error as above. However, if i create a RAID10 pool like above, i can create a 2TB volume on the pool made of the same disks.