Can't create 9 TB virtual hard drive on 11 TB zfs pool - out of space?

stewlg

New Member
Jul 8, 2020
5
1
3
52
I've set up a new zfs pool like so:

Code:
root@hydrox:/mnt# zpool create -m /mnt/droxcamstorage droxcamstorage raidz2 /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
root@hydrox:/mnt# zpool status
  pool: droxcamstorage
state: ONLINE
  scan: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        droxcamstorage  ONLINE       0     0     0
          raidz2-0      ONLINE       0     0     0
            sdd         ONLINE       0     0     0
            sde         ONLINE       0     0     0
            sdf         ONLINE       0     0     0
            sdg         ONLINE       0     0     0
            sdh         ONLINE       0     0     0

errors: No known data errors

  pool: rpool
state: ONLINE
  scan: scrub repaired 0B in 0 days 00:29:38 with 0 errors on Sun Aug  9 01:53:39 2020
config:

        NAME                                             STATE     READ WRITE CKSUM
        rpool                                            ONLINE       0     0     0
          mirror-0                                       ONLINE       0     0     0
            ata-SanDisk_SDSSDH3_1T00_200723A00769-part3  ONLINE       0     0     0
            ata-SanDisk_SDSSDH3_1T00_195008A007C5-part3  ONLINE       0     0     0

errors: No known data errors

The new zpool appears to have enough space to hold a large hard 9 TB drive:

Code:
root@hydrox:/mnt/droxcamstorage# df -h .
Filesystem      Size  Used Avail Use% Mounted on
droxcamstorage   11T  256K   11T   1% /mnt/droxcamstorage

But when I try to create a 9000 GB (~ 9 TB) hard drive, I get an error:

"zfs error: cannot create 'droxcamstorage/vm-100-disk-0': out of space at /usr/share/perl5/PVE/API2/Qemu.pm line 1340. (500) ."

I've experimented, and I can definitely create a 5000 GB hard drive, but not much larger than that.

What's going on?Can't Create Large (9 TB) hard drive on 11 TB ZFS volume.png
 

stewlg

New Member
Jul 8, 2020
5
1
3
52
So I looked around and got half a clue about what the issue is here:

https://forum.proxmox.com/threads/virtual-disk-much-bigger-on-real-disk.73255/

and here:

https://forum.proxmox.com/threads/zfs-counts-double-the-space.71536/#post-320919

I'm still not straight on the precise details of why zfs requires this almost 2x overhead, but I got the message. Here's what I did to get to 9 TB.

  • I destroyed my RAIDZ2 pool
Code:
root@hydrox:/mnt# zpool destroy droxcamstorage
  • I added an additional drive of the same size (4 TB) to get me from 5 drives to 6, an even number. Fortunately I have a lot of them.
  • I created a pool of mirrors, as recommended in the above posts:
Code:
root@hydrox:/mnt# create -m /mnt/droxruststorage droxruststorage mirror /dev/sdd /dev/sde mirror /dev/sdf /dev/sdg mirror /dev/sdh /dev/sdi
root@hydrox:/mnt# zpool status
  pool: droxruststorage
 state: ONLINE
  scan: none requested
config:

        NAME             STATE     READ WRITE CKSUM
        droxruststorage  ONLINE       0     0     0
          mirror-0       ONLINE       0     0     0
            sdd          ONLINE       0     0     0
            sde          ONLINE       0     0     0
          mirror-1       ONLINE       0     0     0
            sdf          ONLINE       0     0     0
            sdg          ONLINE       0     0     0
          mirror-2       ONLINE       0     0     0
            sdh          ONLINE       0     0     0
            sdi          ONLINE       0     0     0

errors: No known data errors
root@hydrox:/mnt# cd /mnt/droxruststorage/
root@hydrox:/mnt/droxruststorage# df -h .
Filesystem       Size  Used Avail Use% Mounted on
droxruststorage   11T  128K   11T   1% /mnt/droxruststorage
root@hydrox:/mnt/droxruststorage#

Now I was able to create the 9000 (~ 9 TB) hard drive on my Windows guest successfully.

I am generally uneasy about anything but RAIDz2 and RAIDz3 for ZFS. I have what I think is a decent understanding of RAIDz, but so far I have avoided anything but the simplest mirror setups. But, in this particular case, I am making a drive to store surveillance camera footage, which for me is not mission crtiical. So this is actually a good place to experiment with this stripped mirror drive configuration, at least for me.

I would gently suggest that the dialog that fails with the error give some feedback indicating WHY this error is happening, if it can be determined.
Perhaps something to explain that a new virtual hard drive it needs far more more space on ZFS than the disk storage space itself. It took me a long time and a lot of scrabbling at searches to stumble on the fix above.
 

j4ys0n

Member
Jan 15, 2020
34
7
13
Confirmed - I tried to create a 3TB volume on a RAIDZ2 pool consisting of 6x 1TB SSDs. There was plenty of available space. I got the same error as above. However, if i create a RAID10 pool like above, i can create a 2TB volume on the pool made of the same disks.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!