[SOLVED] Passing 50TB Virtual Hard Drive to VM - Error About Out of Space?

[dmarc2024]

Hey All -

I created a new ZFS Pool with raidZ2 called "zfs-datastore-rz2" with my 12x6TB drives and I see it via the zpool status and df -h command as shown below. However, when going to the VM, adding a virtual hard drive, and entering 50000 for the drive storage to be created and passed to the VM, it presents an out of space error. Why? It doesnt' exceed the storage amount available according to the image below. The largest value it would accept was 43500GB.

zfs status:
pool: zfs-datstore-rz2
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
zfs-datstore-rz2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ata-ST6000VN0033-2EE110_ZADDG9F4 ONLINE 0 0 0
ata-ST6000VN001-2BB186_ZCTAAMXX ONLINE 0 0 0
ata-ST6000VN001-2BB186_ZA12ABAM ONLINE 0 0 0
ata-ST6000VN0033-2EE110_ZADFG9M6 ONLINE 0 0 0
ata-ST6000VN0033-2EE110_ZADAHAX2 ONLINE 0 0 0
ata-ST6000NM0044_Z4D2R3KA ONLINE 0 0 0
ata-ST6000NM0044_Z4D0DLW7 ONLINE 0 0 0
ata-ST6000NM0044_Z4D1W4Y6 ONLINE 0 0 0
ata-ST6000NM0044_Z4DDFT0M ONLINE 0 0 0
ata-ST6000NM0044_Z4DGGPSZ ONLINE 0 0 0
ata-ST6000NM0044_Z4DSEXC8 ONLINE 0 0 0
ata-ST6000NM0044_Z4A1303S ONLINE 0 0 0


df -h:

Filesystem Size Used Avail Use% Mounted on
udev 252G 0 252G 0% /dev
tmpfs 51G 4.2M 51G 1% /run
rpool/ROOT/pve-1 1.6T 1.3G 1.6T 1% /
tmpfs 252G 46M 252G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
rpool 1.6T 128K 1.6T 1% /rpool
rpool/var-lib-vz 1.6T 6.8G 1.6T 1% /var/lib/vz
rpool/data 1.6T 256K 1.6T 1% /rpool/data
rpool/ROOT 1.6T 256K 1.6T 1% /rpool/ROOT
/dev/fuse 128M 20K 128M 1% /etc/pve
tmpfs 51G 0 51G 0% /run/user/0
zfs-datastore-rz2 50T 256K 50T 1% /zfs-datastore-rz2
zfs-datastore-rz2/data 50T 384K 50T 1% /zfs-datastore-rz2/data

[B]ADD Hard Disk image:[/B]
[B][ATTACH type="full" size="599x386"]79443[/ATTACH][/B]

 

[gfngfn256]

You may be facing bits, bytes & bib's scenario:

50 Terabytes = 46566.1 Gibibytes

Try 46000 & see if it works.

 

[dmarc2024]

You may be facing bits, bytes & bib's scenario:

50 Terabytes = 46566.1 Gibibytes

Try 46000 & see if it works.

That worked - thanks.

Wow, that's a lot off loss - I was assuming it would be more like: 6TB drives x 10 = 72TB, minus formatting = 5.5TB per disk or 66TB, minus two drives for RZ2 = 55TB of usable space - more like what the attached screenshot was showing.

 

[gfngfn256]

Happy it worked for you.

Your problem had nothing todo with wasted space - but simple volume measurement representation standards.

Those drives advertised as "6tb" are in fact about 5.46 tebibyte or TiB which equals to 6 terabytes or TB - hence their advertised volume. All the rest of your observations above (GUI vs CLI etc.), are just the different units of bits/bytes/bibyte being used. No one is stealing your space!

Since it appears you have solved your issue, Maybe mark this thread as solved. At the top of the thread, choose the Edit thread button, then from the (no prefix) dropdown choose Solved.

 

[Kingneutron]

https://jro.io/capacity/
^ ZFS overhead calculator

https://wintelguy.com/zfs-calc.pl
^ ZFS capacity / pricing calculator

You're never going to get 100% of raw storage with any kind of RAID setup, there is always overhead.

' df ' , ' zpool list ' and ' zfs list ' are ALL going to return different results. Always go with the lowest number; you can fudge a bit if you have compression on, but don't forget - you need to leave room for snapshots and housekeeping.

That said, you have a 12-disk, single-vdev RAIDZ2. If your I/O seems a bit slow, break that up into 2x vdevs of 6 disks. You'll lose a bit of free space, but it will be more fault-tolerant (can lose up to 4 disks // 2 per vdev) and scrubs should finish faster.