RAIDZ2 with 6x 480GB SSD only 1.35TB usable total

M

mm3923

New Member
May 19, 2026
7
2
3
Hello!

I've set up a RAID2 with 4x 480GB SSD and got a usable total of 895GB. I've expanded the pool with two more 480GB SSD, and now I have only 1.35TB usable total storage.

Am I missing something, it should be more like 1.7TB, and not 1.35TB, right?

Thanks in advance!
 
Extending a ZFS pool can be done in different ways, and not all of them result in what one expects. Please show the actual output (in CODE-tags) of zpool status for that pool. RaidZ1/2/3 usable space depends a lot on alignment and padding as well (as more than one thread on this forum will show).

EDIT: ZFS is not Proxmox specific and other ZFS resources on the internet will probably also apply
 
Last edited:
Hello, and thank you for the quick reply!

I've used the following command for expansion:
Code: 
zpool attach Datastore1 raidz2-0 ata-SAMSUNG....
Here is the output of zpool status:

Code: 
root@proxmox-host1:~# zpool status Datastore1
  pool: Datastore1
 state: ONLINE
  scan: scrub repaired 0B in 00:10:33 with 0 errors on Tue May 19 22:04:46 2026
expand: expanded raidz2-0 copied 309G in 00:06:05, on Tue May 19 21:54:13 2026
config:

        NAME                                               STATE     READ WRITE CKSUM
        Datastore1                                         ONLINE       0     0     0
          raidz2-0                                         ONLINE       0     0     0
            ata-SAMSUNG_MZ7KM480HAHP-00005_S2HSNXRH401101  ONLINE       0     0     0
            ata-MK000480GWSSC_S521NA0RA00304               ONLINE       0     0     0
            ata-MK000480GWSSC_S521NA0T103050               ONLINE       0     0     0
            ata-SAMSUNG_MZ7KM480HAHP-00005_S2HSNXRH401057  ONLINE       0     0     0
            ata-SAMSUNG_MZ7KM480HAHP-00005_S2HSNXRH401054  ONLINE       0     0     0
            ata-SAMSUNG_MZ7KM480HMHQ-00005_S3F4NX0K505629  ONLINE       0     0     0

errors: No known data errors
 
  • Like
Reactions: leesteken
Code: 
NAME                       USED  AVAIL  REFER  MOUNTPOINT
Datastore3                 421G   833G   140K  /Datastore3
Datastore3/vm-105-disk-0  2.98G   833G  2.98G  -
Datastore3/vm-106-disk-0  31.2G   833G  31.2G  -
Datastore3/vm-107-disk-0  17.2G   833G  17.2G  -
Datastore3/vm-108-disk-0  12.1G   833G  12.1G  -
Datastore3/vm-109-disk-0  26.6G   833G  26.6G  -
Datastore3/vm-110-disk-0  11.0G   833G  11.0G  -
Datastore3/vm-111-disk-0  10.7G   833G  10.7G  -
Datastore3/vm-112-disk-0  37.7G   833G  37.7G  -
Datastore3/vm-112-disk-1   105K   833G   105K  -
Datastore3/vm-112-disk-2  81.4K   833G  81.4K  -
Datastore3/vm-113-disk-0  54.7G   833G  54.7G  -
Datastore3/vm-113-disk-1   105K   833G   105K  -
Datastore3/vm-113-disk-2    93K   833G    93K  -
Datastore3/vm-120-disk-0   107G   833G   107G  -
Datastore3/vm-121-disk-0  37.2G   833G  37.2G  -
Datastore3/vm-122-disk-0  9.04G   833G  9.04G  -
Datastore3/vm-123-disk-0  29.9G   833G  29.9G  -
Datastore3/vm-124-disk-0  33.2G   833G  33.2G  -
Datastore3/vm-124-disk-1  81.4K   833G  81.4K  -
Datastore3/vm-124-disk-2   105K   833G   105K  -

Code: 
Filesystem              Size  Used Avail Use% Mounted on
Datastore3              834G  256K  834G   1% /Datastore3
 
Last edited:
  • Like
Reactions: UdoB
Thanks for the information. I’m just not entirely sure if I understood it correctly.

I also had a Datastore1 that originally consisted of 4x 480GB drives and was later expanded to 6x 480GB — same issue. As a test, I moved all disks to Datastore3 so that there was ultimately no used space left on Datastore1. However, it still only showed 1.35TB total capacity.

I then deleted Datastore1 and rebuilt it directly as a new 6x 480GB SSD RAIDZ2 setup — now it shows 1.85TB total capacity.

So this probably has nothing to do with the rewrite topic, because the datastore was completely empty beforehand and it still only displayed 1.35TB, right?
 
There was or there is an error in capacity reporting. I have no link, but the main point was that "more capacity" was actually present but just not reflected by the displayed values.

As you found your solution by recreating the pool you may please tag this thread "Solved" now.

Edit: there are some problems left with "expansion", e.g.: https://github.com/openzfs/zfs/issues/17784
 
Last edited:
Hello,
it isn't solved - as the Datastore3 still shows 1.35TB. And yes I could move all Disks to Datastore1 and recreate Datastore3, but I would rather find the issue and "repair" Datastore3.
 
UdoB said:
Oh, sorry. I misread it - too quickly...
Click to expand...
No problem. I’ve read about the padding issue, and I’m not entirely sure whether it might actually make more sense to switch to a RAID10 setup with ZFS anyway.

I have a total of 12x 480GB SSDs available that I could use. At the moment, I was planning to use 2 pools with RAIDZ2, each consisting of 6x 480GB SSDs. But with the padding overhead, I probably wouldn’t gain much more usable space compared to putting all 12 drives into a RAID10 configuration, right?

What’s the experts’ opinion on this?

Thank you very much!
 
mm3923 said:
I was planning to use 2 pools with RAIDZ2, each consisting of 6x 480GB SSDs. But with the padding overhead, I probably wouldn’t gain much more usable space compared to putting all 12 drives into a RAID10 configuration, right?
Click to expand...
Well..., it is "50%" versus "66%". For this 16% more space you lose two third of available IOPS. Six mirrors have three times the IOPS as two RaidZ2 offer, every single vdev counts.

If you are storing VMs on the resulting pool then this is important and I would go for six mirrors. If you are "only" storing bulk data on it then two RaidZ2 is fine. Especially as it has two disks redundancy.

More interesting details: https://github.com/jameskimmel/opinions_about_tech_stuff/blob/main/ZFS/The problem with RAIDZ.md
 
Hello Udo,

Thank you very much for the explanation. I use the datastore for VMs only; static data is stored on a NAS with HDDs.

From this, I conclude that I should probably use RAID10 (i.e. 6x mirror vdevs) for performance reasons as well, since I would hardly lose any storage capacity but would gain a lot in terms of speed.

In that case, I’ll go with that setup. Thank you very much!
 
  • Like
Reactions: UdoB
mm3923 said:
I also had a Datastore1 that originally consisted of 4x 480GB drives and was later expanded to 6x 480GB — same issue. As a test, I moved all disks to Datastore3 so that there was ultimately no used space left on Datastore1. However, it still only showed 1.35TB total capacity.
Click to expand...
The way COW works, the 2:2 parity data is still on the zpool after you 'move it out', and still being used to calculate the total space of the pool. You need to actually replace that data with 2:4 parity data so it calculates with that. 1.35TB sounds about right for your 4 to 6 disk expanded Z2 pool with the old 2:2 parity ratio still in place.

mm3923 said:
I then deleted Datastore1 and rebuilt it directly as a new 6x 480GB SSD RAIDZ2 setup — now it shows 1.85TB total capacity.
Click to expand...
In this case the pool is directly setup with 2:4 parity:data ratio and the resulting space is (the expected) 1.85TB.

mm3923 said:
From this, I conclude that I should probably use RAID10 (i.e. 6x mirror vdevs) for performance reasons as well, since I would hardly lose any storage capacity but would gain a lot in terms of speed.
Click to expand...
Mirrors are not only faster than Z2, but the required time to resilver (read: stressed pool) is also significantly less. Although with SSD's this is probably less of a hurdle as it is with large harddisks.

In case you didn't know it yet, this is a really nice tool to figure out the resulting space for ZFS setups:
https://wintelguy.com/zfs-calc.pl
 
Last edited:
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom
Back