zfs expansion

[Han Boetes]

According to https://github.com/openzfs/zfs/pull/12225 zfs expansion is a feature coming soon.

So let's look at this zpool status rpool output:

# zpool status rpool pool: rpool state: ONLINE scan: scrub repaired 0B in 01:10:22 with 0 errors on Sun Aug 14 01:34:23 2022 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ata-CT1000MX500SSD1_1833E14D4FB7-part3 ONLINE 0 0 0 ata-CT1000MX500SSD1_1832E14CD765-part3 ONLINE 0 0 0 ata-CT1000MX500SSD1_2110E583B40F-part3 ONLINE 0 0 0 ata-CT1000MX500SSD1_2149E5EBED2B ONLINE 0 0 0 ata-CT1000MX500SSD1_2110E58502E4-part3 ONLINE 0 0 0 ata-CT1000MX500SSD1_2110E58502F2-part3 ONLINE 0 0 0

Now imagine the rpool was close to being full, and you want to add another terabyte. All you can do to expand is another hard disk that won't be part of the existing raid-array. You can only add another disk, which wouldn't be in raid, so you'd have to add two disks, and then you'd basically create a raid0+1, more or less.

With this new feature you can add a disk and grow the raidz1-0 array with one more disk.

 

[Dunuin]

First you should expand that pool way earlier beforesit is full. A ZFS pool always should have 20% of free space left otherwise it will become slow and fragment faster.

And yes, with that new feature you could add a single disk to get more capacity. But adding a new disk won't increase any performance. So relative to your pool size the pool will become slower by doing this (it will take longer then before to fill the pool from 0% to 80%). Also the parity to data ratio won't get better either so you are wasting usable capacity. You still loose 1/6th of raw capacity for parity after expanding a 6 disk to a 7 disk raidz1. When creating a new raidz1 with 7 disks you would only loose 1/7th of your raw capacity.
So even with that new feature, if you want the full performance and full capacity you still have to destroy and recreate the whole pool from scratch.

In my opinion that feature isn't that great on paper. Raid never replaces a backup so you should have a copy of your pool anyway. If you got another copy of everything it wouldn't be a problem to destroy the pool and and create a new one with more disks.
So only downside is downtime and that you will have to wait to copy all the data. But you get more usable capacity and performance that way. So in case downtime isn't that important I still would ignore the new feature and just destroy and recreate the pool the old way.
Or if you care more about IOPS performance and less about parity overhead its also a good idea to create a second raidz1 vdev and stripe it with the old one. For example adding 6 more disks to that existing pool 6 disk pool so you get a 12 disk striped raidz1. Easier of cause when using a striped mirror instead of a raidz where yu could just add pairs of disks without loosing any relative performance or usable capacity.