Increase RAIDZ2 pool by replacing all disks

[chrisu]

Hello,

I have been searching for a reason an solution for autoexpand not working at my PVE 5.3-5 host.
I intented to increase the local ZFS pool by replacing all 2 TB disks in the pool by 4TB ones. So I replaced every single disk an resilvered the RAIDZ2 pool after each disk swap. Finally a actived autoexpand and was expecting the pool to grow. But the pool is still the same size. Here is the pool config.
pool: data
state: ONLINE
scan: resilvered 854G in 33h37m with 0 errors on Fri Jan 4 06:13:01 2019
config:

NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ata-WDC_WD4003FFBX-68MU3N0_VBG08PSR ONLINE 0 0 0
ata-WDC_WD4003FFBX-68MU3N0_VBG081BR ONLINE 0 0 0
ata-WDC_WD4003FFBX-68MU3N0_VBG0B0VR ONLINE 0 0 0
ata-WDC_WD4003FFBX-68MU3N0_VBG0JMGR ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
ata-TOSHIBA-TL100_27AB6258KSXU-part1 ONLINE 0 0 0
ata-TOSHIBA-TL100_27AB6272KSXU-part1 ONLINE 0 0 0
cache
ata-TOSHIBA-TL100_27AB6258KSXU-part2 ONLINE 0 0 0

autoexpand is on on the pool.

data autoexpand on local



The raidz2 is build on the disk devices and not to partitions. So I would have expected the pool to grow after replacing the disks.

Any help is welcome.

Thanks, Chris

 

[guletz]

Hi,

Any zfs pool use partions, not disk! So can you show your outuput like:

gdisk -l /dev/disk/by-id/ata-WDC_WD4003FFBX-68MU3N0_VBG08PSR

I guess the output is the same for all of the data disks!

 

[chrisu]

Hi,

thanks for the feedback. In contrary to adding partitions, this pool was setup by adding whole disks leading to the behavior that part1 will always cover nearly the whole disk capacity.

Thank you for the hint with gdisk. This shows 4TB assigned to part1

Number Start (sector) End (sector) Size Code Name
1 2048 7814019071 3.6 TiB BF01 zfs-5ab42b67ff0e24e8
9 7814019072 7814035455 8.0 MiB BF07

The same is true for the other 3 disks.

Why isn´t the pool growing?

Greetings, Chris

 

[guletz]

And what is show for:

zpool list -v

zpool get all

 

[chrisu]

Hi,

zpool list -v:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
data 7.25T 5.07T 2.18T - 47% 69% 1.00x ONLINE -
raidz2 7.25T 5.07T 2.18T - 47% 69%
ata-WDC_WD4003FFBX-68MU3N0_VBG08PSR - - - - - -
ata-WDC_WD4003FFBX-68MU3N0_VBG081BR - - - - - -
ata-WDC_WD4003FFBX-68MU3N0_VBG0B0VR - - - - - -
ata-WDC_WD4003FFBX-68MU3N0_VBG0JMGR - - - - - -
log - - - - - -
mirror 31.8G 416K 31.7G - 39% 0%
ata-TOSHIBA-TL100_27AB6258KSXU-part1 - - - - - -
ata-TOSHIBA-TL100_27AB6272KSXU-part1 - - - - - -
cache - - - - - -
ata-TOSHIBA-TL100_27AB6258KSXU-part2 79.8G 7.12G 72.7G - 0% 8%
rpool 59.5G 26.4G 33.1G - 76% 44% 1.00x ONLINE -
mirror 59.5G 26.4G 33.1G - 76% 44%
ata-TOSHIBA-TL100_27AB625LKSXU-part2 - - - - - -
ata-ADATA_SP900_2G2720042258-part2 - - - - - -

zpool get all pool data:
NAME PROPERTY VALUE SOURCE
data size 7.25T -
data capacity 69% -
data altroot - default
data health ONLINE -
data guid 18100051565717258810 -
data version - default
data bootfs - default
data delegation on default
data autoreplace off default
data cachefile - default
data failmode wait default
data listsnapshots off default
data autoexpand on local
data dedupditto 0 default
data dedupratio 1.00x -
data free 2.18T -
data allocated 5.07T -
data readonly off -
data ashift 12 local
data comment - default
data expandsize - -
data freeing 0 -
data fragmentation 47% -
data leaked 0 -
data multihost off default
data feature@async_destroy enabled local
data feature@empty_bpobj active local
data feature@lz4_compress active local
data feature@multi_vdev_crash_dump enabled local
data feature@spacemap_histogram active local
data feature@enabled_txg active local
data feature@hole_birth active local
data feature@extensible_dataset enabled local
data feature@embedded_data active local
data feature@bookmarks enabled local
data feature@filesystem_limits enabled local
data feature@large_blocks enabled local
data feature@large_dnode enabled local
data feature@sha512 enabled local
data feature@skein enabled local
data feature@edonr enabled local
data feature@userobj_accounting enabled local


Thank you and greetings, Chris

 

[guletz]

Hi again,

From your outuput, it seem to me to be OK

data 7.25T

For a raidz2, with 4x 3.6 T => 2 HDD x 3.6 T = 7.2(who is almost the same with 7.25T ). So the pool is larger and also expanded using the new disk capacity !!!!

 

[chrisu]

Hi, thank you, but I though the capacity should be 4x3.6 TB about 14 TB as the gross values are displayed. I have another system here that was initial setup with 4x4TB disks in raidz2 and it shows 14.5 TB:
NAME PROPERTY VALUE SOURCE
data size 14.5T -

Greetings, Chris

 

[guletz]

Hi, thank you, but I though the capacity should be 4x3.6 TB about 14 TB as the gross values are displayed. I have another system here that was initial setup with 4x4TB disks in raidz2 and it shows 14.5 TB:
NAME PROPERTY VALUE SOURCE
data size 14.5T -

Greetings, Chris

For raidz2 the usable capacity is total disk capacity - 2× 1 disk capacity(aka parity disk)

Good luck.

 

[chrisu]

Hello,

well, thats the difference between net and gross values oft the storage. In raidz2 your are loosing two devices from the gross capacity for parity. In my case I have a four disk setup with 4TB each disk. This results in 8TB net (aka usable space) and 16TB gross capacity.
It seems that zfs grabbed the whole disk capacity but has not aligned to pool to the new values....strange. Autoexpand should cover this but it seems that it is not working..

Greetings Chris

 

[chrisu]

Hi,

finally got the issue solved. The guides found for pool expansion are in most cases wrong as they activate autoexpand after the last drive replacement. But Autoexpand fires only when you online a device. So after turning autoexpand on I had to offline one disk and online it again (zpool offline poolname device, zpool online poolname device).

Thanks and best greetings, Chris

 

[ieronymous]

Hi,

finally got the issue solved. The guides found for pool expansion are in most cases wrong as they activate autoexpand after the last drive replacement. But Autoexpand fires only when you online a device. So after turning autoexpand on I had to offline one disk and online it again (zpool offline poolname device, zpool online poolname device).

Thanks and best greetings, Chris

Even though it is an old post, it' just came under my attention while searching for something relevant, I quite missed the meaning of you had to (after turning the pool off and on again) offline one disk and turn it back on. In a pool consisting of 4 drives like yours you had to do it for the last one only of for all 4 of them? The way you describe it cant make sense to me

Thank you

PS Is this way still relevant or something new has come up for zfs expansion