ZFS Special VDEV

[Mrt12]

Hi,
On the beginning of 2024, I set up a new storage server for my work using ZFS and Samba on Proxmox. I added to the ZFS pool a "special" vdev, which gives a really good performance when the 40TB (around 50 Million files) of data are backed up, as the file metadata can be analysed very quickly thanks to the special vdev.
The storage server is very busy and daily used. So, the special device becomes more and more full, of course. I have now an usage of 137GB and 85GB free on the special vdev. I read that the special device shall not be completely filled, and that ZFS starts to allocate blocks from the normal disks as soon as the special device reaches 75% full, but I cannot find this anymore. So I wonder:
from which point on should I increase the special device? at the moment, it seems to me like ~60% full, so should still be okay. I think when I expand it, I will use 400GB SSDs.

Greetings and a happy new year to you all!

 

[Kingneutron]

I would proactively expand it with the 400s or even 480/512GB, based on how much you project to expand over the next year. Then you don't have to worry about it unless something starts dying.

Should be as easy as ' zpool replace olddisk newdisk/by-id ' and let resilver finish before doing 2nd one.

I think on my 14-disk DRAID I currently have ~128GB special device mirror, with ~32TB allocated on the pool it has ~96GB allocated and ~31GB still free. I could throw 2x256GB at it and probably be set for life based on current trend, it's only a tertiary backup that gets powered on once every month or two.

https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954
[[
The metadata special device stops allocating more stuff when it is more than 75% full as well, though this can be tweaked. I’d be more comfortable around 10% +/-.
]]

 

[Mrt12]

Hi,
thanks for that. Yes I will definitely replace it. I only wonder how urgent it is. Because I have no 400s or even 480s at hand, currently is still holidays and I would need to order the disks first. What happens if my special device gets full? will the pool fail? I hope not?

 

[LnxBil]

What happens if my special device gets full? will the pool fail? I hope not?

As you already wrote, it'll spill over to the normal disks, which WILL be slower and you cannot easily move the data back to the SSD if the disk capacity is increased (need to send/receive the datasets). So expand proactively as @Kingneutron already said.

 

[Mrt12]

As you already wrote, it'll spill over to the normal disks, which WILL be slower and you cannot easily move the data back to the SSD if the disk capacity is increased (need to send/receive the datasets). So expand proactively as @Kingneutron already said.

Will do. Actually I also already practiced how to do that, to know in advance how it works. Will "zpool attach" the replacement disk first, then wait for the resilver. Then "zpool detach" one of the old disks, remove the disk and put a new disk. Then "zpool attach" the new disk again, and wait for the resilver. At this point I have 3 disks, 2 new and one old. Then "zpool detach" the remaining old disks and wait for the autoexpand to happen, done.

 

[Mrt12]

By the way, I expanded now to 2x 480GB. I used Kingston DC600ME SSDs. I can recommend them, they are good suited for a special vdev in my opinion.