Proxmox VE 9 existing ZFS vdev expansion?

L

liamlows

Well-Known Member
Jan 9, 2020
40
13
48
Hello,

So I recently performed my upgrade from proxmox VE 8 to 9 and with it I was excited to see that v.9 now includes a form of ZFS expansion:

"ZFS now supports adding new devices to existing RAIDZ pools with minimal downtime."

I currently have a ZFS pool created on one of my PVE nodes that consists of 12 1.2TB disks. I purchased 12 more 1.2TB disks (of same size and features just a different manufacturer) and intend to add the additional capacity to the pool. My hope was to use the newly added feature of ZFS to perform a vdev expansion (not creating a new vdev and adding it to the pool) so my singular vdev in the pool I currently use would go from 12 disks in raidz2 to 24 disks in raidz2 with no data loss. However, when I went into the host i checked the feature flags for feature@vdev_expansion, I realized that it was not present implying this change is not possible.

Is this the case? Is my only option to create another vdev of 12 disks and add it to the pool or destroy the old one and create a new one with all 24 disks? I wasn't sure what the roadmap meant when it said the above so I was hoping someone here could provide some guidance.

Thanks in advance!
 
liamlows said:
Hello,

So I recently performed my upgrade from proxmox VE 8 to 9 and with it I was excited to see that v.9 now includes a form of ZFS expansion:

"ZFS now supports adding new devices to existing RAIDZ pools with minimal downtime."

I currently have a ZFS pool created on one of my PVE nodes that consists of 12 1.2TB disks. I purchased 12 more 1.2TB disks (of same size and features just a different manufacturer) and intend to add the additional capacity to the pool. My hope was to use the newly added feature of ZFS to perform a vdev expansion (not creating a new vdev and adding it to the pool) so my singular vdev in the pool I currently use would go from 12 disks in raidz2 to 24 disks in raidz2 with no data loss. However, when I went into the host i checked the feature flags for feature@vdev_expansion, I realized that it was not present implying this change is not possible.

Is this the case? Is my only option to create another vdev of 12 disks and add it to the pool or destroy the old one and create a new one with all 24 disks? I wasn't sure what the roadmap meant when it said the above so I was hoping someone here could provide some guidance.

Thanks in advance!
Click to expand...
My guess is that you should do
Code: 
zpool POOL upgrade
in order to activate new features...
 
  • Like
Reactions: liamlows
Gilberto Ferreira said:
My guess is that you should do
Code: 
zpool POOL upgrade
in order to activate new features...
Click to expand...
Oh nice I wasn't aware that was needed. I went ahead and ran the command zpool upgrade ZFS1 and it looks like it added the feature raidz_expansion but that isn't strictly the vdev expansion feature (or so it seems). Do you know if the vdev expansion is supported with v9 of proxmox?

EDIT: it looks like this is the intended feature to support vdev expansion. I am curious though, is this the best way to handle this expansion? Would it be better to simply add a secondary vdev rather than attempt to expand the first one? Are there any added benefits other than increasing parity accross 2 vdevs instead of 1?
 
Last edited:
liamlows said:
I currently have a ZFS pool created on one of my PVE nodes that consists of 12 1.2TB disks. I purchased 12 more 1.2TB disks (of same size and features just a different manufacturer) and intend to add the additional capacity to the pool. My hope was to use the newly added feature of ZFS to perform a vdev expansion (not creating a new vdev and adding it to the pool) so my singular vdev in the pool I currently use would go from 12 disks in raidz2 to 24 disks in raidz2
Click to expand...
No, no, no - do not do this.

I see that this is tempting, but a 12 disk wide vdev is already considered "wide".

Additionally: two vdevs will double IOPS, which is always a good idea ;-)


Disclaimer: I've never setup a 24 wide vdev, not even for testing - and I never will!
Disclaimer2: just my 2 €¢ and ymmv - as usual...
 
  • Like
Reactions: Johannes S and liamlows
UdoB said:
No, no, no - do not do this.

I see that this is tempting, but a 12 disk wide vdev is already considered "wide".

Additionally: two vdevs will double IOPS, which is always a good idea ;-)


Disclaimer: I've never setup a 24 wide vdev, not even for testing - and I never will!
Disclaimer2: just my 2 €¢ and ymmv - as usual...
Click to expand...
Thanks for the insight! Yea as I've been thinking about it I actually feel like the two 12 disks Vdevs in the pool would be better than 1. If I simply create the new vdev and add it to the pool, should I do anything after that to optimize the new storage layout?
 
  • Like
Reactions: UdoB
liamlows said:
If I simply create the new vdev and add it to the pool, should I do anything after that to optimize the new storage layout?
Click to expand...
Well..., after adding that second vdev everything will work and the added capacity is available immediately. There is no additional step required.

But all "old" data is still stored on the old vdev. There is no automatic rebalancing. Last time I did this I utilized an external script to shuffle data around - which took days to finish.

If I remember correctly newer ZFS releases offer an integrated helper mechanism, but unfortunately I have not reference available.
 
UdoB said:
But all "old" data is still stored on the old vdev. There is no automatic rebalancing. Last time I did this I utilized an external script to shuffle data around - which took days to finish.
Click to expand...
I don't know amost nothing about zfs and related stuff, but it's should not be a resilvering in place at this point?

But I guess a resilver is just well loose some vdev in the same raid-z, right?
 
Last edited:
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom