Proxmox VE 9 existing ZFS vdev expansion?

L

liamlows

Well-Known Member
Jan 9, 2020
40
13
48
Hello,

So I recently performed my upgrade from proxmox VE 8 to 9 and with it I was excited to see that v.9 now includes a form of ZFS expansion:

"ZFS now supports adding new devices to existing RAIDZ pools with minimal downtime."

I currently have a ZFS pool created on one of my PVE nodes that consists of 12 1.2TB disks. I purchased 12 more 1.2TB disks (of same size and features just a different manufacturer) and intend to add the additional capacity to the pool. My hope was to use the newly added feature of ZFS to perform a vdev expansion (not creating a new vdev and adding it to the pool) so my singular vdev in the pool I currently use would go from 12 disks in raidz2 to 24 disks in raidz2 with no data loss. However, when I went into the host i checked the feature flags for feature@vdev_expansion, I realized that it was not present implying this change is not possible.

Is this the case? Is my only option to create another vdev of 12 disks and add it to the pool or destroy the old one and create a new one with all 24 disks? I wasn't sure what the roadmap meant when it said the above so I was hoping someone here could provide some guidance.

Thanks in advance!
 
liamlows said:
Hello,

So I recently performed my upgrade from proxmox VE 8 to 9 and with it I was excited to see that v.9 now includes a form of ZFS expansion:

"ZFS now supports adding new devices to existing RAIDZ pools with minimal downtime."

I currently have a ZFS pool created on one of my PVE nodes that consists of 12 1.2TB disks. I purchased 12 more 1.2TB disks (of same size and features just a different manufacturer) and intend to add the additional capacity to the pool. My hope was to use the newly added feature of ZFS to perform a vdev expansion (not creating a new vdev and adding it to the pool) so my singular vdev in the pool I currently use would go from 12 disks in raidz2 to 24 disks in raidz2 with no data loss. However, when I went into the host i checked the feature flags for feature@vdev_expansion, I realized that it was not present implying this change is not possible.

Is this the case? Is my only option to create another vdev of 12 disks and add it to the pool or destroy the old one and create a new one with all 24 disks? I wasn't sure what the roadmap meant when it said the above so I was hoping someone here could provide some guidance.

Thanks in advance!
Click to expand...
My guess is that you should do
Code: 
zpool POOL upgrade
in order to activate new features...
 
  • Like
Reactions: liamlows
Gilberto Ferreira said:
My guess is that you should do
Code: 
zpool POOL upgrade
in order to activate new features...
Click to expand...
Oh nice I wasn't aware that was needed. I went ahead and ran the command zpool upgrade ZFS1 and it looks like it added the feature raidz_expansion but that isn't strictly the vdev expansion feature (or so it seems). Do you know if the vdev expansion is supported with v9 of proxmox?

EDIT: it looks like this is the intended feature to support vdev expansion. I am curious though, is this the best way to handle this expansion? Would it be better to simply add a secondary vdev rather than attempt to expand the first one? Are there any added benefits other than increasing parity accross 2 vdevs instead of 1?
 
Last edited:
liamlows said:
I currently have a ZFS pool created on one of my PVE nodes that consists of 12 1.2TB disks. I purchased 12 more 1.2TB disks (of same size and features just a different manufacturer) and intend to add the additional capacity to the pool. My hope was to use the newly added feature of ZFS to perform a vdev expansion (not creating a new vdev and adding it to the pool) so my singular vdev in the pool I currently use would go from 12 disks in raidz2 to 24 disks in raidz2
Click to expand...
No, no, no - do not do this.

I see that this is tempting, but a 12 disk wide vdev is already considered "wide".

Additionally: two vdevs will double IOPS, which is always a good idea ;-)


Disclaimer: I've never setup a 24 wide vdev, not even for testing - and I never will!
Disclaimer2: just my 2 €¢ and ymmv - as usual...
 
  • Like
Reactions: Johannes S and liamlows
UdoB said:
No, no, no - do not do this.

I see that this is tempting, but a 12 disk wide vdev is already considered "wide".

Additionally: two vdevs will double IOPS, which is always a good idea ;-)


Disclaimer: I've never setup a 24 wide vdev, not even for testing - and I never will!
Disclaimer2: just my 2 €¢ and ymmv - as usual...
Click to expand...
Thanks for the insight! Yea as I've been thinking about it I actually feel like the two 12 disks Vdevs in the pool would be better than 1. If I simply create the new vdev and add it to the pool, should I do anything after that to optimize the new storage layout?
 
  • Like
Reactions: UdoB
liamlows said:
If I simply create the new vdev and add it to the pool, should I do anything after that to optimize the new storage layout?
Click to expand...
Well..., after adding that second vdev everything will work and the added capacity is available immediately. There is no additional step required.

But all "old" data is still stored on the old vdev. There is no automatic rebalancing. Last time I did this I utilized an external script to shuffle data around - which took days to finish.

If I remember correctly newer ZFS releases offer an integrated helper mechanism, but unfortunately I have not reference available.
 
UdoB said:
But all "old" data is still stored on the old vdev. There is no automatic rebalancing. Last time I did this I utilized an external script to shuffle data around - which took days to finish.
Click to expand...
I know almost nothing about zfs and related stuff, but it's should not be a resilvering in place at this point?

But I guess a resilver is just well loose some vdev in the same raid-z, right?
 
Last edited:
  • Like
Reactions: Johannes S
Gilberto Ferreira said:
I know almost nothing about zfs and related stuff, but it's should not be a resilvering in place at this point?
Click to expand...
You have a single vdev and you are adding a new, empty, second one. No resilvering, no rebalance, no nothing will happen. Everything is fine as it is.

As already said: all data is on the old vdev, at first. Technically that is fine! Seen from a performance standpoint it is bad. Reading data will happen only utilizing the disks in the old vdev because all of the data is there. Writing data will (mostly) happen only on the new vdev - because this one is empty.

That is the reason one would wish to have the data rebalanced = distributed onto both (all) vdevs. This does not happen automatically. Only after deleting some and writing a lot new data both vdevs may eventually get filled ~equally.

Gilberto Ferreira said:
But I guess a resilver is just well loose some vdev in the same raid-z, right?
Click to expand...
You can not loose "some vdev in the same raid-z"! One vdev is one RaidZ - or a RaidZ2 in your case.

A RaidZ2 vdev may loose two devices without loosing data. You can loose two drives in the old vdev and two drives in the new vdev at the same time and the data is still readable.

At least that is the plan. But I would produce some sweat if this actually happens to me...

Obligatory hint: Mirroring/RaidZ/Z2/Z3 (plus possibly multiple snapshots) does not count as a backup. Never.
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom