Expanding ZFS root file system

[Davidoff]

Hi there. I have a Proxmox installation that I installed on a ZFS zpool. The zpool was configured with the Proxmox installation and consists of two mirrored 500 GB SSDs, like this:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdu2    ONLINE       0     0     0
            sdv2    ONLINE       0     0     0

I'm now starting to run out of space, so got an additional two 2TB SSDs that I was planning to add to the pool as another mirrored set of drives. I've already looked up the general procedure regarding how to add new drives to a pool and believe that I should do the following (assuming the new drives are at /dev/sdw and /dev/sdx):

zpool add rpool mirror /dev/sdw /dev/sdx

However, I noticed that in the original rpool, only certain partitions of the drives are included in pool. Following is how the existing drives are partitioned:

sdu      65:64   0 447.1G  0 disk
├─sdu1   65:65   0  1007K  0 part
├─sdu2   65:66   0 447.1G  0 part
└─sdu9   65:73   0     8M  0 part
sdv      65:80   0 447.1G  0 disk
├─sdv1   65:81   0  1007K  0 part
├─sdv2   65:82   0 447.1G  0 part
└─sdv9   65:89   0     8M  0 part

So the dumb question I have is whether or not I need to do anything special with the two drives being added above and beyond the zpool add command above. For example, would I need to partition the two new drives in a manner similar to the two existing drives to create three partitions? Just a bit worried that I'll mess things up as I'm not too familiar with the mechanics involved in using a ZFS pool as the boot and system drive.

Any thoughts, suggestions or comments would be most appreciated.

 

[jim.bond.9862]

You do not need to do anything with new drives. Zfs will do all by itself.
However you may not see all the space you adding.
Mgs can not accomodate different size disks within the same pool.
So until all drives are the same size you will only get a sum of space equal to the smallest device.
So even though you adding 2t disks. You will only expand the pool by 500gb. Now if you want to expand it to 2tb. You will need to replace the original disks in vdev and expand the pool. I think it will be like zfs replace command for each disk let it resolver and repeat. Once it done zfs expand or something like that.

 

[Davidoff]

Hey thanks for the notes Jim.

For some reason I thought that the rule on same sized devices only applied to drives within a given vdev, but not to the various vdevs within a pool. So for example I could have a ZFS pool comprised of two drives each, with one pool being 500 GB on each drive and the second being, say 2TB each, resulting in a total available capacity of 2.5TB. Is that not the case? Just to be clear, in my situation, I'm trying to add the two new drives as a vdev into the existing ZFS pool.

As for replacement, that may be something I'd be open to considering, but given that the existing drives are the system drives with only partitions in each drive being part of the ZFS pool, wouldn't I also need to somehow replicate both disks to ensure the other partitions are preserved?

 

[jim.bond.9862]

No. If I remeber correctly you can not mix disk size in vdevs as it will not work. You can use different size vdevs but only with raidZ and mirror setups and you still lose the data recovery guaranty on disk failure.

I do not remember 100% but zfs replace command will do everything you need to the disk to make it work.
You may only need to run gdisk and create the partition table. Not actual partitions. But replace may even do that for you.

 

[Davidoff]

Thanks Jim.

 

[frenchfries]

Pretty sure Jim is wrong. You can add VDEVs of any size and they will be incorporated into the pool. eg. 2x 500GB mirrors and you add 2x 2tb mirrors does indeed equal 2.5tb.

Access is where it gets weird, ZFS normally does round robin for access but obviously the smallest one is going to fill up in that case so it actually allocates based on freespace. Therefore the 2tb disks will get more action than the 500GB ones.

Actually that behaviour is no longer correct for zfs 0.7, apparently it's more complicated now. See http://jrs-s.net/category/open-source/zfs/

 

[frenchfries]

In terms of the disk partition structure, I'm not sure. Make a VM to test. You can also test the write behaviour akin to the article I linked to.

 

[jim.bond.9862]

Pretty sure Jim is wrong. You can add VDEVs of any size and they will be incorporated into the pool. eg. 2x 500GB mirrors and you add 2x 2tb mirrors does indeed equal 2.5tb.

Access is where it gets weird, ZFS normally does round robin for access but obviously the smallest one is going to fill up in that case so it actually allocates based on freespace. Therefore the 2tb disks will get more action than the 500GB ones.

Actually that behaviour is no longer correct for zfs 0.7, apparently it's more complicated now. See http://jrs-s.net/category/open-source/zfs/

I never said I was 100% sure.
But I am old-school kind of guy. I remember things that today may not be true anymore but may still hold merits.
MIxing disks in the same pool can be troublesome especially on a root pool. So , personally, I preffere to err on a side of caution.
In my setup I run root on mirrored 120g ssd. And just add a special pool for more system storage.
All my container and vm disks go on that pool. All my configs are on main/root pool.
Is it fast? No since the secondary pool is on old spinners.
Is it fast enought for me? Yes.

 

[Davidoff]

Hey thanks frenchfries, that is helpful. I thought that was the case as well but then started to think I had misunderstood the requirement when I saw Jim's post, so was thinking of different approaches to avoid the need to even deal with it - namely, instead of trying to expand the system drive, just creating a separate pool and moving CTs and VMs over to the new pool. Use by CTs is the primary reason for the additional storage. I would have preferred to keep everything in one pool but Jim's post (along with some other logistical concerns) gave me pause.

Coincidentally while I was typing this Jim also described the same approach that I now plan to use.

 

[frenchfries]

@jim.bond.9862 I wasn't trying to offend, just making sure the correct info was out there.

Separate pool is a good idea as it allows you to move the storage to another host easily.

 

[jim.bond.9862]

@jim.bond.9862 I wasn't trying to offend, just making sure the correct info was out there.

Separate pool is a good idea as it allows you to move the storage to another host easily.

No offence taken. I was just clarifying my response and pov.