Expand Storage

[infinityM]

Hey Guys,

Ok so following some guides I setup a ZFS storage array on RaidZ1...
I started out with 6TB since I wanted to check how it works...

But now I've got a problem. I can't find any documentation on expanding the storage? Can someone please advise how to safely add more disks?
I'm extremely new to ZFS so please excuse my ignorance

 

[Dunuin]

You can't add new drives to a raidz pool. You need to backup everything, destroy the raidz1 pool and create a new raidz pool with more disks.

Also you need to destroy every VM and create new virtual harddisks if your number of drives of a raidz pool changes because the volblocksize of the virtual harddiscs won't be optimal for the new pool or everything will waste a lot of space. Volblocksize also can't be changed after creation.

Look here to calculate the right volblocksize. If you didn't changed the volblocksize from the default value(8K) and using ashift of 12 or 13 you are already wasting space. For example if you are using 4 disks for a raidz1 you will see on paper that 75% of the capacity is available to you. But every VM wastes space and is 50% bigger than it should be and in the end your can't store more data on your 4 drives than on just a mirror-stripes pool (like raid10) which would be much faster but gives you on the paper just 50% of the space.

 

[UdoB]

Well, there is no sane way to add one single drive to an existing raidz (with one or more vdev's). Be careful not to add a single drive to a pool by accident - all redundancy will get lost!

But: if that raidz consists of (for example) 4 drives in one single vdev it is totally acceptable to add another 4 drives as a second vdev. This will double space and also double IOPS.

The "old" VMs will stay on the old single vdev. There is no automatic re-balancing! New VMs will get stored on the vdev with the most space until both vdevs are nearly identical full. Only then new VMs get distributed between the two vdevs. So yes, optimal state requires a clean re-build from scratch.

For syntax and examples ask a search engine of your choice...

 

[oversite]

I like to be able to add and remove, but it's perhaps coming from the complete flexibility in ceph where i have most of the disks.
However, the only way i can use ZFS and sleep well and in the morning add and remove disks, add disks when needed etc is to use mirrored vdevs. I like mirroring anyway, and it's quick resilvering and i can DELETE a vdev when i need. So yes i know raidz1-2-3 give more space but i sleep better knowing i can CHANGE the pool so easily with mirrors.

 

[infinityM]

Does it not make more sense to then rather use normal Software raid rather?
RaidZ then feels extremely unscalable?...

 

[Dunuin]

Sure, but you will loose all of the great zfs featues that makes your data so safe.

 

[oversite]

Does it not make more sense to then rather use normal Software raid rather?
RaidZ then feels extremely unscalable?...

Not for me. I don't know what "normal software raid" is but most raid sw i know is not scalable but stiff. Zfs is great. Try it with mirrors for a while, you will love it. Try it virtually, add a bunch of disks in a vm and play with it.

 

[infinityM]

Not for me. I don't know what "normal software raid" is but most raid sw i know is not scalable but stiff. Zfs is great. Try it with mirrors for a while, you will love it. Try it virtually, add a bunch of disks in a vm and play with it.

The expansion issue is just a bit of a problem? Unless I'm missing something?
What would be the best way to make my proxmox backup disk expandable using ZFS, let's say i've got 4 drives now, and want to add 1 tomorrow (or maybe 2)?

Maybe I'm just missing something about ZFS?

 

[fabian]

raidZ is just not made for scaling in small increments. plan ahead, or be prepared to rebuild or extend in big chunks. the alternative is to use mirrors, where you can add disks in pairs of two (or three, if you are extra cautious and do three-way mirroring)

 

[oversite]

The expansion issue is just a bit of a problem? Unless I'm missing something?
What would be the best way to make my proxmox backup disk expandable using ZFS, let's say i've got 4 drives now, and want to add 1 tomorrow (or maybe 2)?

Maybe I'm just missing something about ZFS?

It's really a matter of taste, but i assume you want some kind of redundancy.
I would put them two on each vdev as mirror so the pool is two vdevs with mirrored disks. You can then any time add two more in a new vdev when you need more space. The good thing is that later, when you might want larger disks, you can remove a smaller vdev and wait for zfs to copy away the data from that vdev, Makes it a breeze to change from smaller to larger disks later on.

 

[infinityM]

It's really a matter of taste, but i assume you want some kind of redundancy.
I would put them two on each vdev as mirror so the pool is two vdevs with mirrored disks. You can then any time add two more in a new vdev when you need more space. The good thing is that later, when you might want larger disks, you can remove a smaller vdev and wait for zfs to copy away the data from that vdev, Makes it a breeze to change from smaller to larger disks later on.

Ok... So it's gonna take some getting used to...
THanks guys . Will attempt at learning a bit