Expand pool

[Alessandro 123]

Hi.
I have to add 4 new SSDs disks to an existing RAIDZ2 pool.

I'm not an ZFS expert and I don't want to loose/reduce redundancy making mistakes.

Is this the right command to use ?

# zpool add -n rpool raidz2 pci-0000:03:00.0-sas-phy4-lun-0 pci-0000:03:00.0-sas-phy5-lun-0 pci-0000:03:00.0-sas-phy6-lun-0 pci-0000:03:00.0-sas-phy7-lun-0
would update 'rpool' to the following configuration:
    rpool
      raidz2
        pci-0000:03:00.0-sas-phy0-lun-0-part2
        pci-0000:03:00.0-sas-phy1-lun-0-part2
        pci-0000:03:00.0-sas-phy2-lun-0-part2
        pci-0000:03:00.0-sas-phy3-lun-0-part2
      raidz2
        pci-0000:03:00.0-sas-phy4-lun-0
        pci-0000:03:00.0-sas-phy5-lun-0
        pci-0000:03:00.0-sas-phy6-lun-0
        pci-0000:03:00.0-sas-phy7-lun-0

Looks good to me.
Any additional operation to do in PVE or the new size is automatically detected ?

 

[aaron]

I don't think it will work with just the pci-0000:03:00.0-sas-phy4-lun-0 but will probably need the full path such as /dev/disk/by-..../disk.

On a quick note for the future (as you probably don't want to re setup that node): If you use a RAIDz2 with 4 disks you might as well set it up as a pool of mirror VDEVs to get a pool resembling RAID10.

Why? Mainly for performance. See, any RAIDz will behave like a single disk regarding IOPS with lots of bandwidth.

A mirror VDEV will behave like 1 disk (IOPS, bandwidth) when writing, but when reading it will behave like the number of disks in the mirror vdev (most times 2 disks) in both, IOPS and bandwidth.

Thus, when comparing a raidz2 with 4 disks to two mirror vdevs, you will get double the IOPS when writing and 4 times the IOPS when reading. And IOPS is usually what you want if you have VMs on that storage.

Additionally, you won't run into the problem on RAIDz with ZVOLs (used for VM disks) that different block sizes can cause a massive overhead of space needed ( see https://forum.proxmox.com/threads/zfs-counts-double-the-space.71536/#post-320919 )

 

[Alessandro 123]

No, I'll never ever use anything lower than RAID-Z6 for redudancy. Never. Had (a lot) of multiple failures on RAID10 in many years. Never more for no reason. Never.

The posted commadn seems to work even with just `pci-0000:03:00.0-sas-phy4-lun-0`. Other than this, is the command correct ? I could try to run without "-n" and see if it works. But if it works, I have to be 10000% sure that redudancy will remain to RAID-Z2

 

[aaron]

The posted commadn seems to work even with just `pci-0000:03:00.0-sas-phy4-lun-0`.

Good to know and to remember as so far I've always worked with full paths.

Other than this, is the command correct ? I could try to run without "-n" and see if it works. But if it works, I have to be 10000% sure that redudancy will remain to RAID-Z2

Well, you already did test it with the -n flag. That is as close as you can get to the real deal

The redundancy will be 2 raidz2 vdevs which the pool will use. In old RAID terminology the closest thing to this would probably be a raid 60 as the pool will stripe new data across both vdevs.

 

[Alessandro 123]

With that, I can loose *ANY* 2 disks for each vdevs. In perfect conditions , I can loose *ANY* two disks from vdev1 plus *ANY* two disks from vdev2, so, 4 disks.

In terms of redudancy and safety , this big pool is the same as 2 different pools with just 1 vdev each. The only difference is that I can stripe data across both.

well, not exaclty, with 2 different pools, in case of disaster, there is canche that one pool is stil working (in example, loose 3 disk, one poll keep working), with "RAID60" i'll loose everything.

 

[LnxBil]

No, I'll never ever use anything lower than RAID-Z6 for redudancy. Never. Had (a lot) of multiple failures on RAID10 in many years. Never more for no reason. Never.

Experience can be so different: Maybe you took the I in RAID literally, because I've been monitoring multiple datacenters with disks numbers in the thousands in various disk configurations and we seldomly have failures of two devices at once and we never lost a RAID. We're using exclusively enterprise grade hardware and also use them longer than 5 years. Don't get me wrong, we also use RAIDz2, but only for backups and never for VM data because it's performance is underwhelming, as @aaron described. Maybe you can elaborate a bit more on your experience and circumstance.

 

[Alessandro 123]

Experience can be so different: Maybe you took the I in RAID literally, because I've been monitoring multiple datacenters with disks numbers in the thousands in various disk configurations and we seldomly have failures of two devices at once and we never lost a RAID. We're using exclusively enterprise grade hardware and also use them longer than 5 years. Don't get me wrong, we also use RAIDz2, but only for backups and never for VM data because it's performance is underwhelming, as @aaron described. Maybe you can elaborate a bit more on your experience and circumstance.

No, i only use enterprise-grade SSD disks from not less than 10 years with high-grade hardware raid controllers (where I can't use software raid, i really hate the raid controller)

I had 4 major failures:
- One time the controller failed, kicked out a disk, then, immediatly when replacing it with the new disk, the other mirror was kicked out too. Raid lost.
- One time a controller started to rebuild the NEW disk over the OLD one. Awesome!
- Two time, two disk, from the same batch, failed during a rebuild.

Keep in mind that not all are able to choose disks from different make/batch, most of the time you are forced certified drives and most of the time, these certified drives can't be choosen, they will be provider from the server vendor almost always from the same batch.
In a RAID1, both disk in the mirror has the same write pattern thus a failure of both (due to bugs, or anything else) is real and this happened two times to me.

 

[LnxBil]

Thanks for sharing! Valid points.

 

[ca_maer]

If I understand correctly you are using hardware raid with ZFS ? Seems like all your previous issues were controller related which shouldn't happened with an HBA